Data Engineer
Posted Updated
Description
Moolya Software Testing in Bengaluru, Karnataka, seeks an experienced data engineer to design, build, and maintain scalable ETL/ELT pipelines using Spark, PySpark, Spark SQL. You will optimize SQL queries, orchestrate workflows with Airflow or Dagster, and manage Databricks workloads.
Responsibilities include designing data models, lakehouse architectures, and governance practices, plus ownership of production workloads, troubleshooting, and collaboration with analysts and stakeholders to
Design, build, and maintain scalable ETL/ELT pipelines using Apache Spark, PySpark, and Spark SQL. Develop and optimize complex SQL queries, with a focus on performance tuning and cost efficiency. Orchestrate and schedule workflows using Airflow, Dagster, or similar platforms. Build and manage data workloads on Databricks or comparable Spark-based platforms. Design data models, warehouse layers, and lakehouse structures that support analytics and reporting. Work with Parquet, Delta Lake, and/or Apache Iceberg table formats. Implement data quality checks, metadata management, lineage tracking, and governance practices. Own production workloads: monitor jobs, troubleshoot failures, and perform root-cause analysis to prevent recurrence. Collaborate with analysts, engineers, and business stakeholders to understand requirements and deliver trusted data. Document pipelines, data models, and operating procedures. Strong experience with Apache Spark, PySpark, and Spark SQL. Proficiency in Python, Java, or Scala. Strong SQL skills, including query optimization and performance tuning. Proven experience building and operating production-grade ETL/ELT pipelines. Experience with Airflow, Dagster, or similar orchestration tools. Hands-on experience with Databricks or a comparable Spark-based platform. Solid understanding of data modeling, data warehousing, and lakehouse concepts. Experience with Parquet, Delta Lake, and/or Apache Iceberg. Familiarity with data quality, metadata, lineage, and governance practices. Experience owning production workloads, troubleshooting issues, and performing root-cause analysis. Strong analytical, problem-solving, and communication skills. Ability to work independently while collaborating effectively across teams.Skills
Sign in to read and join the discussion.