Data Engineer
Posted Updated 3
views
Summary
Data engineer building and maintaining large-scale data ingestion and processing pipelines on AWS (Glue, S3, EMR, Lambda, Step Functions). Day to day involves Spark batch/streaming jobs, SQL and Python/Scala development, dimensional data modeling with SCD, Apache Iceberg lakehouse tables, DBT transformations, and Airflow orchestration.
- Develop and maintain scalable data ingestion and processing pipelines using SQL, Spark, and Python/Scala.
- Develop and optimize Apache Spark batch and streaming jobs for large-scale data processing.
- Design and implement data models including Dimension and Fact tables, with clearly defined grain, relationships, and measures.
- Implement Slowly Changing Dimensions (SCD) and appropriate incremental processing strategies.
- Work with AWS data engineering and serverless services such as Glue, S3, Lambda, EMR, and Step Functions.
- Develop and maintain reliable, scalable data pipelines with appropriate error handling, logging, monitoring, and data quality checks.
- Apply data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
- Work with Apache Iceberg, including table design, partitioning, schema evolution, and incremental processing.
- Develop data transformation workflows using DBT and orchestrate pipelines using Airflow.
- Troubleshoot and optimize data pipelines for performance, scalability, and cost efficiency.
- Participate in code reviews, testing, debugging, and continuous improvement of data engineering solutions.
Required Skills
- 4–7 years of relevant Data Engineering experience.
- Strong SQL skills — SQL is a core requirement.
- Strong hands-on experience with Apache Spark and Python or Scala.
- Hands-on experience with AWS data engineering and serverless services, particularly AWS Glue and S3.
- Good understanding of Data Modelling, including:
- Dimension and Fact table design
- Defining appropriate Fact grain
- Dimension-to-Fact relationships
- Implementing SCD Type 1 / Type 2 and other appropriate SCD patterns
- Strong understanding of data engineering fundamentals including partitioning, incremental processing, schema evolution, data quality, and pipeline reliability.
- Good understanding of distributed data processing and Spark performance optimization.
Additional Responsibilities / Good to Have
- Experience with Apache Iceberg and lakehouse architectures.
- Experience with DBT for data transformation and modelling.
- Experience with Airflow for workflow orchestration.
- Kafka / Kafka Streaming experience is a good to have.
- Exposure to AI/LLM tools for development, debugging, testing, and documentation.