Data Engineer - AWS Data Pipelines
Description
1. Mandatory Technical Skills - Must Have:
A. Apache Airflow - Self-Managed (Critical Requirement):
- Hands-on experience installing, configuring, deploying, and operating Apache Airflow.
- Experience managing DAGs, scheduling, dependencies, retries, monitoring, troubleshooting, and production failures.
- Understanding of Airflow infrastructure, workers, schedulers, executors, and operational maintenance.
- Experience running Airflow independently, not exclusively through managed platforms such as AWS MWAA or Astronomer.
- Important: Candidates who have only created or maintained DAGs within an existing managed Airflow environment are not an exact match.
B. AWS Data Engineering:
- Hands-on experience building and maintaining data pipelines using Amazon S3: Data storage, ingestion, and organization.
- AWS Glue: ETL jobs, Glue Data Catalog, and data transformations.
- Amazon Athena: SQL-based querying, validation, and analysis of data stored in S3.
- Candidates should demonstrate how these services were used together in actual production pipelines.
C. SQL & Python - Strong Hands-On Expertise:
- Advanced SQL skills, including joins, CTEs, window functions, aggregations, and query optimization.
- Strong Python programming skills for building ETL/ELT pipelines, transformations, automation, and data validation.
- Experience troubleshooting data issues and improving pipeline reliability.
D. Data Quality & Reconciliation - Critical Requirement:
- Must have experience ensuring the accuracy and integrity of data processed through pipelines.
- Relevant experience includes: 1. Source-to-target data reconciliation. 2. Record count and aggregate validation. 3. Identifying missing, duplicate, inconsistent, or incorrect records. 4. Investigating data mismatches and transformation errors. 5. Implementing automated data quality checks. 6. Ensuring business metrics and reporting outputs are accurate.
- Important: Candidates focused only on pipeline execution, scheduling, or infrastructure monitoring without data validation experience are not a strong match.
E. dbt (Data Build Tool):
- Hands-on experience developing SQL transformation models using dbt.
- Experience with dbt tests, dependencies, documentation, and data transformation workflows.
- Ability to demonstrate dbt usage in actual projects.
F. Terraform - Infrastructure as Code:
- Practical experience using Terraform to provision and manage cloud infrastructure.
- Experience maintaining Terraform configurations and deploying AWS resources.
- Understanding of infrastructure version control and repeatable deployments.
3. Preferred / Good-to-Have Experience:
- Experience working with healthcare, health-tech, or personal/sensitive data.
- Familiarity with data privacy, security, and compliance requirements.
- Experience handling regulated or confidential datasets.
- Experience troubleshooting and maintaining production data pipelines.
NOTE : Profiles to Avoid / Deprioritize:
- Please avoid submitting candidates whose primary experience is: 1. AI/GenAI/LLM Engineering: Primarily focused on AI agents, machine learning, or GenAI applications rather than traditional data engineering. 2. Large-Scale Big Data Engineering: Primarily focused on Spark, PySpark, Kafka, Hadoop, or streaming platforms without strong experience in the required AWS stack. 3. Managed Airflow Only: Experience limited to using AWS MWAA, Astronomer, or an already configured Airflow environment. 4. ETL Tool Operators: Experience limited to running or monitoring pipelines without developing, troubleshooting, or validating data. 5. Skills-Only Resumes: Candidates listing the required technologies without demonstrating hands-on project experience.
- Spark, Kafka, and AI experience are not disqualifying, but they should not substitute for the mandatory skills.
2. Mandatory Resume Submission Guidelines:
- Before sharing a profile with the client, please ensure: 1. Employment and Project Dates: Include start and end dates (month and year) for each role and relevant project. 2. Project-Based Technical Evidence: Clearly describe how Airflow, AWS S3, Athena, Glue, SQL, Python, dbt, and Terraform were used in actual projects. 3. Airflow Ownership: Explicitly mention whether the candidate independently deployed and managed Airflow or only worked with an existing managed environment. 4. Data Quality Experience: Highlight specific examples of data reconciliation, validation, accuracy checks, and issue resolution. 5. Previous Application Status: Confirm whether the candidate has previously applied, interviewed, or been submitted to this client.
Skills
Sign in to read and join the discussion.