Aws Data Engineer

Job Title

AWS Data Engineer (3-4 Years Experience)

Role Overview

We are looking for a hands-on AWS Data Engineer with 34 years of experience building scalable, production-grade data platforms on AWS.

This role requires strong ownership, analytical thinking, and the ability to troubleshoot data issues end-to-end. The engineer must not only build pipelines but also deploy and manage infrastructure using Infrastructure as Code practices.

If someones idea of deployment is “I ran it locally and it worked,” this role is not for them.

Key Responsibilities

• Design, build, and maintain scalable data pipelines on AWS.
• Develop and orchestrate workflows using Astronomer / Apache Airflow.
• Build and maintain Databricks pipelines, including DAB (Databricks Asset Bundles).
• Write efficient and optimized PySpark code.
• Work with Snowflake for data modeling, transformation, and performance tuning.
• Use Terraform to provision and manage infrastructure (AWS, Databricks, Snowflake integrations where applicable).
• Use GitHub for version control, branching strategies, pull requests, and CI/CD-based deployments.
• Deploy pipelines and infrastructure using structured CI/CD workflows.
• Perform root cause analysis for data discrepancies in dashboards and reports.
• Identify missing data, anomalies, schema mismatches, and transformation issues.
• Debug issues across source systems, ingestion layers, transformations, and downstream reporting.
• Implement data quality checks and monitoring mechanisms.
• Suggest architectural improvements for scalability and cost optimization.

Required Technical Skills

AWS

Strong experience with:

• S3
• Glue
• IAM
• CloudWatch

Orchestration

• Hands-on experience with Astronomer / Apache Airflow
• DAG design, dependency management, retries, error handling

Databricks

• Experience building Databricks pipelines
• DAB (Databricks Asset Bundles)
• Workflow orchestration
• Cluster management
• PySpark transformations

Snowflake

• Data loading and transformation
• Performance tuning
• Query optimization
• Data modeling concepts

Infrastructure & Deployment

• Strong understanding of Terraform
• Creating reusable modules
• Managing environments (dev, UAT, prod)
• Remote state management
• Experience using GitHub
• PR reviews
• Branching strategies
• CI/CD integration
• Code versioning and deployment workflows

Programming & Querying

• Strong SQL
• Strong Python
• PySpark

Analytical & Problem-Solving Expectations

The candidate must be able to:

• Investigate why dashboard data is incorrect.
• Identify whether issues originate from:
• Source systems
• Ingestion logic
• Transformation layer
• Orchestration failure
• Downstream modeling
• Perform row-level validation and reconciliation.
• Detect anomalies and data quality gaps.
• Propose long-term structural fixes instead of temporary patches.

We are looking for someone who can answer:
“Why is this number wrong?”
Not someone who says, “Pipeline succeeded, so it’s fine.”

Soft Skills

• Strong written and spoken English.
• Ability to clearly communicate technical findings.
• Provide meaningful feedback in code reviews.
• Suggest better patterns and design improvements.
• Ownership mindset with attention to detail.