AWS & Airflow Developer

Role & responsibilities

• Develop data transformation logic using Python, SQL and PySpark where applicable.
• Design, develop and maintain scalable and reliable data pipelines using Apache Airflow and AWS services.
• Build and optimize ETL/ELT workflows to ingest, transform and process data from various source systems.
• Develop and manage Airflow DAGs, task dependencies, scheduling, monitoring and error handling.
• Implement data ingestion solutions using AWS services such as S3, Glue, Lambda, Step Functions, EventBridge, Athena and RDS/Aurora.
• Monitor data pipeline performance and troubleshoot workflow failures, data quality issues and operational incidents.
• Collaborate with architects, business analysts and application teams to understand data requirements and deliver scalable solutions.
• Support production deployments, incident management, root cause analysis and performance tuning activities.
• Implement data validation, reconciliation and quality checks to ensure accurate and reliable data delivery.
• Participate in cloud modernization and migration initiatives to AWS-based platforms.
• Contribute to CI/CD implementation, automation, documentation and engineering best practices.
• Develop and maintain operational dashboards, monitoring and alerting solutions for pipeline observability.
• Participate in Agile ceremonies, sprint planning and technical design discussions.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 4-6 years of experience in Data Engineering, ETL Development, or Data Integration.
• Strong proficiency in Python programming.
• Strong hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Experience with AWS cloud services, including:
• Amazon S3, Glue, Lambda, Step Functions, Athena, EventBridge, RDS/Aurora PostgreSQL, CloudWatch
• Strong SQL skills with experience in relational databases such as PostgreSQL, Oracle, or SQL Server.
• Experience developing and supporting batch and event-driven data pipelines.
• Understanding of data modeling, data warehousing and ETL/ELT concepts.
• Experience with source control systems such as Git.
• Knowledge of monitoring, logging and troubleshooting distributed data workflows.
• Strong analytical, problem-solving and debugging skills.
• Excellent verbal and written communication skills.

Preferred candidate profile

• Experience with PySpark and distributed data processing frameworks.
• Knowledge of AWS EMR, Redshift, or Snowflake.
• Experience working with Docker and containerized applications.
• Exposure to Kubernetes or Amazon EKS.
• Familiarity with REST API integration and event-driven architectures.
• Experience implementing CI/CD pipelines using GitHub Actions, GitLab CI, or AWS CodePipeline.
• Understanding of data governance, metadata management and data lineage concepts.
• Experience working in Agile/Scrum environments.
• Exposure to infrastructure-as-code tools such as Terraform or AWS CloudFormation.