Senior Data Engineer
Posted Updated
Description
We are seeking a Senior Data Engineer with 4-5 years of experience in designing and building scalable data pipelines on AWS. The ideal candidate should have strong expertise in Python, SQL, ETL/ELT, data warehousing, and workflow orchestration.
Responsibilities:
- Design and maintain scalable ETL/ELT pipelines on AWS.
- Build and optimise data warehouses and data models.
- Develop data ingestion pipelines for batch and streaming data.
- Write efficient Python and advanced SQL for data processing.
- Implement orchestration using Airflow or AWS Step Functions.
- Ensure data quality, monitoring, security, and governance.
- Optimise performance, storage, and cloud costs.
- Support CI/CD, Infrastructure as Code, and production deployments.
Requirements:
- AWS: S3 Glue, Athena, Lambda, Redshift, EMR/Glue Spark, Step Functions, IAM, CloudWatch, CloudFormation/Terraform.
- Programming: Python (Pandas, PySpark), Advanced SQL.
- Orchestration: Apache Airflow or AWS Step Functions.
- Data: ETL/ELT, Data Warehousing, Data Modelling (Star/Snowflake), Parquet, Delta/Iceberg.
- Version Control: Git, CI/CD.
- Nice to Have: Kinesis, Kafka (MSK), Docker, ECS/EKS, Snowflake, Lake Formation.
Skills
- Airflow
- Athena
- AWS
- CI/CD
- Cloud
- CloudFormation
- CloudWatch
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Docker
- ECS
- EKS
- ELT
- ETL
- Git
- IAM
- Infrastructure as Code
- Kafka
- Kinesis
- Lambda
- pandas
- Parquet
- PySpark
- Python
- Redshift
- S3
- Snowflake
- Spark
- SQL
- AWS Step Functions
- Terraform
- Version Control
- Workflow Orchestration
Sign in to read and join the discussion.