Data Engineer
Posted Updated
Description
Responsibilities:
- Design, build, and maintain scalable ETL/ELT pipelines and data workflows on AWS.
- Develop and optimize data models (OLTP/OLAP), schemas, and warehousing solutions (e. g., Redshift, Snowflake, or similar).
- Build data ingestion frameworks for batch and streaming data from diverse sources (APIs, RDBMS, files, event streams).
- Implement data quality checks, validation rules, and monitoring for data reliability and SLAs.
- Write efficient, production-grade Python and SQL for data transformations and orchestration.
- Manage and optimize data storage, partitioning, and performance (query tuning, indexing, cost optimization).
- Implement orchestration and workflow management (e. g., Airflow/Step Functions).
- Collaborate with stakeholders to translate business requirements into technical specs and data contracts.
- Ensure data security, governance, lineage, and compliance (IAM, encryption, PII handling).
- Set up observability for pipelines (logging, metrics, alerting) and support on-call rotations.
- Contribute to CI/CD for data workloads, infrastructure-as-code, and best practices documentation.
Requirements:
- AWS: S3 Glue, Athena, Lambda, Step Functions, EMR or Glue Spark, Redshift, IAM, CloudWatch, CloudFormation/Terraform.
- Programming: Python (pandas, PySpark), SQL (advanced).
- Orchestration: Apache Airflow (or MWAA), Step Functions.
- Data Warehousing/Lakes: Redshift, Lake Formation, Parquet/Delta/Iceberg.
- Streaming (nice-to-have): Kinesis, MSK (Kafka).
- Containers (nice-to-have): Docker, ECS/EKS.
Preferred:
- 4-5 years of hands-on data engineering experience in production environments.
- Strong expertise in AWS data services and cost/performance trade-offs.
- Advanced SQL skills (window functions, CTEs, query tuning) and Python for data engineering.
- Solid experience with ETL/ELT design patterns, data modeling (star/snowflake), and schema evolution.
- Experience with Spark (Glue/EMR) or similar distributed processing frameworks.
- Proficiency with workflow orchestration (Airflow or equivalent).
- Practical knowledge of data quality frameworks and automated testing for data pipelines.
- Experience with version control (Git) and CI/CD for data jobs.
- Strong understanding of security best practices (IAM roles/policies, encryption, VPC, Secrets Manager).
Skills
- Airflow
- API
- Athena
- AWS
- CI/CD
- CloudFormation
- CloudWatch
- Data Engineering
- Data Ingestion
- Data Modeling
- Data Pipelines
- Data Quality
- Data Warehousing
- Design Patterns
- Docker
- ECS
- EKS
- ELT
- ETL
- Git
- IAM
- Infrastructure as Code
- Kafka
- Kinesis
- Lambda
- Observability
- OLAP
- pandas
- Parquet
- PySpark
- Python
- RDBMS
- Redshift
- S3
- Snowflake
- Spark
- SQL
- Terraform
- Test Automation
- Version Control
- VPC
- Workflow Orchestration
Sign in to read and join the discussion.