Data Engineer
Posted Updated
Description
Responsibilities
• Design, build, and optimize ETL/ELT pipelines for batch and streaming data
• Develop and maintain data warehouses and data lakes (e.g., Snowflake, BigQuery, Redshift)
• Collaborate with data scientists, analysts, and product teams to understand data needs
• Ensure data quality, reliability, and observability across pipelines
• Build and manage orchestration workflows (e.g., Airflow, Prefect, Dagster)
• Implement best practices for data modeling, partitioning, and performance tuning
• Monitor and troubleshoot data infrastructure issues in production
Requirements
• 5+ years of experience in data engineering or a related role
• Proficiency in SQL and Python (or Scala/Java)
• Experience with cloud platforms (AWS, GCP, or Azure)
• Hands-on with big data tools: Spark, Kafka, dbt, or similar
• Familiarity with data warehousing concepts and dimensional modeling
• Strong understanding of CI/CD and version control (Git)
Nice to Have
• Experience with real-time/streaming pipelines (Kafka, Flink, Kinesis)
• Exposure to ML pipelines or MLOps tooling
• Knowledge of data governance and data catalog tools (e.g., Atlan, Alation)
• Design, build, and optimize ETL/ELT pipelines for batch and streaming data
• Develop and maintain data warehouses and data lakes (e.g., Snowflake, BigQuery, Redshift)
• Collaborate with data scientists, analysts, and product teams to understand data needs
• Ensure data quality, reliability, and observability across pipelines
• Build and manage orchestration workflows (e.g., Airflow, Prefect, Dagster)
• Implement best practices for data modeling, partitioning, and performance tuning
• Monitor and troubleshoot data infrastructure issues in production
Requirements
• 5+ years of experience in data engineering or a related role
• Proficiency in SQL and Python (or Scala/Java)
• Experience with cloud platforms (AWS, GCP, or Azure)
• Hands-on with big data tools: Spark, Kafka, dbt, or similar
• Familiarity with data warehousing concepts and dimensional modeling
• Strong understanding of CI/CD and version control (Git)
Nice to Have
• Experience with real-time/streaming pipelines (Kafka, Flink, Kinesis)
• Exposure to ML pipelines or MLOps tooling
• Knowledge of data governance and data catalog tools (e.g., Atlan, Alation)
Skills
- Airflow
- AWS
- Azure
- Big Data
- BigQuery
- CI/CD
- Cloud
- Dagster
- Data Engineering
- Data Governance
- Data Lake
- Data Modeling
- Data Quality
- Data Warehousing
- dbt
- Dimensional Modeling
- ELT
- ETL
- Flink
- GCP
- Git
- Java
- Kafka
- Kinesis
- Machine Learning
- MLOps
- Observability
- Prefect
- Python
- Redshift
- Scala
- Snowflake
- Spark
- SQL
- Version Control