Data Engineer
Posted Updated
Description
As a Data Engineer, you will design, build, and operate the data pipelines, storage layers, and transformation frameworks that power analytics, reporting, and AI-driven solutions. You will work in Agile, cross-functional teams alongside data scientists, analysts, engineers, and client stakeholders to turn raw, messy, and high-volume data into clean, reliable, and query-ready assets. Your work directly enables evidence-based decisions across client engagements and internal platforms.
Key Responsibilities
- Design and build scalable batch and streaming data pipelines.
- Develop ETL and ELT workflows to extract, transform, and load data from varied sources.
- Model data for warehouse, lake, and lakehouse architectures.
- Optimize queries, partitions, and schemas for performance and cost.
- Integrate structured and unstructured data from APIs, databases, and files.
- Implement data quality checks, validation rules, and lineage tracking.
- Build and maintain data catalogues and metadata documentation.
- Support analytics, reporting, and machine-learning data needs.
- Automate pipeline orchestration, monitoring, and alerting.
- Enforce data security, access controls, and privacy compliance.
- Troubleshoot pipeline failures and drive root-cause fixes.
- Review and validate AI-generated code, with full ownership of the output.
Requirements
- Education: Degree in Computer Science, Data Engineering, Information Systems, Mathematics, or equivalent experience.
- Experience: 2+ years (mid) to 5+ or 8+ years (senior) in data engineering, ETL, or data platform roles.
- Languages: Strong proficiency in Python and SQL; Scala, Java, or Go is a plus.
- Pipelines: Experience with Airflow, DBT, Dagstter, or similar orchestration and transformation tools.
- Warehouses: Hands-on experience with Snowflake, BigQuery, Redshift, or Synapse.
- Lakehouse: Familiarity with Databricks, Delta Lake, Iceberg, or Apache Hudi.
- Streaming: Experience with Kafka, Kinesis, Pub/Sub, or Spark Streaming.
- Cloud: Experience with AWS, Azure, or Google Cloud Platform, including managed data services.
- Databases: Familiarity with relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) stores.
- AI Fluency: Comfort with Claude Code, GitHub Copilot, Cursor, or similar AI coding agents as core daily tools.
- Soft Skills: Strong problem-solving, communication, and stakeholder collaboration skills; a self-learning attitude is a must-have.
Skills
- Agile
- AI
- Airflow
- Analytics
- API
- AWS
- Azure
- BigQuery
- Cassandra
- Claude Code
- Cloud
- Data Engineering
- Data Pipelines
- Data Quality
- Databricks
- dbt
- Delta Lake
- ELT
- ETL
- GCP
- GitHub
- Github Copilot
- Java
- Kafka
- Kinesis
- Lakehouse
- Machine Learning
- MongoDB
- MySQL
- NoSQL
- PostgreSQL
- Python
- Redshift
- Scala
- Snowflake
- Spark
- SQL
Sign in to read and join the discussion.