Data Engineer
We are looking for an experienced ETL / Data Engineer to design and implement robust, end-to-end data pipelines on our lakehouse platform. You will work with modern Python-based frameworks and open-source tooling to deliver high-quality, trusted data for analytics and enterprise reporting.
Key Responsibilities
Design and implement end-to-end data pipelines across a medallion architecture using Python-based frameworks.
Develop scalable ETL/ELT solutions for ingestion, validation, cleansing, and transformation of large datasets.
Ensure data quality, lineage, and auditability by integrating with metadata catalogs and validation frameworks.
Optimize data workflows for performance, reliability, and maintainability using modern open-source lakehouse tools.
Collaborate with DataOps, DevOps, and Analytics teams to ensure seamless integration of pipelines into enterprise data platforms.
Develop and maintain technical documentation, including pipeline architecture, transformation logic, and validation rules.
Monitor, troubleshoot, and continuously improve pipeline execution and data quality across environments.
Mentor junior data engineers, promoting best practices in coding, testing, and data governance.
Stay current with emerging technologies in data processing, orchestration, and lakehouse architectures, and propose continuous improvements.
Qualifications
Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, or a related field.
3–5 years of experience in data engineering or ETL development.
Strong Python programming skills, with the ability to design efficient, modular, and testable code.
Proven experience building data pipelines with DataFrame processing frameworks (e.g., Polars, Pandas, Dask, or equivalent).
Deep understanding of data lakehouse concepts, Parquet, and relational databases such as PostgreSQL and Oracle.
Experience implementing orchestration workflows (e.g., Prefect, Airflow) and data validation frameworks (e.g., Great Expectations).
Familiarity with metadata and lineage tools such as OpenMetadata or DataHub.
Solid knowledge of SQL, schema design, and data modeling.
Experience with CI/CD pipelines, version control (Git), and containerization (Docker/Kubernetes).
Understanding of event-driven automation and alerting mechanisms (e.g., n8n, SMTP, API-based notifications).
Strong analytical and problem-solving skills, with a focus on system performance and maintainability.
Excellent communication and documentation abilities.
Experience working in Agile/Scrum methodologies and collaborative development environments.