Data Engineer (Cloud Data Pipelines & Data Quality)
Posted 7
views
Summary
Design and optimize cloud-based data pipelines using SQL, Python, and Airflow. The role involves implementing automated data quality checks, developing reliable ingestion patterns, and troubleshooting production incidents to ensure data reliability and performance.
Willing to start a new freelance challange? Get in touch.
What You'll Do
- Design, build and optimise cloud-based data pipelines using SQL, Python/PySpark and Airflow or a similar orchestration platform.
- Implement automated data-quality checks covering completeness, thresholds, referential integrity and business rules.
- Develop reliable ingestion patterns, including idempotent and merge-based processing.
- Investigate failed pipelines, data-quality issues and production incidents, identifying and implementing effective fixes.
- Improve pipeline performance, schemas and processing logic to reduce unnecessary compute and improve maintainability.
- Define and implement data contracts where appropriate to improve reliability between data producers and consumers.
- Collaborate with Data Engineers and business/technical stakeholders to understand requirements and deliver robust data solutions.
- Produce clear technical documentation and contribute to knowledge-transfer sessions to ensure solutions remain maintainable after handover.
What you bring
- 3+ years of professional experience in Data Engineering, with hands-on experience building cloud data pipelines.
- Strong SQL skills and professional experience with Python and/or PySpark.
- Practical experience with Apache Airflow or a comparable workflow orchestration tool.
- Experience implementing idempotent, incremental or merge-based data ingestion patterns.
- Understanding of automated data-quality testing, ideally with tools such as Great Expectations, Soda or equivalent.
- Experience troubleshooting pipeline failures and improving data-processing performance.
- A strong engineering mindset around reliability, documentation, testing and maintainability.
- Comfortable collaborating with other Data Engineers and sharing knowledge.
Interested? Apply today!
myNEBIRU: Not into this role, but interested in what NEBIRU does? That’s totally fine.
Visit https://nebiru.be/mynebiru to see how we can support you - even outside our client missions. Let’s build the bridge to your next step, together.