Senior Data Engineer - Abu Dhabi (On site)
Summary
The Senior Data Engineer will design and maintain scalable data pipelines, ETL/ELT processes, and lakehouse architectures on Azure to support healthcare analytics. The role requires expertise in Python, SQL, and big data technologies like Spark and Kafka to ensure high-performance data processing and compliance.
Excellent opportunity to work on site in Abu Dhabi.
Contract -6-12 Months, extendable
Employment Type: Full-Time
Benefits: Health insurance, Medical insurance, flight tickets etc will be taken care by the hiring company
Responsibilities:
- Design, develop, and maintain scalable data pipelines, ETL/ELT processes, data models, and schemas to process large-scale datasets for analytics and machine learning.
- Build and optimize high-performance data processing systems using technologies such as Apache Spark, Delta Lake, Kafka, and Hadoop.
- Develop real-time data pipelines using Apache Kafka, Azure Event Hub, or similar streaming technologies.
- Design and implement cloud-native data solutions on Azure ensuring scalability, reliability, performance, and cost optimization.
- Build and maintain data lakes, data warehouses, and lakehouse architectures, including data modeling, ETL processes, and data quality management.
- Implement data validation, monitoring, and testing frameworks using tools such as Great Expectations.
- Ensure data governance, security, access control, and compliance with industry standards for handling sensitive data, including Protected Health Information (PHI).
- Support healthcare data interoperability standards such as FHIR.
- Collaborate with Data Scientists, Data Analysts, Software Engineers, and Product teams to deliver reliable, high-quality data solutions.
- Continuously evaluate and adopt modern data engineering tools, technologies, and best practices.
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
- 5+ years of experience building and managing production-grade data engineering platforms and pipelines.
- Strong programming skills in Python and SQL.
- Hands-on experience with:
- Apache Spark
- Apache Kafka
- Delta Lake
- Hadoop ecosystem
- Presto/Trino
- Workflow orchestration tools (Airflow or similar)
- Docker and Kubernetes
- Experience designing scalable ETL/ELT pipelines, data models, and distributed data systems.
- Experience with cloud platforms such as Microsoft Azure
- Strong knowledge of data lakes, data warehouses, and lakehouse architecture.
- Experience with data governance, Master Data Management (MDM), data cataloging, data anonymization, and pseudonymization.
- Strong analytical, problem-solving, and communication skills.
- Ability to work effectively in a collaborative Agile environment.