Senior Data Engineer

Role Overview : Excellent opportunity to work on site in Abu Dhabi.

Contract : 6-12 Months, extendable

Employment Type : Full-Time

Benefits : Health insurance, Medical insurance, flight tickets etc will be taken care by the hiring company

Responsibilities :

- Design, develop, and maintain scalable data pipelines, ETL/ELT processes, data models, and schemas to process large-scale datasets for analytics and machine learning.

- Build and optimize high-performance data processing systems using technologies such as Apache Spark, Delta Lake, Kafka, and Hadoop.

- Develop real-time data pipelines using Apache Kafka, Azure Event Hub, or similar streaming technologies.

- Design and implement cloud-native data solutions on Azure ensuring scalability, reliability, performance, and cost optimization.

- Build and maintain data lakes, data warehouses, and lakehouse architectures, including data modeling, ETL processes, and data quality management.

- Implement data validation, monitoring, and testing frameworks using tools such as Great Expectations.

- Ensure data governance, security, access control, and compliance with industry standards for handling sensitive data, including Protected Health Information (PHI).

- Support healthcare data interoperability standards such as FHIR.

- Collaborate with Data Scientists, Data Analysts, Software Engineers, and Product teams to deliver reliable, high-quality data solutions.

- Continuously evaluate and adopt modern data engineering tools, technologies, and best practices.

Required Qualifications :

- Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.

- 5+ years of experience building and managing production-grade data engineering platforms and pipelines.

- Strong programming skills in Python and SQL.

- Hands-on experience with :

1. Apache Spark

2. Apache Kafka

3. Delta Lake

4. Hadoop ecosystem

5. Presto/Trino

6. Workflow orchestration tools (Airflow or similar)

7. Docker and Kubernetes

- Experience designing scalable ETL/ELT pipelines, data models, and distributed data systems.

- Experience with cloud platforms such as Microsoft Azure.

- Strong knowledge of data lakes, data warehouses, and lakehouse architecture.

- Experience with data governance, Master Data Management (MDM), data cataloging, data anonymization, and pseudonymization.

- Strong analytical, problem-solving, and communication skills.

- Ability to work effectively in a collaborative Agile environment.