Lead Data Engineer
Summary
Lead a team building and optimizing large-scale data pipelines, data lakes, and warehouses using Azure, Spark, Kafka, and Snowflake to power analytics and ML for Emirates’ enterprise systems.
Role - Big Data / Lead Data Engineer
We are seeking an experienced Senior Developer - Big Data / Lead Data Engineer to design, develop, and optimize large-scale data engineering solutions within Emirates' enterprise analytics ecosystem. The ideal candidate will have strong expertise in building end-to-end data pipelines, Data Lakes, Data Warehouses, and real-time streaming solutions using modern Azure and Big Data technologies.
The candidate will work closely with Data Architects, Data Engineers, BI Developers, and Business Stakeholders to deliver scalable, secure, and high-performing data platforms that support enterprise reporting, analytics, and machine learning initiatives.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines for Data Lake, Data Warehouse, Business Intelligence, and Analytics platforms.
- Translate business and technical requirements into robust data engineering solutions and technical designs.
- Develop ingestion, transformation, and data modeling frameworks for structured and unstructured data.
- Build and optimize batch and real-time data processing pipelines.
- Ensure solutions meet performance, scalability, security, and data governance standards.
- Troubleshoot complex production issues and provide timely resolutions.
- Conduct code reviews and enforce coding standards and best practices.
- Perform unit testing, defect fixing, and deployment activities.
- Collaborate with architects and cross-functional teams on solution design and implementation.
- Maintain metadata, data lineage, and data inventory documentation.
- Contribute to data engineering standards, frameworks, playbooks, and reusable components.
- Support Agile delivery processes including sprint planning, estimation, and retrospectives.
Required Technical Skills
Big Data Technologies
- Hadoop
- Spark
- Scala
- Hive
- HBase
- Sqoop
- Oozie
- HDFS
- Apache NiFi
- Airflow
- Kafka
- Spark Streaming
- Elasticsearch
Cloud & Data Engineering
- Microsoft Azure
- Azure Data Factory (ADF)
- Azure Databricks
- Azure Data Lake Storage Gen2 (ADLS Gen2)
- Snowflake
- Cloudera Data Platform
Integration Technologies
- SnapLogic
- TIBCO
Data Warehousing & BI
- SQL
- Power BI
- MicroStrategy
- Dimensional Data Modeling
- Data Warehouse & Data Lake Architecture
DevOps & Source Control
- Git
- Bitbucket
- Azure DevOps
- CI/CD Pipelines
File Formats & Data Processing
- AVRO
- PARQUET
- Structured and Unstructured Data Processing
Required Experience
- 8+ years of hands-on experience in Data Engineering.
- Strong experience designing and building enterprise-scale data pipelines.
- Experience with Data Lakes, Data Warehouses, and Analytics platforms.
- Expertise in real-time streaming and batch data processing.
- Strong SQL and data modeling skills.
- Experience working with large-scale datasets and distributed computing frameworks.
- Experience supporting production environments and incident resolution.
- Exposure to Data Governance, Data Lineage, and Data Quality frameworks.
- Experience working in Agile and Waterfall delivery models.
Preferred Qualifications
- Experience in Airline, Travel, Transportation, Banking, or Enterprise Analytics domains.
- Azure, Databricks, Snowflake, or Cloudera certifications.
- Experience supporting Machine Learning and Advanced Analytics platforms.
- Strong communication and stakeholder management skills.
Essential Traits
- Strong analytical and problem-solving abilities.
- Ability to work independently with minimal supervision.
- Team player with excellent collaboration skills.
- Adaptable and willing to learn new technologies.
- Focus on quality, performance optimization, and continuous improvement.