Senior Data Engineer (Permanent)
Description
About the Role
We are seeking an experienced Senior Data Engineer to design, develop and maintain large-scale data processing solutions for aviation systems. You will build reliable batch and real-time data pipelines, integrate multiple data sources and implement scalable data storage, processing and recovery mechanisms.
Key Responsibilities
Design, develop and maintain batch and real-time data pipelines using Apache Spark, Apache Flink and Apache Kafka.
Implement distributed data processing solutions using Java or Scala and SQL.
Integrate multiple data sources and S3-compatible object storage for data ingestion and processing.
Manage data schemas, metadata, lineage and data quality across processing pipelines.
Deploy, maintain and scale data processing workloads in Kubernetes environments.
Implement automated testing, GitLab CI/CD pipelines, monitoring and operational troubleshooting.
Establish checkpointing, recovery, data retention and lifecycle management mechanisms.
Support high availability, backup, disaster recovery and data synchronisation across multiple sites.
Requirements and Qualifications
5–8 years of relevant experience in data engineering, distributed systems or large-scale data platform development.
Strong hands-on experience designing and developing batch and streaming data pipelines using Apache Spark, Apache Flink and Apache Kafka.
Proficiency in Java or Scala for distributed data processing, together with strong SQL skills.
Experience deploying and scaling data workloads using Docker and Kubernetes.
Familiarity with S3-compatible object storage, data schemas, metadata management, data lineage and data quality controls.
Experience with checkpointing, fault recovery, data retention and disaster recovery mechanisms.
Hands-on experience with GitLab, CI/CD pipelines, automated testing and monitoring.
Strong analytical, troubleshooting and problem-solving skills.
Only shortlisted candidates will be notified.
(EA Reg No: 20C0312)
Skills
Sign in to read and join the discussion.