Principal Data Engineer – PySpark & Real-Time Pipelines
Summary
Designs and maintains real-time and batch data pipelines using PySpark, Kafka, Flink, and Azure Data Factory, while mentoring a data engineering team.
Confiz Limited is seeking a Principal Data Engineer with 5+ years of experience in real-time and batch data processing. This role involves designing and maintaining robust data pipelines using technologies like Kafka, Apache Flink, and Spark, alongside cloud orchestration with Azure Data Factory.
The ideal candidate will also mentor a data engineering team, focusing on data quality and efficient code practices. A Bachelor's degree in computer science is required, and expertise in cloud platforms, particularly Azure, is preferred.