Principal Data Engineer: Real-Time PySpark Architect
Summary
Designs and optimizes real-time and batch data pipelines using PySpark, Kafka, Flink, and cloud orchestration tools like Azure Data Factory and Airflow.
Confiz is seeking a Principal Data Engineer with 5+ years of hands-on experience to design and optimize robust real-time and batch data pipelines. The role emphasizes scalable architectures, containerization, and cloud-based orchestration in an agile team.
You will build streaming pipelines with Kafka, Flink, and Spark, develop batch workloads with PySpark, and collaborate with DevOps on Terraform, Helm, Airflow, and Azure Data Factory to deliver reliable data solutions.