Principal Data Engineer: Real-Time PySpark Leader
Summary
Lead design and implementation of real-time and batch data pipelines using PySpark, Kafka, and Azure, while optimizing lakehouse storage and mentoring engineers.
Confiz is seeking a Principal Data Engineer to lead design and implementation of real-time and batch data pipelines. You will size and optimize data flows, implement lakehouse storage, and collaborate with cross-functional teams to drive data-driven decisions.
The role requires strong Python and SQL skills, cloud experience (Azure), and hands-on experience with Kafka, Spark, Airflow, and Kubernetes. You will mentor engineers and own critical data platforms.