Principal Data Engineer (Databricks)
Summary
Designs and builds scalable ETL pipelines on Databricks, using PySpark, Delta Lake, and Medallion Architecture to process and optimize large-scale data workflows.
Required CertificationsDatabricks Certified Data Engineer Professional
Role requirements
7+ years of experience building highly reliable ETL pipelines and managing complex data workflows. Elite-level proficiency with Apache Spark, Delta Lake architecture, and deploying optimized data infrastructure in production environments.
Technical requirements
- Core Architecture: Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Unity Catalog
- Engines: PySpark, Spark SQL, Photon Engine, Delta Live Tables (DLT)
- Orchestration: Databricks Workflows, dbt (Data Build Tool), Airflow
- Performance: Z-Ordering, Liquid Clustering, Data Skipping, and Spark UI Debugging
Focus areas
DatabricksSparkDelta Lake