Data Engineer Lead

About the role

Lead the transformation of legacy ETL pipelines into modular, high-performance solutions using Databricks, dbt, Fivetran, and Airflow. You will architect reusable data design patterns, develop and lead POCs/POVs to validate new technologies, and leverage AI-enabled tools to accelerate engineering. This role involves designing and implementing complex data pipelines, modernizing SSIS/SSRS workloads into cloud-native architectures, and optimizing performance and cost across Azure and GCP environments.

Key responsibilities

• Modernize legacy ETL pipelines: Lead the transformation of SSIS/SSRS workloads into modular, high-performance pipelines using Databricks, dbt, Fivetran, and Airflow
• Architect reusable data design patterns: Define and implement standardized frameworks for ingestion, transformation, curation, and consumption layers across the Lakehouse
• Develop and lead POCs/POVs: Experiment with new technologies (e.g., Delta Live Tables, Iceberg, streaming ingestion, AI-driven observability) to validate architecture choices and influence the enterprise roadmap
• Leverage AI to accelerate engineering: Use AI-enabled tools like Databricks Assistant, Cursor AI, GitHub Copilot, and dbt Mesh AI tests for code generation, automated testing, documentation, and pipeline optimization
• Apply ML for operational intelligence: Integrate predictive models to detect pipeline anomalies, data drift, and optimize compute and scheduling
• Enforce engineering excellence: Drive CI/CD, version control, peer reviews, and observability practices across the data platform
• Collaborate cross-functionally: Partner with data architects, platform engineers, analysts, and business product owners to translate business needs into technical solutions
• Mentor data engineers: Provide technical guidance, foster continuous learning, and help the team adopt modern data engineering best practices
• Optimize performance and cost: Continuously tune Spark workloads, storage tiers, and orchestration logic across Azure and GCP environments
• Administer and test the reliability of data systems; monitor performance and stability as requirements change

About you

• Bachelor's degree
• 8+ years of experience in data engineering or related technical fields, with at least 3+ years in a lead or senior role
• Proven experience designing and implementing data design patterns (e.g., CDC, SCD, Medallion, Data Vault, streaming, and batch patterns)
• Deep expertise with Databricks, Apache Spark, dbt, Fivetran, Census, Airflow, and Kafka
• Solid experience across Azure and/or GCP (e.g., Synapse, Data Factory, BigQuery, Pub/Sub)
• Hands-on experience modernizing legacy ETL (SSIS/SSRS) workloads into cloud-native pipelines
• Demonstrated ability to build POCs and POVs that validate new tools, frameworks, or architectures
• Working knowledge of AI-assisted engineering tools for development, observability, or optimization
• Proficiency in SQL and one programming language (Python, Scala, or Java)
• Strong problem-solving, architectural thinking, and collaboration skills