Lead Data Engineer

Role Summary

Technical leader responsible for the modernization and future-state architecture of the Sales Validation platform.

Responsibilities

Lead Databricks architecture and delivery

Drive Synapse-to-Databricks migration

Establish engineering standards and reusable frameworks

Mentor engineers and conduct design reviews

Own technical delivery across major workstreams

Drive performance optimization and scalability

Required Skills

Databricks (Highest Priority)

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Databricks Workflows

Databricks Asset Bundles

Lakehouse Architecture

Azure Platform

Azure Data Factory (ADF)

ADLS Gen2

Azure SQL

Azure Security Patterns

Migration Experience

Synapse Pipelines

Synapse Spark

Synapse SQL Pools

Synapse Notebooks

Synapse Data Flows

Modernization of legacy ETL workloads

Engineering

Advanced SQL

Data Modeling

GitLab CI/CD

Performance Tuning

Code Reviews

GenAI

Hands-on experience with GenAI coding assistants (Claude Code, GitHub Copilot, or similar) for AI-assisted development.

Prompt engineering skills writing clear instructions and feedback to guide AI toward desired output.

Ability to critically review and validate AI-generated code and content before use.

Basic understanding of LLM concepts — RAG, context windows, hallucinations, and responsible AI practices.

Preferred

Apache Airflow

Snowflake

Data Governance

Enterprise-scale Azure platform implementation