Lead Data Engineer - Medallion Architecture

Were looking for a hands-on Lead Data Engineer with strong expertise in Databricks Lakehouse architecture. Youll design and optimize scalable data platforms, lead small teams, and drive innovative solutions for complex data challenges.

Role : Lead Data Engineer

Location : Bangalore (Onsite WFO 5 Days) (India)

Experience : 7 to 12 years

Certification : Databricks Certified Data Engineer Associate (mandatory); Professional preferred

What Youll Do :

- Build batch & streaming pipelines using PySpark, Spark SQL, Structured Streaming

- Implement Medallion architecture (Bronze, Silver, Gold layers)

- Work with Delta Lake features (Time Travel, Schema Evolution, MERGE, CDC, Data Quality)

- Apply Unity Catalog for governance, lineage, and security

- Optimize Spark jobs for performance and cost efficiency

- Lead and mentor engineers, review code, and advise stakeholders

- Automate deployments with CI/CD, Asset Bundles, CLI, REST APIs, Git

Must-Have Skills :

- Databricks (Workflows, Delta Live Tables, Autoloader, Unity Catalog, SQL, Asset Bundles)

- Apache Spark (PySpark, Spark SQL, Structured Streaming, performance tuning)

- Advanced Python & SQL

- AWS Databricks + S3

Nice-to-Have Skills :

- MLflow, Kafka, Airflow, Snowflake, dbt

- Experience with AdTech/MarTech data problems (identity resolution, campaign attribution, name normalization)

- Generative AI pipelines for data engineering

Soft Skills :

- Strong problem-solving & design thinking

- Excellent communication & stakeholder management

- Agile/Scrum experience