Senior Data Engineer - Databricks & AI Solutions

Role Overview:

Seeking a highly skilled Senior Data Engineer with deep expertise in Databricks, Real-Time Data Processing, Lakehouse Architecture, and AI-driven solutions. The role involves designing, developing, and supporting scalable data platforms and streaming pipelines, leveraging modern Databricks capabilities such as Lakeflow, Unity Catalog, and Generative AI technologies.

Key Responsibilities:

- Design, develop, and maintain end-to-end streaming and batch data pipelines using Databricks and PySpark.

- Build, optimize, and manage Delta Lake tables within Unity Catalog.

- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validations.

- Integrate enterprise systems into the Databricks Lakehouse.

- Ensure data quality, governance, security, and operational excellence across platforms.

- Monitor, troubleshoot, and optimize workflows for reliability and performance.

- Collaborate with stakeholders to deliver scalable, business-aligned solutions.

- Build and support AI-powered data solutions leveraging LLMs, RAG, and vector search.

- Implement CI/CD best practices and maintain deployment pipelines.

- Drive deliverables independently, proactively resolving challenges with minimal supervision.

Tech Stack & Requirements :

- Hands-on experience with Databricks and modern Lakehouse architecture.

- Expertise in PySpark, Spark Structured Streaming, SQL, NoSQL databases, Delta Lake.

- Proficiency with Unity Catalog for governance and security.

- Experience building AI agents, copilots, and generative AI solutions using LLMs, RAG, and vector search.

- Experience with Git-based source control, CI/CD pipelines, Databricks Repos, and Databricks Asset Bundles.

Educational Qualifications :

- BE/BTech/ME/MTech/MCA only.