Azure AI Data Engineer
Summary
Designs and builds scalable Azure-based data pipelines and Lakehouse platforms for enterprise analytics and AI/ML workloads using Databricks, Delta Lake, and Azure Data Factory.
Project description
Designed and implemented a scalable enterprise Lakehouse platform on Azure using Azure Data Factory, Delta Lake, ADLS Gen2, and Databricks. Migrated legacy ETL processes to cloud-native ELT pipelines supporting both batch and near real-time workloads. Established reusable data frameworks, metadata-driven orchestration, data quality validation, and automated CI/CD pipelines.
Responsibilities
- Designed and implemented scalable data pipelines using Azure Data Factory, Databricks, Delta Lake, Python, and SQL. Architected Lakehouse solutions supporting enterprise analytics, AI/ML, and Generative AI workloads. Developed structured and unstructured data pipelines for RAG, vector search, and AI-powered applications. Built reusable frameworks, improving code quality, scalability, and development productivity. Optimized ETL/ELT processes, reducing data latency and improving performance. Implemented CI/CD, automated testing, monitoring, and DataOps best practices. Established data quality, governance, security, and metadata management standards. Collaborated with cross-functional teams to deliver business-driven data and AI solutions. Mentored engineers and led technical design and architecture discussions. Drove cloud modernization initiatives, improving reliability, scalability, and operational efficiency.
SKILLS
Must have
- 5+ years of experience Azure Data Factory (ADF) Databricks Delta Lake Python SQL / PL-SQL PySpark Azure Data Lake (ADLS) CI/CD (Azure DevOps) Data Modeling ETL/ELT Pipelines
Nice to have
Azure AI Search Microsoft Fabric Kafka Snowflake Terraform Docker / Kubernetes Microsoft Purview