Engineering Lead/Azure Solution Architect
Summary
Leads a team designing and scaling Azure-based big data platforms using Databricks, PySpark, Kafka, and Synapse to process petabyte-scale data for enterprise analytics.
Senior Azure Solution Architect/Engineering Lead
Core Skills\: Azure Databricks, PySpark, Spark, SQL, Azure Synapse, Kafka, Big Data, Python, Java, Azure Data Factory (ADF)
Professional Summary
Senior Data Architect with 15+ years of experience designing, building, and scaling enterprise-grade data platforms. Proven expertise in cloud-native big data architectures on Azure, real-time and batch data processing, and high-performance analytics systems. Strong background in data modeling, distributed computing, and end-to-end data pipeline orchestration, with hands-on leadership in large-scale enterprise and mission-critical projects.
Technical Expertise
Cloud & Data Platforms
- Azure Databricks (Lakehouse architecture, Delta Lake, Unity Catalog)
- Azure Synapse Analytics (Dedicated & Serverless Pools)
- Azure Data Factory (ADF) for ELT/ETL orchestration
- Azure Data Lake Storage Gen2
Big Data & Distributed Processing
- Apache Spark (Core, SQL, Structured Streaming)
- PySpark performance tuning & optimization
- Apache Kafka (real-time ingestion, streaming pipelines)
- Large-scale batch and streaming data architectures
Programming & Query Languages
- Python (advanced data engineering, automation, frameworks)
- Java (Spark internals, Kafka consumers/producers, microservices)
- SQL (complex analytics, query optimization, warehouse design)
Data Architecture & Design
- Lakehouse, Lambda, and Kappa architectures
- Dimensional modeling & data warehouse design
- Data governance, security, and lineage
- Scalability, fault tolerance, and cost optimization
· Technical depth in designing and building data platforms for high volume data processing with low latency
· Strong Expertise in Azure, Databricks, Python, Pyspark, SparkSQL, CI-CD (Jenkins, Github), Obs (Dynatrace),
· Good to have \: Java, Kafka , Casandra, Microservices, API, Flink, ZeroMQ, Protocol Buffers
· Experience in recruiting , people leadership for managing team of 40+ , govern delivery and also act as technical mentor to set processes to bring innovation, goals to derive business outcomes
· Act as Module lead for India operations, liaise with NA teams to drive the objectives
Key Responsibilities & Achievements
- Architected and delivered enterprise-scale Azure Lakehouse platforms supporting petabyte-scale data.
- Designed real-time streaming solutions using Kafka and Spark Structured Streaming.
- Led migration of on-premise data warehouses to Azure Synapse + Databricks, improving performance and reducing cost.
- Built reusable PySpark frameworks for data ingestion, transformation, and validation.
- Optimized Spark jobs through partitioning, caching, and memory tuning to achieve significant runtime improvements.
- Implemented end-to-end ADF pipelines with CI/CD and parameter-driven orchestration.
- Mentored senior and junior engineers; provided architectural guidance across multiple teams.
- Collaborated with business stakeholders to translate analytics requirements into scalable technical designs.
Leadership & Soft Skills
- Technical leadership and architectural decision-making
- Cross-team collaboration and stakeholder communication
- Design reviews, code quality standards, and best practices
- Agile/Scrum execution in large enterprise environments
· Functional experience of capital markets, securities processing will be advantage