Sr Azure Data Engineer - Hyderabad
Summary
Senior Data Engineer in Hyderabad (on-site) who designs and runs end-to-end batch and streaming data pipelines using PySpark/Apache Spark on Databricks with Delta Lake, plus Azure data services like Data Factory, ADLS Gen2 and Event Hubs, and advanced SQL data modeling. Full-time, permanent role for candidates with 8-14 years of experience.
Job Title: Senior Data Engineer
Employment Type: Full Time, Permanent
Position Location: Hyderabad
Qualifications: BE/B.Tech/MCA Degree in Computer Science, Engineering, or similar relevant field
Total Experience: 8 to 14 years
Working Model: Work from office
• Design and implement end-to-end batch and streaming data pipelines using PySpark and Apache Spark.
• Build and manage data workflows on Databricks, including Delta Lake architecture and medallion (Bronze/Silver/Gold) patterns.
• Develop and maintain ETL/ELT processes to ingest, transform, and load data from diverse sources.
• Optimize Spark jobs for performance, cost-efficiency, and reliability at scale.
• Manage data solutions using Azure services including Azure Data Lake Storage (ADLS Gen2), Azure Event Hubs, and Azure Key Vault.
• Ensure data security, access control, and compliance within Azure environments.
• Monitor and troubleshoot cloud infrastructure, pipelines, and data flows.
• Design and implement dimensional and normalized data models for analytical and operational use cases.
• Write complex SQL queries for data transformation, validation, and reporting.
• Work with structured, semi-structured, and unstructured data across various storage formats (Parquet, Delta, JSON, CSV).
• 6+ years of hands-on experience in data engineering roles.
• Strong expertise in Apache Spark and PySpark for large-scale data processing.
• Deep experience with Databricks platform — workspace setup, clusters, jobs, Delta Live Tables.
• Strong experience in building Azure Data Factory Pipeline.
• Proficiency with Azure cloud services for data engineering workloads.
• Advanced SQL skills including window functions, CTEs, and query optimization.
• Solid Python programming skills for data engineering, scripting, and automation.
• Experience with version control (Git) and CI/CD pipelines for data code deployments.
• Familiarity with data governance, data lineage, and metadata management.
• Azure Synapse & Azure EventHub are good to have.