Senior Azure Data Engineer
Description
Dexian India is looking for experienced Azure Data Engineers (5 - 8 years) with strong hands-on expertise in building scalable, cloud-based data engineering solutions.
Work mode: Hybrid (4 days WFO, 1 day remote)
Shift: 1.30 - 10.30PM IST
About the position
Design, build, test, deploy, and support scalable data pipelines and trusted data products for our business and engineering use cases. The Data Engineer works across source systems, integration services, cloud data platforms, analytics teams, and governance functions to deliver secure, reliable, observable, and reusable data solutions on Microsoft Azure.
Key responsibilities
Data pipeline engineering
- Design, develop, and maintain batch and streaming data pipelines using Azure Data Factory, Azure Databricks, Apache Spark, Event Hubs, or approved enterprise services.
- Ingest data from relational databases, APIs, files, SaaS applications, operational systems, IoT sources, and third-party platforms.
- Build reusable ingestion, transformation, validation, enrichment, and publishing frameworks using Python, SQL, PySpark, and Delta Lake patterns.
- Implement incremental loading, change data capture, partitioning, schema evolution, retry, reconciliation, and error-handling mechanisms.
- Develop curated data products that are accurate, documented, discoverable, and suitable for analytics, reporting, data science, and operational consumption.
Cloud data platform and architecture
- Engineer solutions using Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Azure SQL, Microsoft Fabric, or comparable approved platforms.
- Apply medallion or equivalent layered architecture patterns across raw, validated, curated, and serving data zones.
- Optimize workload performance, storage layout, compute utilization, query efficiency, and cloud cost.
- Collaborate with data architects to implement data models, integration patterns, metadata standards, and reference architectures.
- Support migration and modernization of legacy ETL, data warehouse, and reporting workloads to cloud-native platforms.
- Data quality, governance, and security - Implement automated data-quality checks for completeness, validity, uniqueness, consistency, timeliness, and referential integrity.
- Create lineage, metadata, data dictionaries, ownership, classification, and retention information using Microsoft Purview or approved governance tools.
- Apply identity and access controls, managed identity, encryption, secrets management, masking, and least-privilege principles.
- Ensure engineering solutions comply with cybersecurity, privacy, records-management, and information-protection requirements.
- Partner with data owners and stewards to resolve quality issues and establish measurable data-quality service levels.
DevOps, reliability, and collaboration
- Build CI/CD pipelines in Azure DevOps or GitHub Actions for notebooks, code, infrastructure, configuration, and data-pipeline deployments.
- Use Git, automated tests, code reviews, infrastructure as code, environment promotion, and controlled rollback practices.
- Implement logging, metrics, alerts, pipeline monitoring, data observability, and operational runbooks; troubleshoot failures and complete root-cause analysis.
- Work with Product Owners, analysts, data scientists, architects, source-system teams, security, and operations to translate requirements into production solutions.
- Provide estimates, technical documentation, knowledge transfer, and support for releases and production incidents.
Required qualifications
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, Engineering, or a related field; equivalent experience may be considered.
- 5-8 years of technology experience, including at least 3 years in data engineering, ETL/ELT, or cloud data-platform development.
- Strong hands-on experience with Python, SQL, Spark/PySpark, data modeling, and distributed data processing.
- Experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure SQL, Synapse Analytics, or Microsoft Fabric.
- Experience designing batch and/or streaming pipelines, dimensional or lakehouse models, and reusable data-integration frameworks.
- Knowledge of data-quality controls, metadata, lineage, security, access management, and data-governance practices.
- Experience with Git, CI/CD, automated testing, Azure DevOps, monitoring, and production support.
- Strong analytical, troubleshooting, documentation, communication, and cross-team collaboration skills.
Preferred qualifications
- Microsoft Certified: Azure Data Engineer Associate, Azure Databricks certification, or Microsoft Fabric
Skills
- Analytics
- API
- Azure
- Azure Data Factory
- Azure DevOps
- Azure Synapse
- CI/CD
- Cloud
- Cloud Native
- Cybersecurity
- Data Engineering
- Data Governance
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Science
- Data Warehousing
- Databricks
- Delta Lake
- DevOps
- ELT
- ETL
- Git
- GitHub
- GitHub Actions
- Infrastructure as Code
- IoT
- Lakehouse
- Microsoft Fabric
- Microsoft Purview
- Observability
- PySpark
- Python
- RDBMS
- Root Cause Analysis
- SaaS
- Secrets Management
- Spark
- SQL
- Test Automation
Sign in to read and join the discussion.