Big Data Engineer
Posted Updated
Summary
Big Data Engineer in Gurugram designing and maintaining batch and streaming data pipelines and transformations using Apache Spark, Python, and Databricks on Azure, supporting high-volume production data workloads for a role requiring ~5-9 years of experience.
Key Skills: Azure Databricks, Databricks, Azure, Python, Spark
Roles and Responsibilities:
- Design and develop scalable batch and streaming data processing solutions using Spark and Python.
- Build, maintain, and enhance data pipelines and transformation workflows in Databricks and Azure Databricks.
- Develop data engineering solutions for high-volume datasets while ensuring data quality, reliability, and maintainability.
- Integrate data from multiple sources and implement robust ingestion, transformation, and validation workflows.
- Investigate pipeline issues, improve processing efficiency, and support production data workloads across Azure environments.
Skills Required:
- 6 - 9 years of relevant experience in big data engineering and cloud data platforms.
- Strong hands-on experience with Apache Spark and Python for data engineering and distributed data processing.
- Practical experience developing solutions on Microsoft Azure and Azure Databricks.
- Experience working with Databricks for data pipelines, transformations, and large-scale processing.
- Ability to design reliable data workflows and support scalable production data platforms.
Good to Have:
- Additional hands-on experience with Apache Spark-based data engineering solutions.
Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.