Senior Data Engineer
Posted
Summary
Senior Data Engineer at an IT services firm in Mumbai/Thane (hybrid), designing scalable data models and architectures, building ETL pipelines, and orchestrating large-scale data workflows using Apache Airflow, Databricks, PySpark, and Spark SQL.
Senior Data Engineer
Summary
We are seeking an experienced Senior Data Engineer to join our IT Services team, where you will play a pivotal role in designing, building, and maintaining robust data infrastructure. This position is critical for ensuring the integrity, efficiency, and scalability of our data ecosystems, enabling advanced analytics and business intelligence initiatives. The ideal candidate will leverage their deep expertise in data architecture and engineering to drive high-quality data solutions that support organizational goals.
Responsibilities
- Design and implement scalable data models and comprehensive data architectures to support enterprise needs.
- Establish and enforce rigorous data validation protocols to ensure high data quality and accuracy across all systems.
- Develop sophisticated error handling mechanisms and logging frameworks to monitor pipeline health and troubleshoot issues efficiently.
- Orchestrate complex data workflows and scheduling using tools such as Apache Airflow, Tidal, or Control-M.
- Manage version control and automate deployment processes through Git and continuous integration/continuous deployment (CI/CD) pipelines.
- Build and optimize Extract, Transform, Load (ETL) processes to facilitate seamless data movement and transformation.
- Utilize Databricks, PySpark, and Spark SQL to process large-scale datasets and perform advanced data engineering tasks.
Requirements
Requirements:
- Possess 5 to 7 years of professional experience in data engineering roles within the IT Services sector.
- Demonstrate strong proficiency in Data Modeling and Data Architecture principles.
- Show proven expertise in Data Validation and Data Quality assurance methodologies.
- Have hands-on experience with Error Handling and Logging strategies in production environments.
- Be skilled in orchestrating workflows using Apache Airflow, Tidal, or Control-M.
- Display competency in using Git for version control and managing CI/CD pipelines.
- Maintain a solid background in Data Engineering and ETL development.
- Possess practical experience with Databricks, PySpark, and Spark SQL technologies.
Salary: ₹6 -8 LPA