Senior Data Engineer
Posted Updated
Description
Job Summary
Job Purpose - To effectively design, develop, and manage data solutions using ETL technologies such as Azure Databricks (ADB) and Azure Data Factory (ADF) and SQL. To manage and maintain Azure Data Lake platform including the storage optimisation. To effectively manage and maintain Databricks platform including the admin activities and willing to work on new features like Databricks Apps, Agent Bricks, AI/BI dashboard. Should be well versed in Python, PySpark and Scala.
Responsibilities - Translate business requirements into technical solutions in collaboration with the PMO team.
- Own end-to-end delivery of data projects, ensuring on-time execution and adherence to quality standards.
- Design technical architecture and guide development efforts for enhancements and new projects.
- Develop and maintain robust ETL pipelines and data integration modules across systems.
- Ensure high data quality, platform stability, and resolution of critical process issues.
- Monitor and resolve performance bottlenecks in data workflows and programs.
- Establish best practices, standard operating procedures, and drive their implementation across teams.
- Act as a liaison with business users and product managers to support daily data needs and strategic initiatives.
- Coordinate with internal and external development teams to troubleshoot and resolve issues efficiently.
- Manage workload through effective planning, prioritization, and progress tracking.
- Azure Databricks - Python, PySpark, SQL - Must Have
- Azure Data Factory - For ETL & Data Integrations - Must Have
- OOPs Concept Implementation in Python
- Knowledge in dashboarding tool like Power BI - Good to Have