Senior Data Engineer - Python/Big Data

Scope of the job:

Sr. Data Engineer, Global Data Management.

Your Role:

You will be a part of the data engineering team and working on in-house products for Data Foundation, collaborating closely with Data Engineering Leads, managers, and other DataOps teams. This role involves hands-on development work and will directly report to the Data Engineering Lead.

Must have:

- Strong experience in Azure cloud data services & technologies

- Azure Databricks, Unity Catalog, Delta live tables

- Data Modelling & Architecture

- ETL pipeline design

- Expert in Python, Pyspark and SQL

- Azure Data Factory

- Azure DevOps

- Logging and Monitoring using Azure / Databricks services

- Apache Kafka

- Technology

Nice to have:

- Synapse, Fabric, PowerBI, Azure Functions, Azure Logic apps, Azure API services

- Microsoft Entra ID

- GCP & AWS cloud services knowledge

- Azure Networking

- Databricks online table

- Jira

- Shell scripting / Azure CLI knowledge

Key responsibilities:

- Design, develop, and deliver scalable, reliable, and high performing software solutions.

- Design, develop, and maintain scalable data pipelines and ETL processes while monitoring and optimizing data infrastructure performance.

- Participate in code reviews, design reviews, testing, and deployment processes.

- Ensure adherence to coding standards, best practices, and architectural guidelines.

- Inspire, advise, and drive the selection of development approach.

- Participate in technical discussions with other teams and departments.

- Communicate with Engineering Manager, Product Owner, Business Analyst and Scrum Master to align on project sprint goals, timeline and resource allocation.

Budget responsibilities:

Skill Required:

- 5 - 12+ years of working experience in the field of Data Engineering and Software Engineering.

- Hands-on experience and in-depth knowledge of the technologies listed as mandatory in the Technology Stack section.

- Strong understanding and implementation of software development principles, coding standards, and modern architecture.

- Experience in working in diverse projects with varying technologies, products, and systems.

- Strong problem-solving skills and ability to make critical technical decisions.

Language(s):

- Ability to guide / mentor other team members.

- Stakeholder engagement/influence skills.

- Pragmatic, and collaborative team player.

- Experience with Databricks, Data pipelines, CI/CD.

- Proficiency in Python, SQL and big data technologies in Cloud environment.

- Excellent written and verbal English.

- Ability to set clear goals and objectives for the team.