Senior Data Engineer - Python/PySpark

Job Overview :

We are seeking an experienced Senior Data Engineer to join our dynamic team and drive our data engineering initiatives.

The ideal candidate will have extensive experience with Azure cloud technologies and a proven track record of designing, implementing, and maintaining scalable data solutions. You will play a crucial role in building robust data pipelines, optimizing data workflows, and enabling data-driven decision making across the organization.

Key Responsibilities :

- Design, develop, and maintain complex data pipelines using Azure Data Factory (ADF) and other Azure data services.

- Build and optimize data processing workflows using Azure Databricks with Spark and PySpark.

- Implement and manage Lake flow Declarative Pipelines for efficient data processing and transformation.

- Develop and optimize SQL queries and stored procedures for data extraction, transformation, and loading (ETL) processes.

- Create and maintain data models, schemas, and documentation to ensure data quality and consistency.

- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions.

- Monitor and troubleshoot data pipeline performance, implementing improvements and optimizations as needed.

- Ensure data security, privacy, and compliance with industry standards and regulations.

- Mentor junior team members and provide technical guidance on data engineering best practices.

- Stay current with emerging technologies and industry trends in data engineering and cloud computing.

- Participate in code reviews, testing, and deployment processes to maintain high-quality deliverables.

- Design and implement data governance frameworks and data quality monitoring solutions.