Azure Data Engineer - Python/Databricks

Azure Data Engineer

Key Responsibilities:

- Design, develop, and maintain scalable data pipelines using Azure Data Factory (ADF).

- Build and optimize data processing workflows using Azure Databricks and Python.

- Develop ETL/ELT processes to extract, transform, and load data from multiple sources.

- Work with structured and unstructured data to support analytics and reporting requirements.

- Implement data transformation, cleansing, validation, and integration processes.

- Optimize data pipelines and Spark jobs for performance, scalability, and cost efficiency.

- Integrate data from various sources into Azure-based data platforms.

- Ensure data quality, consistency, reliability, and security across data workflows.

- Monitor and troubleshoot data pipelines, jobs, and production issues.

- Collaborate with data analysts, data scientists, architects, and business stakeholders to understand data requirements.

- Follow best practices for data governance, access management, documentation, and deployment.

- Support automated deployment and version control processes for data engineering solutions.

Required Skills and Qualifications:

- 5 - 10 years of relevant experience in data engineering or related technical roles.

- Strong hands-on experience with Microsoft Azure data services.

- Proficiency in Python for data processing and automation.

- Hands-on experience with Azure Data Factory (ADF) for building and managing data pipelines.

- Strong experience with Azure Databricks for data transformation and processing.

- Understanding of ETL/ELT concepts, data integration, and data warehousing.

- Knowledge of SQL and database concepts.

- Experience with data pipeline monitoring, troubleshooting, and performance optimization.

- Strong analytical, problem-solving, and communication skills.