Azure Data Engineer (ID: 4098)
Posted 1
view
Summary
Designs, builds, and maintains scalable data pipelines on Microsoft Azure, primarily using Python, Azure Data Factory, Azure Databricks, and Apache Spark. Day to day involves ETL/ELT development, API-based data integration, pipeline monitoring and troubleshooting, and performance optimization in an Agile team environment.
- Design, develop, and maintain scalable data pipelines using Microsoft Azure data services.
- Develop data processing and transformation solutions using Python.
- Build, configure, and manage data workflows using Azure Data Factory (ADF).
- Develop and optimize data processing solutions using Azure Databricks and Apache Spark.
- Integrate data from multiple sources using APIs and other data integration techniques.
- Work with Azure Databricks and Azure data services to process, transform, and prepare data for analytics and reporting.
- Implement data ingestion, transformation, and loading processes to support business requirements.
- Monitor data pipelines, troubleshoot failures, and resolve data processing issues.
- Optimize pipeline performance, reliability, and scalability.
- Collaborate with cross-functional teams to understand data requirements and deliver effective technical solutions.
- Follow coding standards, version control practices, testing procedures, and Agile ways of working.
- Maintain technical documentation and support data platform improvements.
- 4-6 years of professional experience in data engineering, Azure data solutions, or related technical roles.
- Hands-on experience with Microsoft Azure data engineering services.
- Strong programming knowledge in Python for data processing and transformation.
- Experience developing and maintaining data pipelines using Azure Data Factory.
- Practical experience with Azure Databricks and Apache Spark.
- Experience integrating data from external systems using APIs.
- Understanding of data ingestion, transformation, ETL/ELT processes, and data integration concepts.
- Knowledge of data pipeline monitoring, troubleshooting, and performance optimization.
- Familiarity with Agile methodologies and collaborative development practices.
- Strong analytical, problem-solving, and communication skills.
- Develop reliable and scalable data pipelines using Python, Azure Data Factory, and Azure Databricks.
- Extract, transform, and load data from different sources into target data platforms.
- Build API-based integrations to ingest data from external applications and systems.
- Process and transform large datasets using Databricks and Spark.
- Troubleshoot pipeline failures, data quality issues, and performance bottlenecks.
- Optimize data processing workflows to improve performance and reliability.
- Collaborate with technical teams and business stakeholders to translate requirements into data engineering solutions.
- Follow established development standards and participate in Agile development activities.
- Document data pipelines, integration processes, and technical solutions.
- An opportunity to work on Azure-based data engineering projects.
- Exposure to Python development, cloud data integration, and large-scale data processing.
- Opportunities to contribute to data pipeline development and optimization.
- Collaboration with technical teams in an Agile working environment.