Data Engineer
Summary
Ubique Systems is hiring an Azure Data Engineer in Amsterdam to design, build, and optimize scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, Python, and Azure Data Factory, with Delta Lake/ADLS storage and CI/CD via Azure DevOps.
Job Description: Azure Data Engineer – Azure Databricks & PySpark
Job Overview:
We are looking for an experienced Azure Data Engineer with strong expertise in Azure Databricks, PySpark, Python, and Azure Data Factory to design, develop, and maintain scalable data pipelines and ETL/ELT solutions.
The candidate should have hands-on experience in big data processing, cloud data engineering, pipeline optimization, and CI/CD implementation using Azure DevOps.
Key Responsibilities:
- Design, develop, and maintain data pipelines using Azure Data Factory and Azure Databricks.
- Build and optimize large-scale data processing solutions using PySpark and Python.
- Develop ETL/ELT workflows for data extraction, transformation, and loading.
- Work with Azure Data Lake Storage (ADLS) and Delta Lake.
- Implement CI/CD pipelines using Azure DevOps.
- Write complex SQL queries and optimize data processing performance.
- Monitor, troubleshoot, and optimize data pipelines.
- Ensure data quality, security, and compliance.
- Collaborate with cross-functional teams to deliver effective data solutions.
Required Skills (Must Have):
- Strong hands-on experience with Azure Databricks.
- Expertise in PySpark and Python programming.
- Experience with Azure Data Factory (ADF) and pipeline orchestration.
- Strong knowledge of SQL and Spark SQL.
- Experience with Azure DevOps and CI/CD pipelines.
- Understanding of ETL/ELT processes and data integration.
- Good problem-solving, communication, and documentation skills.
Good-to-Have Skills:
- Azure Data Lake Storage (ADLS Gen2).
- Delta Lake and Delta Tables.
- Azure Synapse Analytics.
- Databricks Workflows and Jobs.
- Git and version control.
- Medallion Architecture (Bronze, Silver, Gold).
Skills
Sign in to read and join the discussion.