Azure Databricks Data Engineer

Role and Responsibility:

• Build and maintain scalable pipelines with Azure Databricks, Python, and PySpark.
• Orchestrate data movement and workflows with ADF pipelines, triggers, and integration runtimes.
• Ingest and transform large datasets from sources such as SQL Server and Oracle.
• Manage data governance and access control with Unity Catalog.
• Implement cloud-native data solutions following best practices, with experience in the Azure or AWS cloud.
• Monitor, troubleshoot, and tune pipelines for performance and reliability; take part in the on-call rotation.
• Turn business requirements into robust data solutions and support production releases.
• Maintain documentation and runbooks.

Qualifications:

Graduate

• Minimum 5 years of experience in Data Engineering, Data Warehousing, or related areas.
• Bachelors degree in Computer Science, Information Technology, or a related discipline or equivalent practical experience.
• Strong hands-on experience with Azure Databricks (jobs/workflows, Delta Lake, Unity Catalog) in enterprise-scale environments.
• Advanced Python skills and experience developing distributed data processing solutions using PySpark.
• Hands-on experience with Azure Data Factory pipelines and ETL/ELT design, development, and optimization.
• Strong understanding of Azure cloud services, networking and storage, data modeling, and SQL (T-SQL preferred).
• Working knowledge of Windows and Linux environments.
• Experience with CI/CD and DevOps practices for automated deployment of data solutions (GitHub Actions preferred).
• Proven ability to troubleshoot complex data processing and pipeline issues in production environments.