Senior Data Engineer
Summary
Senior Data Engineer building modern cloud-based data solutions in the automotive industry using Microsoft Azure and Azure Databricks. Day to day: developing pipelines with Python and Spark, managing Databricks workflows and ADLS, building CI/CD with Azure DevOps/GitHub Actions, integrating FastAPI REST APIs, and deploying Docker containers, with NATS/Redis for messaging and caching in SCRUM teams
Jora Malaysia will close on 16th September 2026. Thank you for being with us, we are cheering you on as you continue your career journey.
We are looking for an experienced Data Engineer to join Automotive Industry and support the development of modern cloud-based data solutions using Microsoft Azure and Azure Databricks.
Responsibilities
- Develop and maintain data pipelines using Azure Databricks, Python, and Spark.
- Manage Databricks workflows, ADLS, Hive Metastore, and compute clusters.
- Develop and maintain CI/CD pipelines using Azure DevOps and GitHub Actions.
- Work with Azure Key Vault, Entra ID, and Azure Log Analytics.
- Develop and integrate REST APIs using FastAPI.
- Build and deploy containerized applications using Docker and Azure Container Registry.
- Work with NATS and Redis for messaging and caching.
- Collaborate with project teams using SCRUM.
Requirements
- 7+ years of experience in Data Engineering.
- 7+ years of hands-on experience with Azure Databricks, Python, and Azure DevOps.
- Strong knowledge of Apache Spark and SQL.
- Experience with Microsoft Azure, ADLS, Azure Key Vault, and Entra ID.
- Experience with CI/CD, GitHub/Azure Repos, and YAML pipelines.
- Experience with FastAPI, REST APIs, and Docker.
- Strong understanding of cloud-native data engineering.
Good to Have: Unity Catalog, Medallion Architecture, Pandas/Polars, KQL, PowerShell/Bash, SAFe, NATS, and Redis.