Data Engineer (Databricks)

Salary: RM 5,500 – RM 8,000 per month

We are looking for a skilled Databricks Data Engineer to join our team for a client project. The successful candidate will be responsible for designing, developing, and optimizing data pipelines, ETL/ELT processes, and machine learning workflows using the Databricks platform.

You will play a key role in transforming Proof of Concept (POC) solutions into production-ready implementations while ensuring data quality, governance, security, and performance.

Key Responsibilities

• Design, develop, and deploy scalable data pipelines and ETL/ELT processes using Databricks.

• Build and support data engineering and machine learning workflows to facilitate active Proof of Concept (POC) initiatives.

• Transform experimental solutions into reliable, production-ready data engineering implementations.

• Apply Databricks best practices in data engineering and platform implementation.

• Work with Delta Lake, Apache Spark, and Unity Catalog to support data management, governance, security, and lineage tracking.

• Develop scalable and maintainable data architectures aligned with project requirements.

• Operationalize successful POCs by transitioning notebooks and pipelines into production environments.

• Implement workflow orchestration using Databricks Workflows, Apache Airflow, or similar tools.

• Support CI/CD pipeline implementation and deployment automation.

• Optimize Apache Spark jobs, cluster configurations, and SQL queries to improve performance and cost efficiency.

• Troubleshoot data pipeline issues and resolve technical bottlenecks.

• Monitor and enhance the reliability, scalability, and efficiency of Databricks workloads.

• Collaborate with project teams and stakeholders to understand technical requirements and deliver appropriate data engineering solutions.

• Troubleshoot technical issues and recommend improvements.

• Prepare and maintain technical documentation, including solution architectures, operational runbooks, deployment guides, and handover documentation.

Job Requirements

• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.

• 3–5 years of relevant experience in data engineering, preferably involving Databricks implementations.

• Hands-on experience with Databricks, Apache Spark, and PySpark.

• Strong knowledge of data pipeline development and ETL/ELT processes.

• Experience working with Delta Lake and Unity Catalog.

• Familiarity with workflow orchestration tools such as Databricks Workflows or Apache Airflow.

• Understanding of CI/CD practices and deployment of data pipelines into production environments.

• Strong troubleshooting, analytical, and problem-solving skills.

• Good communication and collaboration skills, with the ability to document technical solutions effectively.