Data Engineer
Summary
Data engineer building and optimizing Databricks-based data pipelines, ETL/ELT processes, and ML workflows for a client project, turning POCs into production-ready solutions. Core stack includes Apache Spark, PySpark, Delta Lake, Unity Catalog, and orchestration tools like Airflow.
Salary: RM 5,500 – RM 8,000 per month
We are looking for a skilled Databricks Data Engineer to join our team for a client project. The successful candidate will be responsible for designing, developing, and optimizing data pipelines, ETL/ELT processes, and machine learning workflows using the Databricks platform.
You will play a key role in transforming Proof of Concept (POC) solutions into production-ready implementations while ensuring data quality, governance, security, and performance.
Key Responsibilities
Design, develop, and deploy scalable data pipelines and ETL/ELT processes using Databricks.
Build and support data engineering and machine learning workflows to facilitate active Proof of Concept (POC) initiatives.
Transform experimental solutions into reliable, production-ready data engineering implementations.
Apply Databricks best practices in data engineering and platform implementation.
Work with Delta Lake, Apache Spark, and Unity Catalog to support data management, governance, security, and lineage tracking.
Develop scalable and maintainable data architectures aligned with project requirements.
Operationalize successful POCs by transitioning notebooks and pipelines into production environments.
Implement workflow orchestration using Databricks Workflows, Apache Airflow, or similar tools.
Support CI/CD pipeline implementation and deployment automation.
Optimize Apache Spark jobs, cluster configurations, and SQL queries to improve performance and cost efficiency.
Troubleshoot data pipeline issues and resolve technical bottlenecks.
Monitor and enhance the reliability, scalability, and efficiency of Databricks workloads.
Collaborate with project teams and stakeholders to understand technical requirements and deliver appropriate data engineering solutions.
Troubleshoot technical issues and recommend improvements.
Prepare and maintain technical documentation, including solution architectures, operational runbooks, deployment guides, and handover documentation.
Job Requirements
Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field.
3–5 years of relevant experience in data engineering, preferably involving Databricks implementations.
Hands-on experience with Databricks, Apache Spark, and PySpark.
Strong knowledge of data pipeline development and ETL/ELT processes.
Experience working with Delta Lake and Unity Catalog.
Familiarity with workflow orchestration tools such as Databricks Workflows or Apache Airflow.
Understanding of CI/CD practices and deployment of data pipelines into production environments.
Strong troubleshooting, analytical, and problem-solving skills.
Good communication and collaboration skills, with the ability to document technical solutions effectively.