Senior Data Engineer (Databricks)

Salary: RM 10,000 – RM 15,000 per month

We are seeking a highly experienced Senior Data Engineer with strong Databricks expertise to design, develop, and maintain scalable, secure, and high-performance data infrastructure.

This role will play a key part in enabling advanced analytics, enterprise reporting, and digital transformation initiatives by building robust data pipelines, integrating multiple data sources, and optimising data processing workflows. The successful candidate will bring strong technical expertise in modern data engineering, cloud platforms, data governance, and performance optimisation, while working closely with technical teams and business stakeholders to deliver reliable, data-driven solutions.

Key Responsibilities

1. Data Engineering & Platform Development

• Design, develop, and maintain scalable, secure, and high-performance data pipelines and infrastructure using Databricks and modern data engineering technologies.

• Develop and optimise data architectures, including data lakehouse solutions, data storage, processing, and retrieval mechanisms.

• Build and maintain reliable ETL/ELT pipelines to support analytics, reporting, and enterprise data consumption.

• Implement engineering best practices for code quality, version control, testing, deployment, and production support.

• Monitor, troubleshoot, and resolve complex data pipeline and platform issues to ensure reliability, availability, and scalability.

2. Data Integration & Workflow Automation

• Design and implement automated data ingestion and transformation workflows across multiple internal and external data sources.

• Develop and maintain data integration solutions using batch and, where required, streaming processing techniques.

• Leverage Databricks capabilities, including Apache Spark, Delta Lake, and related data engineering tools, to process and manage large-scale datasets.

• Ensure data is delivered accurately, consistently, and on time to downstream analytics platforms, applications, and reporting solutions.

• Identify opportunities to automate manual processes and improve data engineering efficiency.

3. Data Quality, Governance & Security

• Implement data quality checks, validation rules, reconciliation mechanisms, and monitoring frameworks to maintain data integrity.

• Collaborate with data governance and security teams to support compliance with PDB's data management policies, security requirements, and applicable standards.

• Support data lineage, metadata management, documentation, access controls, and auditability across data pipelines.

• Apply appropriate security practices to protect sensitive data and ensure controlled access to data assets.

4. Performance & Cost Optimisation

• Analyse and optimise data pipeline performance, Spark workloads, queries, storage layouts, and resource utilisation.

• Implement cost-efficient data processing and cloud resource management strategies.

• Identify and resolve performance bottlenecks across data ingestion, transformation, storage, and consumption layers.

• Evaluate emerging data engineering technologies and recommend improvements to platform architecture, operational efficiency, and scalability.

5. Technical Leadership & Stakeholder Collaboration

• Provide technical guidance and best-practice recommendations on data engineering architecture, development standards, and implementation approaches.

• Collaborate with data scientists, data analysts, solution architects, and business stakeholders to translate business requirements into scalable technical solutions.

• Lead technical discussions, troubleshoot complex issues, and support the delivery of data engineering initiatives.

• Prepare and maintain technical documentation, architecture diagrams, operational procedures, and knowledge transfer materials.

• Communicate technical concepts, project progress, risks, and recommendations effectively to both technical and non-technical stakeholders.

Requirements

• Bachelor's degree in Computer Science, Information Technology, Data Engineering, Software Engineering, or a related discipline.

• Minimum 8–10 years of relevant experience in data engineering, including substantial hands-on experience designing and delivering enterprise-scale data solutions.

• Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, notebooks, workflows/jobs, and performance optimisation.

• Proven expertise in developing and maintaining complex ETL/ELT pipelines and processing large volumes of structured and unstructured data.

• Strong programming skills in Python and SQL; experience with PySpark is highly desirable.

• Experience with cloud platforms such as Microsoft Azure or AWS, including cloud storage, data processing services, security, and resource optimisation.

• Strong understanding of modern data architecture, including data lake, data warehouse, and lakehouse concepts.

• Experience implementing data quality controls, data validation, monitoring, lineage, and governance practices.

• Familiarity with CI/CD, Git, automated testing, and deployment practices for data engineering solutions.

• Demonstrated ability to troubleshoot complex technical issues and deliver reliable, production-ready data solutions.

• Strong analytical, problem-solving, communication, and stakeholder management skills.

Preferred Qualifications

• Databricks certifications or other relevant cloud data engineering certifications.

• Experience supporting large-scale enterprise data platforms within complex organisational environments.

• Exposure to enterprise data governance, regulatory compliance, and security controls.

• Experience with infrastructure as code, orchestration tools, and data observability solutions.

• Experience supporting digital transformation, advanced analytics, or business intelligence initiatives.

What Success Looks Like

• Reliable, scalable, and secure data pipelines supporting PDB's analytics and digital initiatives.

• Improved data availability, quality, and consistency across integrated data sources.

• Optimised Databricks workloads, platform performance, and cloud resource consumption.

• Strong adherence to data governance, security, documentation, and operational standards.

• Effective collaboration with business and technical stakeholders to deliver measurable improvements in data capability and operational efficiency.