Data Engineering Tech Lead — Databricks & AWS
Posted Updated
Description
BP in Kuala Lumpur is seeking a Senior Data Engineer to own the data domain within an autonomous squad. You will design, build, migrate, operate, and continuously improve scalable data products and pipelines using Databricks, PySpark, Spark SQL, and AWS, ensuring reliability, security and cost efficiency.
You will lead data ingestion, transformation, and storage architectures, enforce data quality and governance, and drive observability and incident resolution.
Lead the design, development, migration, and operation of scalable data pipelines and data products using Databricks, PySpark, Spark SQL, and AWS. Own the squad’s data engineering domain, providing technical direction and ensuring secure, reliable solutions. Design and implement scalable data ingestion and migration frameworks using Databricks Auto Loader and Delta Live Tables (DLT). Develop and maintain data models, transformation pipelines, and storage solutions with emphasis on data quality, integrity, performance, and scalability. Build and maintain pipeline orchestration and deployment capabilities using Databricks Workflows, Asset Bundles (DAB), CI/CD, and Infrastructure as Code. Integrate data platforms with the AWS ecosystem, including IAM, storage, security, networking, and related services. Implement governance, security, lineage, and access controls using Unity Catalog and enterprise standards. Establish observability, monitoring, alerting, and cost optimization across pipelines and Databricks workloads. Drive operational excellence through incident resolution, root cause analysis, automation, and reliability improvements. Provide technical leadership through architecture decisions, code reviews, and mentoring. 8+ years of experience in Data Engineering, Software Engineering, or related technical disciplines. 5+ years hands-on Python experience. 3+ years working with PySpark and distributed data processing. Strong hands-on experience with Databricks, PySpark, Spark SQL, SQL, Delta Lake and data modeling. Experience designing and operating enterprise-scale data ingestion, transformation, migration, and processing frameworks. Hands-on experience with Databricks Auto Loader, Delta Live Tables (DLT), Workflows, and Databricks Asset Bundles (DAB). Strong AWS experience, including IAM, S3 and integration with AWS cloud services. Experience with Unity Catalog including governance, metadata, lineage, and access control. Experience with CI/CD, Infrastructure as Code, automated testing, Git, and deployment automation. Strong understanding of observability, data quality, performance optimization, cost monitoring, reliability, and production support. Life and health insurance Learning opportunitiesSkills
Sign in to read and join the discussion.