Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Focal Consulting

Data Engineering Lead: PySpark & Big Data Pipelines

Posted Updated

Description

Focal Consulting in Kuala Lumpur is seeking a Lead Data Engineer to architect scalable PySpark pipelines and guide a team of data engineers. You will design and optimize large-scale batch and streaming workflows, emphasizing reliability and performance.

You will also shape data lake architectures (raw/staged/curated) and collaborate with analysts and ML engineers to deliver production-grade analytics datasets in a fast-paced environment.

Architect and oversee large-scale batch and streaming data pipelines using PySpark and distributed data frameworks. Lead, mentor, and guide a team of data engineers, conducting code reviews and driving engineering best practices. Design and optimize data lake layers (raw/staged/curated), managing partitioning, storage formats, and schema design. Diagnose bottlenecks, optimize PySpark job performance (memory, shuffling, caching), ensuring high pipeline reliability. Collaborate with data analysts, ML engineers, and business leaders to deliver production-grade analytics datasets. Bachelor’s or Master’s Degree in Computer Science, Data Engineering, Information Systems, or a related field. 3+ years of data engineering experience, with at least 1 year in leadership/mentoring. Expert-level PySpark, Python, and advanced SQL for complex data transformation and processing. Hands-on experience with Spark clusters, workflow orchestrators (Airflow, Dagster), and data lake formats (Delta Lake or Iceberg). Strong analytical mindset and ability to balance system design with team coordination. 13-Month Salary Yearly Performance Bonus Monthly Transport Allowance Monthly Rental Subsidies

Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →