Senior Data Engineer | Spark & Delta Lake Architect
Summary
Senior data engineer in Johannesburg optimizing Spark/PySpark jobs (memory, shuffle, partitioning) for cost-efficient performance. Builds cloud-agnostic Python and Delta Lake (Bronze/Silver/Gold) lakehouses, migrates SQL ETL into modular Python libraries, and leads code reviews, mentoring, and Pytest-based automated testing.
Hire Resolve in Gauteng seeks an experienced Spark/PySpark data engineer to optimize Spark jobs, memory usage, shuffle tuning and partitioning for cost-efficient performance. The role involves cloud-agnostic development using Python and Delta Lake, migrating SQL-based ETL to modular Python libraries, and managing a Delta Lake lakehouse (Bronze/Silver/Gold) with good storage performance.
You will lead code reviews, mentor junior engineers and implement automated testing (Pytest); 6+ years in