Python / spark / ai developer
Summary
Data engineering role in Cape Town (remote/hybrid/office flexible) building Python and PySpark pipelines on Delta Lake, replatforming legacy T-SQL to Spark SQL, and exposing FastAPI REST APIs with type-safe, CI-tested Python.
Build Python & Py Spark pipelines on Delta Lake
Replatform legacy T-SQL to Spark SQL
Develop tested, type-safe Python with CI quality gates
Build Fast API REST APIs with authentication & job tracking
Validate migrated pipelines against legacy systems
Location:
Cape Town, Remote/Hybrid or In office candidate can decide
Job Type:
Permanent
Date:
2 days ago
Requirements4+ years Python development
2+ years production Spark/Py Spark
Strong Python 3.12, pytest, Pydantic & modern tooling
Spark SQL, Data Frames, performance tuning & distributed architecture
Docker, Git & CI/CD
Able to work independently in a spec-driven environment
Nice to Have
Observability Data Migration & Parity Testing Databricks Certification
Qualifications:
Bachelor’s Degree in Computer Science, Engineering or equivalent experience
Spark/Databricks certification advantageous #J-18808-Ljbffr