Production-Scale PySpark Data Engineer
Summary
Designs, builds, and optimizes production-scale PySpark pipelines for data ingestion, transformation, and validation using Python and Spark performance tuning.
Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.
The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.