PySpark Data Engineer

We are seeking an experienced PySpark Data Engineer to design, develop, and maintain scalable, high-performance data pipelines. The role requires strong expertise in Apache Spark, large-scale data processing, and enterprise data integration within complex environments.

Responsibilities:

• Design, develop, and support scalable data pipelines using PySpark in production environments.
• Build and maintain robust data ingestion frameworks, integrating data from multiple heterogeneous source systems.
• Process and transform large datasets (500GB+), ensuring data quality, consistency, and performance optimisation.
• Optimise Spark jobs by leveraging a deep understanding of Spark architecture (DAG execution, partitioning, shuffles, caching, executors, cluster resource management).
• Work extensively with HDFS and structured/semi-structured data formats such as Parquet, ORC, Avro, and CSV.
• Perform data cleansing, standardisation, and transformation to deliver curated datasets for downstream consumption.
• Develop and maintain Python and shell scripts to automate workflows and reduce manual intervention.
• Collaborate with cross-functional teams, including Data Architects, Business Analysts, and DevOps teams.
• Ensure adherence to enterprise data governance, security, and compliance standards.
• Troubleshoot and resolve performance bottlenecks and production issues.

Requirements:

• 5+ years of experience in data engineering with strong hands-on expertise in PySpark and Apache Spark.
• Solid understanding of distributed data processing and big data ecosystems.
• Strong programming skills in Python and advanced SQL (Oracle preferred).
• Hands-on experience with HDFS and large-scale data environments.
• Proven ability to handle and optimise high-volume data processing workloads.
• Strong analytical and problem-solving skills.

Preferred Skills:

• Experience with cloud platforms (AWS/Azure/GCP).
• Exposure to scheduling/orchestration tools such as Airflow, Control-M, or Oozie.
• Familiarity with CI/CD practices and DevOps integration.
• Experience working in enterprise environments with strict SLAs and compliance requirements.

Soft Skills:

• Strong communication and stakeholder management skills.
• Ability to work in structured, process-driven environments.
• Proactive approach to issue resolution and continuous improvement.