Senior Technical Lead
Posted Updated
Description
HCL Technologies Middle East FZLLC Dubai Branch is seeking a Senior PySpark Developer to design, develop, and maintain large-scale data processing systems using PySpark to ingest, transform, and analyze massive data volumes.
You will build scalable ETL/ELT pipelines, optimize Spark jobs, shape data models, and integrate Spark apps with AWS S3, Azure ADLS, and GCP Dataproc, while applying CI/CD practices and unit testing.
Design and build scalable ETL/ELT pipelines to move and transform data. Develop PySpark scripts for large-scale data processing (DataFrames, Datasets, Spark SQL). Tune Spark jobs for performance, partitions, caching, and shuffles. Model data schemas and storage formats (Parquet, Delta Lake, ORC). Integrate Spark with cloud storage (S3/ADLS/Blob) and databases. Automate data workflows with CI/CD practices and testing. Extensive PySpark ETL/ELT pipeline experience. Strong SQL and data modeling skills. Cloud data platform experience (AWS/Azure/GCP). Familiarity with Parquet and Delta Lake formats. Hands-on CI/CD and data workflow automation.Skills
Sign in to read and join the discussion.