PySpark Developer
Summary
Builds and maintains large-scale data processing pipelines and ETL/ELT workflows using PySpark, Apache Spark, Python, and SQL, with a focus on performance tuning, integrating data from databases, APIs, and data lakes, and supporting production deployments.
Key Responsibilities :
- Design, develop, and maintain scalable data processing pipelines using PySpark and Apache Spark.
- Build and optimize ETL/ELT workflows to process large volumes of structured and unstructured data.
- Develop complex SQL queries and perform data transformation, cleansing, validation, and aggregation.
- Optimize Spark jobs for performance, scalability, and efficient resource utilization.
- Integrate data from multiple sources, including databases, APIs, data lakes, and enterprise systems.
- Implement data quality checks, error handling, logging, and monitoring for reliable data pipelines.
- Collaborate with data engineers, data architects, analysts, and business teams to deliver data solutions.
- Support production deployments, troubleshoot pipeline failures, and resolve data processing issues.
- Follow coding standards, version control, testing, documentation, and CI/CD best practices.
Required Skillsets :
- Strong hands-on experience in PySpark, Python, SQL, and Apache Spark.
- Experience developing ETL/ELT pipelines and processing large datasets.
- Strong understanding of Spark architecture, DataFrames, RDDs, Spark SQL, joins, partitioning, and caching.
- Experience in performance tuning, debugging, and optimizing distributed data processing jobs.
- Knowledge of data warehousing, data modeling, and data lake concepts.
- Experience with databases such as Oracle, PostgreSQL, or other relational/NoSQL databases.
- Familiarity with cloud platforms such as AWS, Azure, or GCP and their data engineering services.
- Knowledge of workflow orchestration tools such as Airflow and version control tools such as Git.
- Understanding of data quality, security, and governance practices.