Pyspark ETL Developer
Posted
Summary
A 5-month contract ETL developer role in Toronto (via isgSearch) serving a government health data organization: building and maintaining big-data pipelines and analytics products with Python, PySpark, and SQL on AWS (EMR), orchestrated with Apache Airflow, plus Linux shell scripting and ETL documentation.
- Python and PySpark programing languages to manipulate data and draw insights from large datasets
- Experience with AWS EMR, AWS Cloud
- Hands-on programming skills, able to write modular, maintainable code, preferably Python & SQL. Good understanding of SQL, dimensional modeling, and analytical big data warehouses
- Experience with Apache Airflow
- Experience with Python, PySpark programming in the context of data processing / data science applications
- Profound knowledge and experience in shell scripting in Linux/Unix environment. Experience in creating and maintaining documentation of project ETL design, technical artifacts and ETL environment
Duration:
5 Month contract
Must have:
- Python and PySpark programing languages to manipulate data and draw insights from large datasets
- Experience with AWS EMR, AWS Cloud
- Hands-on programming skills, able to write modular, maintainable code, preferably Python & SQL. Good understanding of SQL, dimensional modeling, and analytical big data warehouses
- Experience with Apache Airflow
- Experience with Python, PySpark programming in the context of data processing / data science applications
- Profound knowledge and experience in shell scripting in Linux/Unix environment. Experience in creating and maintaining documentation of project ETL design, technical artifacts and ETL environment
Our client…
A government health data organization.
Responsibilities:
- Understand systems and Business processes related to the products in their assigned portfolio
- Works closely with client analytical teams to develop, maintain and support products using Python and/or R and PySpark programming languages.
- Provides support and guidance on creating top-quality clean code and participates in code review.
- Analyses Software deficiencies/problems and support to fix them
- Prepares documentation, reports, and presentations related to projects and activities.