Data Engineer Python/PySpark/OpenShift
Posted
Summary
Data Engineer in Mississauga building and maintaining scalable data pipelines and ETL/ELT workflows with Python, PySpark, and SQL. Manages Snowflake and data lake storage, orchestrates workflows with Azure Data Factory, and deploys containerized apps on OpenShift/Kubernetes.
The Data Engineer will design, develop, and maintain scalable data pipelines and workflows using Python, PySpark, and cloud technologies.
This role involves managing data storage solutions, orchestrating ETL processes, and deploying containerized applications to support enterprise data initiatives.
Key Responsibilities
- Develop, construct, test, and maintain scalable data pipelines and ETL/ELT workflows using Python, PySpark, and SQL.
- Manage and optimize data storage solutions in Snowflake and Data Lakes.
- Integrate data from multiple sources while ensuring data quality, integrity, and security.
- Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
- Deploy and manage containerized applications and services on OpenShift.
- Collaborate with Data Scientists, Analysts, and stakeholders to understand data requirements.
- Monitor and troubleshoot data pipelines, resolving performance bottlenecks.
- Maintain documentation of data architecture, workflows, and best practices.
- Ensure compliance with data governance and security policies.
Required Qualifications
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
- Minimum 10 years of overall experience in data engineering or similar roles.
- Strong proficiency in Python and PySpark.
- Expertise in SQL and experience with Snowflake Data Warehouse.
- Hands-on experience with data orchestration tools such as Azure Data Factory.
- Knowledge of Data Lakes architecture and management.
- Experience deploying and managing containerized applications on OpenShift or Kubernetes.
- Familiarity with cloud platforms (Azure, AWS, GCP).
- Understanding of data security, governance, and compliance best practices.
- Strong problem-solving skills and attention to detail.
- Excellent communication and teamwork abilities.
Preferred Qualifications
- Experience with big data tools (e.g., Kafka, Hadoop).
- Knowledge of CI/CD pipelines and DevOps practices.
- Familiarity with additional cloud data services and tools.