Big Data Engineer (PySpark / Spark / Kafka)
Posted Updated
Description
Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms.
The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation, and real-time processing. You will build scalable pipelines, implement ETL/ELT workflows, ensure data quality and governance, and collaborate with cross-functional teams.
Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi. Build batch and real-time data processing solutions and ensure data quality. Develop and support Kafka-based streaming applications and event-driven architectures. Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets. Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting. Monitor, troubleshoot, and optimize Spark jobs and data pipelines. Hands-on experience with PySpark, Spark SQL, and DataFrames for large-scale data processing. Strong knowledge of Hadoop components (HDFS, Hive, YARN) and data warehousing concepts. Proficient SQL and Python for data ingestion, transformation, and validation. Experience with real-time processing and streaming pipelines is a plus.Skills
Sign in to read and join the discussion.