Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Astra-North Infoteck Inc.

Big Data Engineer (PySpark / Spark / Kafka)

Posted Updated

Description

Astra-North Infoteck Inc. in Toronto seeks an experienced Data Engineer with deep Big Data expertise to design, develop, and support enterprise-scale data platforms.

The role focuses on PySpark, Apache Spark, Kafka, Hadoop ecosystem, and NiFi, covering ingestion, transformation, and real-time processing. You will build scalable pipelines, implement ETL/ELT workflows, ensure data quality and governance, and collaborate with cross-functional teams.

Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi. Build batch and real-time data processing solutions and ensure data quality. Develop and support Kafka-based streaming applications and event-driven architectures. Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets. Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting. Monitor, troubleshoot, and optimize Spark jobs and data pipelines. Hands-on experience with PySpark, Spark SQL, and DataFrames for large-scale data processing. Strong knowledge of Hadoop components (HDFS, Hive, YARN) and data warehousing concepts. Proficient SQL and Python for data ingestion, transformation, and validation. Experience with real-time processing and streaming pipelines is a plus.

Skills

Apply

See also

Data Engineering jobs by country — openings, pay and top skills →