Data Engineer
Posted
Summary
Data engineer in Toronto building Kafka-based streaming and event-driven data pipelines: ingesting from databases, APIs, files, and streams; writing complex SQL for transformation and validation; and monitoring performance, data quality, security, and releases. Core stack is PySpark/Spark, Kafka, Hadoop (HDFS, Hive, YARN), Apache NiFi, and SQL.
- Build Kafka-based streaming and event-driven solutions.
- Write complex SQL queries for data transformation and validation.
- Implement data ingestion from databases, APIs, files, and streaming sources.
- Monitor performance, troubleshoot issues, and ensure data quality, security, and governance.
- Support deployments, testing, upgrades, and production releases.
- PySpark and Apache Spark (Spark SQL, DataFrames)
- Apache Kafka
- Hadoop Ecosystem (HDFS, Hive, YARN)
- Apache NiFi
- SQL
Skills
Sign in to read and join the discussion.