Data Engineer
Summary
Data Engineer (remote, Canada, full-time) building and optimizing scalable and real-time data pipelines with Python, PySpark, Spark SQL, and Azure Databricks, using Azure Data Lake, Data Factory, and EventHub/Kafka for streaming ingestion, within Agile Scrum teams.
Position: Data Engineer
Location: Canada (Remote)
Duration: Fulltime
Role Overview
•5-8 years of hands-on experience in software development, with a strong emphasis on Python programming.
•Experinec in AI (Artificial Intelligence) & Claude code.
•Deep expertise in PySpark for building and optimizing scalable data pipelines.
•Proficient in designing and implementing real-time data streaming solutions using Spark SQL, DataFrames, and Datasets.
•Solid experience with Azure Databricks, including working with Delta tables, workflows, and scripting in Python.
•Familiar with Azure cloud services, especially Azure Data Lake, Azure Data Factory, and EventHub or Kafka for streaming data ingestion.
•Skilled in writing complex SQL queries and building views involving joins across multiple tables.
•Experience across the Software Development Lifecycle (SDLC), with a strong grasp of Agile Scrum methodologies.
•Proficient in tools like Jira (project management), Confluence (documentation), and GitHub (version control and CI/CD pipelines).
Soft Skills & Collaboration:
•Strong collaboration mindset with enthusiasm for mobbing sessions and cross-team coordination.
•Excellent problem-solving and code review skills.
•Strong communication skills with the ability to understand business requirements and translate them into technical solutions.
•Proactive and responsive during business hours for any ad-hoc calls or meetings.
•Demonstrated ability to prioritize effectively, think creatively, and adapt quickly to new technologies and feedback.
Skills
Sign in to read and join the discussion.