Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Yochana

NewBe an early applicant

Sr. Lead Data Engineer ( AI Claude, Azure Databricks)

Posted Updated

Summary

Senior/lead data engineer (2 full-time, remote-Canada positions) building and optimizing scalable data pipelines and real-time streaming with Python, PySpark, and Azure Databricks (Delta tables, workflows), ingesting via Azure Data Lake/Data Factory and EventHub/Kafka; hands-on AI and Claude code experience is mandatory.


Position Name – Sr. Data Engineer or Lead ( 2 Positions)

Type of hiring – Fulltime

Location – Remote Canada


Looking for a Sr Data Engineer who has experience in AI & Claude, Python, PySpark, Azure Databricks, Spark(SQL, Dataframes, Datasets, Real-time Data Streaming, Azure Data Services & Advanced SQL.


(5-8) or (8-12 Years)



Job Description:


Role Overview

  • 5-8 or (8-12 Years) Years of hands-on experience in software development, with a strong emphasis on Python programming.
  • Experience in AI (Artificial Intelligence) & Claude code is mandatory.
  • Deep expertise in PySpark for building and optimizing scalable data pipelines.
  • Proficient in designing and implementing real-time data streaming solutions using Spark SQL, Dataframes, and Datasets.
  • Solid experience with Azure Databricks, including working with Delta tables, workflows, and scripting in Python.
  • Familiar with Azure cloud services, especially Azure Data Lake, Azure Data Factory, and EventHub or Kafka for streaming data ingestion.
  • Skilled in writing complex SQL queries and building views involving joins across multiple tables.
  • Experience across the Software Development Lifecycle (SDLC), with a strong grasp of Agile Scrum methodologies.
  • Proficient in tools like Jira (project management), Confluence (documentation), and GitHub (version control and CI/CD pipelines).


Soft Skills & Collaboration:

  • Strong collaboration mindset with enthusiasm for mobbing sessions and cross-team coordination.
  • Excellent problem-solving and code review skills.
  • Strong communication skills with the ability to understand business requirements and translate them into technical solutions.
  • Proactive and responsive during business hours for any ad-hoc calls or meetings.
  • Demonstrated ability to prioritize effectively, think creatively, and adapt quickly to new technologies and feedback.


Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →