Senior Databricks Data Engineer - Oil & Gas
Posted 1
view
Summary
Senior Databricks Data Engineer building Azure Databricks data pipelines and production AI/ML solutions for oil & gas client projects — sensor/telemetry pipelines, anomaly detection, and natural-language data interfaces. Core stack: Azure Databricks, Spark, Python, SQL, and modern LLM tooling.
We are looking for a Senior Databricks Data Engineer to join our team.
This role is ideal for someone who enjoys building end-to-end solutions and wants to work on real-world industrial use cases that comes from an Engineering background, and has experience in Oil & Gas.
In this position, you’ll contribute to a variety of impactful client projects, including:
Building AI-powered anomaly detection systems for operational and industrial data
Modernizing asset management workflows
Developing natural language interfaces for querying enterprise and operational data
Implementing Azure Databricks foundations and pipelines with best practices
Creating scalable data pipelines and ML workflows in cloud environments
Enabling real-time and batch data processing for analytics and AI use cases
Working through complex and challenging data conditions, including incremental processing, late-arriving or changing records, complex business and engineering rules, and reconciliation of results across processing runs
The ideal candidate combines strong data engineering experience with an engineering or applied-science background. Direct experience industrial time-series data, scientific measurements, telemetry, financial reconciliation, or other datasets where accuracy, traceability, and incremental recalculation are critical.
Key Responsibilities
Design, build, and optimize pipelines for sensor and related operational data.
Develop complex transformation and calculation logic based on engineering requirements.
Implement robust incremental-processing patterns for high-volume and continuously changing datasets.
Design, build, and deploy end-to-end AI/ML solutions in production environments
Develop robust backend systems and APIs to support AI-driven applications
Build and maintain data pipelines and feature engineering workflows
Implement and operationalize machine learning models (training, deployment, monitoring)
Work with modern AI tooling (LLMs, agents, orchestration frameworks)
Collaborate with clients to translate business problems into technical solutions
Contribute to architecture decisions and best practices across projects
Mentor client team members and contribute to internal capability building
Work directly with engineering and operational SMEs to understand physical processes and translate their knowledge into technical requirements.
Make engineering calculations and data transformations explainable, traceable, testable, and auditable.
Document data lineage, calculation logic, assumptions, dependencies, and exception-handling rules.
Ideal Background
Senior-level experience designing and developing production data pipelines with Azure Databricks including strong experience with complex SQL, Python, Spark, or comparable data-processing technologies.
Complex Excel and CSV integration experience
Experience with Databricks dashboards and Genie
Strong requirements gathering experience with Field Engineers and Technical SMEs
Demonstrated experience with incremental processing, change detection, reconciliation, and pipeline design
Experience handling time-series, telemetry, sensor, operational, scientific or industrial data
Medallion architecture and modelling
Strong analytical and investigative skills, with the patience to work through detailed logic and difficult data-quality problems.
Degree or professional background in petroleum, reservoir, chemical, mechanical, geological, geophysical, or another relevant engineering or applied-science discipline is strongly preferred.
Experience in upstream oil and gas, thermal operations, SAGD, well surveillance, production engineering, or subsurface data would be a significant asset.
This role presents an exciting opportunity to work on practical, high-impact AI use cases - not just prototypes, shape how AI is applied to client environments, and change the game on traditional processes and platforms. Come join a growing organization helping clients take a new, lean and value-driven approach to data and engineering!
This role is ideal for someone who enjoys building end-to-end solutions and wants to work on real-world industrial use cases that comes from an Engineering background, and has experience in Oil & Gas.
In this position, you’ll contribute to a variety of impactful client projects, including:
Building AI-powered anomaly detection systems for operational and industrial data
Modernizing asset management workflows
Developing natural language interfaces for querying enterprise and operational data
Implementing Azure Databricks foundations and pipelines with best practices
Creating scalable data pipelines and ML workflows in cloud environments
Enabling real-time and batch data processing for analytics and AI use cases
Working through complex and challenging data conditions, including incremental processing, late-arriving or changing records, complex business and engineering rules, and reconciliation of results across processing runs
The ideal candidate combines strong data engineering experience with an engineering or applied-science background. Direct experience industrial time-series data, scientific measurements, telemetry, financial reconciliation, or other datasets where accuracy, traceability, and incremental recalculation are critical.
Key Responsibilities
Design, build, and optimize pipelines for sensor and related operational data.
Develop complex transformation and calculation logic based on engineering requirements.
Implement robust incremental-processing patterns for high-volume and continuously changing datasets.
Design, build, and deploy end-to-end AI/ML solutions in production environments
Develop robust backend systems and APIs to support AI-driven applications
Build and maintain data pipelines and feature engineering workflows
Implement and operationalize machine learning models (training, deployment, monitoring)
Work with modern AI tooling (LLMs, agents, orchestration frameworks)
Collaborate with clients to translate business problems into technical solutions
Contribute to architecture decisions and best practices across projects
Mentor client team members and contribute to internal capability building
Work directly with engineering and operational SMEs to understand physical processes and translate their knowledge into technical requirements.
Make engineering calculations and data transformations explainable, traceable, testable, and auditable.
Document data lineage, calculation logic, assumptions, dependencies, and exception-handling rules.
Ideal Background
Senior-level experience designing and developing production data pipelines with Azure Databricks including strong experience with complex SQL, Python, Spark, or comparable data-processing technologies.
Complex Excel and CSV integration experience
Experience with Databricks dashboards and Genie
Strong requirements gathering experience with Field Engineers and Technical SMEs
Demonstrated experience with incremental processing, change detection, reconciliation, and pipeline design
Experience handling time-series, telemetry, sensor, operational, scientific or industrial data
Medallion architecture and modelling
Strong analytical and investigative skills, with the patience to work through detailed logic and difficult data-quality problems.
Degree or professional background in petroleum, reservoir, chemical, mechanical, geological, geophysical, or another relevant engineering or applied-science discipline is strongly preferred.
Experience in upstream oil and gas, thermal operations, SAGD, well surveillance, production engineering, or subsurface data would be a significant asset.
This role presents an exciting opportunity to work on practical, high-impact AI use cases - not just prototypes, shape how AI is applied to client environments, and change the game on traditional processes and platforms. Come join a growing organization helping clients take a new, lean and value-driven approach to data and engineering!