Data Engineer (GYG4706)
Summary
Build and maintain Databricks-based pipelines for anomaly detection and KPI monitoring, using Azure, MLflow, and streaming/batch processing to support predictive maintenance in a climate-tech context.
Information
Start: ASAP
Period: 6 months with possible extension
Remote: full
Project focus: Databricks-based central operation validation & ML-based anomaly detection vs threshold monitoring
Task
Streaming / Batch pipeliens for integration, Data Quality, Quality Gating, Transformations, Auditing
Feature Engineering, Data Exploration
Align with Operators from Bas, requesting/adjusting KhZ waveform data, uniformisation (data)
KPIs PoC evaluation
IaC asset bundling
Unit Testing / integration testing
Docs formatting
Map business pages
Identify data sources
Find domain expert colleagues
Clarify ownership
Analyze ime series
Regime normalization
Predictive maintenance journey
Anomaly detection
Requirement
Azure
Databricks
CI/CD
MLflow knowledge
Anomaly detection background
DLT
minor: Snowflake