Senior Data Engineer
Description
Let’s breathe life into great tech ideas! With more than 3 000 people globally, Intellias is a company where benchmark technological solutions are born. Join in and take your part in digitalizing the world.
What project we have for you:
Our client is an independent, active global asset manager with over R3 trillion in assets under management. They are executing a firm-wide data strategy to govern, manage and engineer data as a product across more than 70 business-owned sub-domains.
As part of this, they are migrating to Databricks as the foundational layer of their Enterprise Data Platform, adopting a lakehouse architecture built on open formats, declarative pipelines and Unity Catalog.
What you will do:
- Perform the end-to-end delivery of one or more priority data domains on Azure Databricks.
- Design and build medallion (Bronze / Silver / Gold) pipelines using Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming and Delta Lake with Liquid Clustering.
- Register curated data products in Unity Catalog with the correct tags, masks, row filters, lineage and access policies.
- Implement data quality gates at the Silver-to-Gold boundary, and refine rules with domain stewards.
- Package and deploy work using Databricks Asset Bundles through Azure DevOps CI/CD
- Write automated tests and documentation
- Work at AI-as-Collaborator level today with a path to AI-as-Orchestrator, using Claude Code, Databricks Assistant, GitHub Copilot and MCP-connected agent tooling to accelerate the recurring parts of pipeline build, test and deploy.
What you need for this:
- 6+ years of data engineering experience, with at least the last three years on Databricks in production.
- Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.
- Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.
- Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.
- Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.
- CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).
- Fluent working English.
Nice to have
- Active Databricks certifications
- Prior exposure to investment management platforms, asset management operations data, market data feeds from major providers
- CDMP DAMA certification or equivalent data governance credential.
- Experience applying agentic engineering tooling to production data engineering (not just personal productivity), including MCP-connected Databricks or cloud servers.
What it’s like to work with us:
We are committed to being an equal opportunity employer, fostering equity, diversity, and inclusion. We welcome and celebrate the differences of all qualified applicants. Join our team for a career where your unique perspectives are not only valued but crucial to our success.
Skills
Sign in to read and join the discussion.