Data Engineer
Summary
Senior data engineer (3-6 month contract) migrating a client's finance and HR data pipelines from Matillion ETL to Airflow + dbt + Snowflake using a data-mesh approach: reverse-engineering undocumented pipelines, building Airflow DAGs and dbt models/tests/docs, and reconciling old vs new reports. Remote within Sweden, with a Stockholm office available.
Our client is looking for 1-2 Senior and selfgoing Data Engineers to help them migrate their finance and HR data domains. The assignment can be done remote, but there is a office in Sweden/Stockholm if you would like to visit.
The assignment:
We have a set of ETL jobs loading and transforming financial data using ETL tool Matillion and Snowflake as the database. The source is in Azure data lake (backed by dynamics). The goal of the project is to understand how the current system is running, migrate it to a combination of Airflow+dbt+Snowflake jobs while using a data mesh/data products strategy. The migration will not be 1:1 but rather an improvement of the current implementation e.g., currently Row Level Security is used to manage access to sensitive data. The new implementation could possibly use role level access control instead of RLS.
Some of the task would be
Understand and document current pipelines written in Matillion
Write ingestion pipeline using Airflow dag
Simplify and convert the mappings to dbt models (some parts may run in airflow if they have python code snippets)
Write dbt docs to document all models
Write unit tests in dbt for data tests
Help with reconciliation process between the current reports and new reports from the newly developed system
In all of these tasks, the most complex part would be to understand and document the existing implementation as currently there is no documentation and we would need to read the pipelines manually.
Start: Asap
Workplace: Sweden or remote from Sweden.
Length: 3-6 months