Senior Data Engineer

Role Overview

DeepRiver Soft is looking for a Data Engineer to own and evolve the data platform that powers our iGaming products (casino and sportsbook) across EU and African markets. The platform ingests change-data-capture streams from five core business systems at a target scale of tens to hundreds of millions of events per day, models them into an analytical warehouse, and serves financial, payments, and product analytics to the whole company.

You will be the primary data engineer on the platform, working end-to-end: from Kafka/CDC ingestion, through dbt transformations on ClickHouse, to orchestration with Dagster and dashboards in Grafana and Superset. This role sits on the tech side but collaborates heavily with Infrastructure/DevOps, backend service teams, BI analysts, and external data vendors — ensuring high performance, scalability, and secure data operations.

Key Responsibilities

Data Pipelines & Platform

• Build, maintain, and optimize ELT pipelines for real-time and batch data on our core stack: Debezium CDC → Kafka (Amazon MSK) → S3 data lake (Parquet) → dbt on ClickHouse, orchestrated by Dagster.
• Own production pipeline operations: freshness-tiered scheduling (from 1-minute real-time jobs to daily reports), idempotent incremental loads, backfills, and schema evolution across the CDC flow.
• Extend ingestion to new sources: payment-provider APIs, DynamoDB streams, CRM and product-analytics platforms (Customer.io, MixPanel), and an event-tracking capture layer (Snowplow-style) for user-activity analytics.
• Ensure data reliability, fault-tolerance, and high availability within distributed systems; contribute to load testing, capacity planning, and autoscaling readiness.

Collaboration & Architecture

• Work closely with Infrastructure/DevOps to align pipelines with our AWS/EKS cloud environment, GitLab CI/CD, and GitOps deployments (ArgoCD/Helm).
• Partner with BI analysts and backend teams to define event contracts and data needs, and design scalable models for financial metrics (GGR/NGR, player activity, sportsbook risk), payments, and marketing analytics.
• Integrate and maintain external partner integrations (e.g., affiliate and payment platforms) including reconciliation and delivery guarantees.
• Document data models, pipeline logic, and architectural decisions (ADRs, runbooks, data contracts).

Data Quality, Governance & Security

• Implement best practices for data validation, testing, governance, and lineage (dbt tests and docs, pipeline unit tests, CI quality gates).
• Monitor data-quality KPIs and proactively address issues (late data, duplicates, contract drift, noisy events).
• Ensure compliance with data-security and privacy standards (PII handling, GDPR, access scoping in BI tools) and support upcoming regulatory reporting for gaming authorities.

Required Qualifications

• 4+ years of experience as a Data Engineer (or similar role), with proven end-to-end ownership of production pipelines.
• Strong SQL and analytical data modeling skills (dimensional/medallion architectures); production experience with dbt.
• Strong Python for data engineering (our entire orchestration and integration layer is Python).
• Hands-on experience with a modern orchestrator — Dagster is a strong advantage; solid Airflow/Prefect experience with willingness to switch is fine.
• Experience with streaming technologies — Kafka required; Debezium/Kafka Connect CDC experience is a strong advantage.
• Experience with an analytical/OLAP warehouse — ClickHouse strongly preferred; deep experience with BigQuery, Snowflake, or Redshift is transferable.
• Solid understanding of AWS (S3 data lakes, IAM, EKS/containerized workloads) and comfort with Docker and Kubernetes-deployed services.
• Familiarity with CI/CD workflows and testing culture: pipeline tests (pytest), data-quality gating, code review discipline.

Nice to Have

• Experience in iGaming, betting, fintech, or similar real-time data environments (GGR/NGR, RTP, sportsbook risk, affiliate programs, PSP/payments analytics).
• GitOps deployment experience (ArgoCD, Helm); Terraform exposure.
• Dashboards-as-code with Grafana and/or Apache Superset.
• Event tracking and product analytics: Snowplow, RudderStack, MixPanel, Customer.io; sessionization, attribution, identity resolution.
• DynamoDB; PostgreSQL internals (logical replication/WAL); Confluent Schema Registry/Avro.
• Load testing and capacity planning; observability stacks (Prometheus, Grafana).
• Data governance: GDPR/PII pseudonymization, SSO/RBAC/row-level security in BI tools.

Key Skills

• Strong analytical and problem-solving mindset.
• High autonomy and ownership — you will be the primary engineer on the platform, taking pipelines end-to-end from design to production operations.
• Excellent communication skills to collaborate across engineering, BI, product, and external vendors.
• Proactive approach to reliability, scalability, and documentation (design docs, ADRs, runbooks).