Data Engineer (Data Ops)
Summary
Own production data delivery for a recruitment-tech client's AI-powered search and matching platform: building reliable ETL/ELT ingestion pipelines, workflow orchestration, data quality and vector search. Stack includes PostgreSQL/pgvector, VoyageAI embeddings via Amazon Bedrock, and an existing Go migration/orchestration layer.
Overview
We are supporting an established recruitment organisation developing an AI-powered platform. We need a hands-on engineer to improve ingestion and the pipelines feeding production search and matching.
You will own production data delivery, improve reliability and quality, and work alongside specialist technical leadership. The emphasis is on strong ETL/ELT, data modelling and workflow orchestration, supporting production search and matching.
Data migrations run through an existing Go layer, so candidates need sufficient Go understanding to follow the workflows and investigate issues with platform engineering colleagues.
About the Project
- Remove ingestion bottlenecks and make the flow from approved sources to search and matching reliable.
- The stack includes PostgreSQL/pgvector, VoyageAI embeddings via Amazon Bedrock, and an existing Go layer driving data migrations and orchestration.
- Work with specialist data and ML expertise, platform engineering, Product and DevOps.
- Follow approved consent and access rules; escalate unresolved decisions.
Requirements
Essential Technical Experience
- Hands-on ownership of production ETL/ELT pipelines, including transformations, incremental processing, failure investigation and recovery.
- Strong SQL and PostgreSQL data modelling.
- Experience managing complex workflow dependencies, retries, partial failures and backfills. DAG-based orchestration using Apache Airflow, Dagster or equivalent tools or approaches is relevant evidence.
- Working understanding of Go sufficient to follow existing migration/orchestration code, trace data flows and collaborate with platform engineering on investigation and changes.
- APIs, batch/asynchronous processing, schema evolution, metadata and lineage.
- Idempotency, reconciliation and safe recovery or reruns.
- Production vector search and embedding pipelines for search or matching.
- Diagnosing input, embedding, indexing and downstream quality issues.
- Cloud operation, automated testing, code review and deployment discipline.
- Monitoring, alerts, runbooks, privacy-aware modelling and access controls.
Desirable Technical Experience
- Direct pgvector and Amazon Bedrock experience is strongly preferred.
- VoyageAI embedding integrations and embedding evaluation.
- Python data-processing experience.
- Entity resolution and search relevance.
Market Experience
- Data pipelines supporting production search, matching or ML products - Required.
- Recruitment Technology / HRTech and recruitment data - Desirable.
Soft Skills and Working Style
- Clear communication and collaboration with Product, Engineering and DevOps.
- Independent investigation, practical problem-solving and attention to detail.
- Ownership of implementation, operational reliability and follow-through.
- Clarify ambiguity, explain decisions, verify results and escalate blockers.
Responsibilities
Pipeline Ownership, ETL/ELT and Orchestration
- Document the existing sources, transformations, processing dependencies, scheduling or triggers, and failure modes during handover.
- Automate ingestion, validation, deduplication and incremental updates from approved sources.
- Make source onboarding repeatable and reduce the validated backlog.
- Manage dependent processing stages, including partial failures, retries and backfills, with clear checks on downstream data correctness.
Recovery, Data Quality and Lineage
- Implement safe retries, checkpoints, replay/backfill and quarantine without silent loss or duplication.
- Reconcile record counts and maintain metadata, source identifiers and lineage.
- Detect schema changes, apply quality checks and investigate failures.
Production Operation and Matching
- Maintain matching-service interfaces and contribute to agreed ML/matching changes.
- Test embedding/indexing consistency, compatibility and matching quality.
- Work with DevOps on logs and alerts for freshness, failures, throughput, latency and cost.
- Maintain tests, deployment discipline and runbooks for independent operation and recovery.
Privacy and Collaboration
- Implement approved consent, access, separation, retention and deletion rules; report unresolved mappings.
- Clarify requirements, report progress and escalate unresolved legal decisions to the client
Assessment Process
1. CV Review
2. Screening Call (30 mins) – Technical screening to understand experience
3. Technical assessment/challenge. (1 hr)
4. Final Interview (30 mins) – Cultural fit, collaboration instincts, MVP mindset
Skills
Sign in to read and join the discussion.