Data Engineer

We are hiring engineers to build enterprise data platforms, Kafka-based streaming systems, source integrations, and data infrastructure for AI applications and agents.

You will both build reusable platform capabilities and deploy them in enterprise environments, owning integration design, implementation, testing, deployment, and production operations. The role emphasises software engineering, distributed systems, and platform development, with SQL as a supporting skill.

Responsibilities:

• Design source integrations: Assess enterprise systems and select ingestion patterns based on source interfaces, data volumes, freshness requirements, rate limits, and operational constraints.
• Build reusable connectors: Implement authentication, pagination, source-to-target mapping, validation, incremental state, retries, and recovery across databases, APIs, applications, files, and streams.
• Engineer reliable streaming: Develop Kafka-based pipelines with appropriate partitioning, state management, monitoring, replay, and failure handling.
• Build lakehouse capabilities: Develop pipelines and data products using S3, Iceberg, Glue, Spark, and Athena, including incremental updates, schema changes, and backfills.
• Enable AI consumption: Expose governed enterprise data and context through APIs, retrieval pipelines, and MCP interfaces.
• Automate deployment: Provision infrastructure and deploy pipelines using Terraform, containers, and GitLab CI/CD.
• Own production quality: Implement automated tests, reconciliation, data quality checks, observability, security controls, and recovery procedures.
• Deliver as an FDE: Work directly with enterprise teams to understand unfamiliar systems, resolve integration challenges, and own implementation through operational handover.
• Improve the shared platform: Turn recurring project needs into reusable services, connectors, SDKs, and deployment templates.
• Apply the Software Factory: Use and improve our engineering platform to accelerate implementation, testing, and deployment while remaining accountable for correctness.

The core requirements for the job include the following:

Core skills:

• Software engineering: Python, object-oriented programming (OOP), data structures, modular design, asynchronous programming, concurrency, automated testing, and debugging.
• Backend services: FastAPI, REST APIs, API contracts, authentication, validation, error handling, and reusable service components.
• Apache Kafka: Producers, consumers, topics, partitions, message keys, offsets, consumer groups, rebalancing, ordering, and consumer lag.
• Streaming reliability: Delivery semantics, idempotency, deduplication, retries, dead-letter handling, replay, backpressure, and schema evolution.
• Source integrations: Relational databases, REST APIs, enterprise applications, webhooks, Kafka streams, files, and object storage.
• Ingestion design: Batch and incremental ingestion, CDC, initial snapshots, pagination, cursor management, rate limiting, token refresh, checkpoints, backfills, and reconciliation.
• Distributed systems: Event-driven architecture, state management, partial failures, consistency, fault tolerance, horizontal scaling, throughput, and latency.
• Data platforms: Data lakes, lakehouse architecture, ingestion and transformation pipelines, metadata catalogues, data contracts, data quality, and lineage.
• AWS: S3, IAM, VPC, CloudWatch, Secrets Manager, and deployment using EKS, ECS/Fargate, or Lambda.
• Infrastructure and delivery: Terraform, Docker, Git, GitLab CI/CD, environment configuration, automated deployment, and production troubleshooting.
• Database fundamentals: SQL, PostgreSQL or MySQL, schema design, transactions, indexing, and query performance.

Relevant technology experience:

You do not need every technology listed below. Depth in software engineering, source integration, Kafka, and production deployment is especially important.

• CDC and connectors: Debezium, Kafka Connect, source/sink connectors, database change logs, and snapshot-to-stream transitions.
• Stream processing: Apache Flink, Flink SQL, Apache Spark, PySpark, Spark Structured Streaming, event time, watermarks, windowing, and stateful processing.
• Event contracts: Avro, Protobuf, JSON Schema, schema registries, and backwards-compatible schema evolution.
• Lakehouse: Apache Iceberg, Parquet, Amazon S3, AWS Glue Data Catalogue, Amazon Athena, compaction, partitioning, and small-file optimisation.
• Orchestration: Apache Airflow, Dagster, AWS Step Functions, or Temporal.
• Kubernetes: Amazon EKS, Helm, workload configuration, scaling, resource management, and troubleshooting.
• Databases and serving: PostgreSQL, MySQL, Amazon RDS, Aurora PostgreSQL, DynamoDB, and low-latency data APIs.
• AI data infrastructure: RAG pipelines, document ingestion, chunking, embeddings, vector databases, OpenSearch, and pgvector.
• Agent integration: Model Context Protocol (MCP), tool APIs, dataset discovery, governed query interfaces, and agent-accessible data services.
• Ontology and semantic data: Experience with ontologies, knowledge graphs, metadata management, and mapping business concepts to enterprise datasets is good to have.
• Developer tooling: SDKs, CLI tools, reusable connector frameworks, pipeline templates, Terraform modules, and Copier templates.
• Security and governance: OAuth2/OIDC, RBAC, least-privilege IAM, encryption, audit logging, and AWS Lake Formation.
• Observability: OpenTelemetry, Prometheus, Grafana, CloudWatch, pipeline freshness, consumer lag, throughput, and data quality monitoring.

Relevant candidate backgrounds:

• Data Platform Engineer
• Data Infrastructure Engineer
• Streaming Data Engineer / Kafka Engineer
• Software Engineer - Distributed Systems
• Backend Engineer with data infrastructure experience
• Platform Engineer with data pipeline and AWS experience