Data Engineer
Posted Updated
Description
Requirements:
- Data Engineering (Core): Deep expertise in distributed stream processing frameworks: Apache Kafka, Apache Flink, or Apache Spark Structured Streaming. Strong command of SQL and experience optimising complex analytical queries at petabyte scale. Hands-on experience with real-time analytical databases, particularly StarRocks; familiarity with Snowflake or ClickHouse is a plus. Proficiency with workflow orchestration tools: Apache Airflow, Prefect, or Dagster. Solid understanding of data modelling, dimensional modelling, OBT patterns, and event-driven schemas relevant to ad impression and attribution data. Familiarity with open table formats: Apache Iceberg, Delta Lake, or Apache Hudi.
- Backend and Systems Proficiency in Python (primary) and/or Java/Scala for building production-grade services and data processing jobs. Experience designing and building RESTful APIs and/or gRPC-based microservices. Working knowledge of low-latency data stores (Aerospike, Cassandra) used for real-time lookups in the bidding path. Comfortable with containerised deployments on Kubernetes and CI/CD pipelines (GitHub Actions, ArgoCD, or similar).
- Ad-Tech Domain Knowledge: Understanding of the RTB ecosystem, DSPs, SSPs, ad exchanges, bid request/response lifecycle, and OpenRTB protocol. Familiarity with identity resolution approaches: cookie IDs, device IDs, UID2 / RampID, and privacy-preserving identity. Exposure to audience segmentation, frequency capping, pacing algorithms, or attribution models is a significant plus. Awareness of privacy regulations (GDPR, CCPA) and their implications for data pipelines and user-level data handling.
Skills
- Airflow
- API
- Argo CD
- Cassandra
- Ccpa
- CI/CD
- ClickHouse
- Dagster
- Data Engineering
- Data Modeling
- Data Pipelines
- Delta Lake
- Dimensional Modeling
- Event Driven Architecture
- Flink
- Gdpr
- GitHub
- GitHub Actions
- gRPC
- Iceberg
- Java
- Kafka
- Kubernetes
- Microservices
- Prefect
- Python
- REST
- Scala
- Snowflake
- Spark
- SQL
- Workflow Orchestration
Sign in to read and join the discussion.