Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

Purplle

Data Engineer

Posted Updated

Description

Responsibilities:

  • AI-Augmented Pipeline Architecture: Design and build self-healing, schema-validated data pipelines on GCP (BigQuery, Dataflow, Pub/Sub) that power real-time analytics. Pipelines must be idempotent, observable, and auto-recover from failures without human intervention.
  • Streaming and Batch Processing at Scale: Implement and optimise large-scale processing workflows using Apache Spark, Flink, or Beam. Own both real-time Kafka consumer pipelines and scheduled batch jobs, ensuring sub-hour SLAs for data freshness across all critical business domains.
  • Schema Contracts and Data Integrity: Implement and enforce schema registry and data contract standards across all producers and consumers: target 100% data quality scores, no silent failures, and no downstream surprises.
  • ML Pipeline Integration: Build and maintain feature stores and data pipelines that feed production ML models (recommender systems, demand forecasting, pricing). Collaborate with data science to ensure training-serving skew is eliminated.
  • Data Governance and Cost Engineering: Champion data cataloguing, lineage tracking, and access-control policies. Monitor BigQuery slot usage and pipeline costs, and implement optimisations that reduce cloud spend by measurable percentages each quarter.


Requirements:

  • DE2 3-6 years; end-to-end ownership of production pipelines at scale.
  • Ability to ship data products independently with clear quality and latency guarantees.


Tech Stack:

  • Languages: Python, Scala, Java, SQL.
  • Processing: Apache Spark, Apache Beam / Dataflow, Apache Flink, Kafka, Databricks.
  • Cloud and Storage: GCP (BigQuery, Dataflow, Pub/Sub, DataProc, and GCS) and AWS (S3 and EMR).
  • Orchestration: Apache Airflow (Cloud Composer).
  • Data Quality: Great Expectations, dbt.
  • Infrastructure: Kubernetes, Docker, Terraform, Databricks.
  • AI Productivity: Cursor, Claude.

Skills

Apply

See also

Data Engineering jobs by country — openings, pay and top skills →