Skip to content
Search jobs inside your AI agent. Claude, ChatGPT, Claude Code, Codex, Gemini, Cursor and Grok
Connect

GoComet

Lead Data Engineer

Posted Updated

Description

The core responsibilities for the job include the following:


Data Platform (Core):

  • Architect and scale a modern data lake supporting both batch and real-time (streaming) workloads.
  • Design and manage reliable data ingestion pipelines.
  • Data modeling and governance.
  • Data quality and lineage.
  • Build and maintain a scalable feature store for ML and AI systems.
  • Develop a robust semantic layer that enables analytics, experimentation, and AI use cases.


ML and Data Science:

  • Own the complete ML lifecycle from training and validation to deployment, monitoring, and retraining.
  • Build and operationalize predictive and optimization models.
  • Establish experimentation frameworks and model evaluation standards.
  • Implement drift detection, monitoring, and performance tracking to ensure model reliability.


AI Readiness:

  • Design and deploy RAG (Retrieval-Augmented Generation) pipelines.
  • Build and manage embeddings infrastructure.
  • Structure and maintain AI-ready datasets optimized for LLMs and agent-based systems.
  • Implement feedback loops that enable continuous learning and system improvement.


What You'll Build:

  • Central data lake (events, operational, external).
  • Real-time pipelines (Kafka / PubSub / streaming).
  • ML pipelines (forecasting, anomaly detection, optimization), feature stores, experiment tracking, data observability, decision intelligence dashboards, and AI-ready datasets.


Architecture First:

  • Define the complete data architecture: source, ingestion, storage, transformation, and serving.
  • Design and optimize offline vs. real-time processing strategies.
  • Make informed cost vs. performance trade-offs.


ML in Production:

  • Model deployment pipelines.
  • Versioning.
  • Monitoring.
  • Retraining strategies.


Cross-Functional:

  • Work with the product to define signals.
  • Work with engineering to integrate pipelines.
  • Working with AI teams enables agent intelligence.


Requirements:

  • 6+ years of experience across data engineering, data science, or machine learning roles.
  • Proven experience designing and building data lake or data warehouse architectures in production.
  • Demonstrated ability to ship ML models to production.
  • Strong understanding of data modeling, feature engineering, ML lifecycle, experimentation, and data governance.
  • Hands-on programming experience with Python and SQL.
  • Experience working with modern cloud data stacks on GCP or AWS.

Skills

What Lead Data Engineering jobs ask for — and how much of it you have →
Apply

See also

Data Engineering jobs by country — openings, pay and top skills →