Data platform engineer for autonomous vehicles

Описание:

We are seeking a Data Platform Engineer to join a co-innovation program building the data and MLOps foundation for collision mitigation safety systems on autonomous mining and construction vehicles, for a global heavy-equipment manufacturer and long-standing pioneer in autonomous machinery. In this role, you will work alongside ML teams to design and own a multi-tier AWS data platform that feeds safety and foundation model training, ingesting sensor telemetry, simulation outputs and annotated datasets at scale

Задачи:

Design and implement a medallion data architecture on AWS S3 with IAM-enforced tier promotion rules, replacing manual ingestion triggers with event-driven pipelines and data discovery logic that guarantees complete capture. Build an adapter layer that converts heterogeneous sources into a canonical format preserving temporal metadata, sensor calibration and spatial context, plus a gateway for large-scale real-world telemetry with automated PII scrubbing and sensor frequency alignment. Deploy a centralized metadata registry with immutable dataset versioning and branching per tier, enforce episode-level partitioning for strict train/validation splits, and establish provenance chains from raw data through processed datasets and training runs to model artifacts. Build automated schema validation gates and configurable circuit breakers that stop corrupted data from advancing between tiers, and a unified dashboard tracking ingestion and rejection rates, validation failures, augmentation throughput and latency. Integrate a data augmentation pipeline as a managed flow stage with full lineage from source data through augmentation parameters to output, and define with ML teams the contracts for schema completeness, temporal windowing, annotation quality thresholds and approval gates. Prove the end-to-end toolchain, from data ingestion to safety model training, on a first pilot workflow such as wheel-loader stockpile management, then scale it as the foundation for further autonomy use cases.

Требования:

• 5+ years of experience building production-grade data pipelines and data platforms.
• Deep expertise with S3, EventBridge, Step Functions, FSx for Lustre, CloudWatch and IAM, with IaC proficiency and GitOps practices.
• Hands-on experience with Ray or Spark and strong Python for building ETL/ELT pipelines at scale.
• Solid knowledge of columnar and binary formats (Parquet, HDF5) and event-driven ingestion architectures.
• Practical experience with LakeFS, DVC, Delta Lake or equivalent dataset versioning and lineage tools.
• Proven experience designing schema validation, circuit breakers and rejection-rate monitoring.
• Familiarity with MLflow or an equivalent artifact registry and experiment tracking system.
• Будет плюсом: Experience with autonomous systems data formats such as ROS bags, LiDAR, IMU and camera streams; Prior work with medallion or lakehouse architectures; Exposure to tools such as NVIDIA Cosmos, Stable Diffusion-based pipelines or similar; Experience in regulated or safety-critical data environments such as automotive, aerospace or industrial autonomy

Условия:

Opportunity to work on bleeding-edge projects Work with a highly motivated and dedicated team Competitive salary Flexible schedule Benefits package - medical insurance, sports Corporate social events Professional development opportunities Well-equipped office