Foundational Data Engineer – Multimodal ML Pipelines
Posted Updated
Description
Harmattan AI is seeking a Data Engineer on the Foundational team in Paris to build massive data infrastructure for deep learning models. You will manage terabytes—soon petabytes—of EO and IR video data, ensuring data pipelines are robust and performant so ML engineers can focus on model design.
You will implement high-throughput loading, synchronization of EO/IR frames, and scalable processing using Spark, Ray, or Beam, while optimizing storage and data quality.
Build ETL/ELT pipelines to ingest EO/IR video into optimized formats. Develop algorithms for EO/IR frame synchronization. Architect storage-to-GPU pipelines for multi-node training clusters. Write distributed data processing jobs with Spark, Ray, or Beam. Implement automated data quality checks and reproducible training with versioning. Evaluate storage solutions (MinIO, S3 tiering) for cost and latency. BS or MS in CS, SE, or distributed systems preferred. 5–6+ years building terabyte-scale pipelines for unstructured data. Proven track record in maximizing multi-node GPU utilisation.Skills
Sign in to read and join the discussion.