Senior Data Infrastructure Engineer (Spatial Data Platform)
Summary
Builds, scales, and operates a petabyte-scale data lakehouse on AWS behind spatial data products: ingesting and versioning robotics/headset/video captures, running distributed CPU/GPU processing, and feeding GPU clusters for 3D reconstruction (SLAM, Gaussian Splats) and ML training. Core stack includes Spark, Ray, Flink, Iceberg, LakeFS, and Terraform.
Our client is looking for a hands-on Senior Data Infrastructure Engineer to build, scale, and operate the petabyte-scale data lakehouse behind our spatial data products.
Every field and lab capture—robotics, egocentric headsets, multi-camera video, depth, IMU—lands in this platform, gets versioned, and powers downstream 3D reconstructions (Gaussian Splats, SLAM) and ML training clusters.
This is a true engineering role. We need someone who has personally built and run a PB-scale platform in production, not just used one.
Key Responsibilities
- PB-Scale Lakehouse: Design object storage layouts (S3), partitioning strategies, catalog architectures, and open table formats (Iceberg/Delta).
- Data Versioning & Ingest: Build reproducible dataset lineage (LakeFS) and high-throughput ingest pipelines (validation, checksumming, deduplication, bronze $→$ gold promotion).
- Distributed CPU/GPU Compute: Operate large-scale processing (Spark, Ray, Flink) for frame extraction, sync/calibration, SLAM/3DGS batch jobs, and auto-labeling.
- ML Training Integration: Keep GPU clusters fed using sharded training formats (WebDataset, Mosaic Streaming), caching, and optimized cross-region transfers.
- Curation & Querying: Build indexing, search, and high-performance SQL query layers (Trino, DuckDB, Spark SQL) so researchers can slice data without copying it.
- Reliability & Infrastructure: Maintain IaC (Terraform) on AWS, handle pipeline backfills/recovery, optimize storage/egress costs, and participate in on-call rotation.
Qualifications
- Experience: 6+ years in data infrastructure or distributed systems engineering.
- Scale: Proven track record setting up and operating a live petabyte-scale data platform (able to speak concretely to throughput, object limits, and failure modes).
- Languages: Python + at least one of Go, Rust, C++, Java, or Scala.
- Table Formats & Versioning: Iceberg/Delta Lake, Catalogs (Glue/Nessie), LakeFS or content-addressed storage.
- Processing & Orchestration: Spark, Ray, or Flink; Airflow, Argo, or Prefect.
- Cloud/IaC: Strong AWS (S3, EKS/ECS, IAM) and Terraform experience.
Nice to Have
- Experience with unstructured/multimodal spatial data (video, LiDAR, point clouds, ROS bags/MCAP).
- GPU-accelerated pipelines (NVIDIA DALI, NVDEC, RAPIDS) or ML data loaders (WebDataset, Petastorm).
- Query engines (Trino, DuckDB, ClickHouse) and embedding/vector search.
