Senior Data Engineer

Responsibilities:

• Multimodal Ingestion and Preprocessing: Build and maintain pipelines that ingest real-world audio, egocentric video, and image data at scale, covering format normalization, chunking, metadata extraction, and landing into S3-based storage with consistent schema and partitioning.
• AWS Data Lake: Manage and extend the KGen data lake: Athena query optimization, Glue crawlers and cataloguing, Apache Hudi table management, Lake Formation column-level permissions, and S3 lifecycle policies at TB-PB scale.
• GPU Validation Pipeline Handoff: Design and maintain the data layer that feeds GPU-backed technical validation workers; own sharding strategies, manifest generation, throughput optimization, and I/O design so validation compute is never the bottleneck. Understand how data format choices (Parquet, WebDataset, sharded archives) affect GPU-side loading performance.
• Airflow DAG Management: Author, debug, and monitor Airflow DAGs for scheduled processing, GPU job orchestration, and pipeline coordination across ingestion, validation, and delivery stages.
• QC and Annotation Tooling: Support the FastAPI-backed audio QC portal used by annotation workers; extend data validation and quality-check scripts across egocentric video and audio datasets.
• Universal Data Schema (UDS): Contribute to and enforce the Universal Data Schema for audio, image, and code modalities in the Humyn Labs dataset marketplace, covering schema evolution, versioning, and partition strategies.
• Infrastructure and Access Management: Maintain AWS IAM, Lake Formation, and S3 bucket policies; manage data engineer access controls; handle cross-region data movement and vendor data sharing infrastructure.
• ETL and Third-Party Integrations: Build and maintain ingestion pipelines from APIs (Twitch, gaming analytics, and Google Forms) into DynamoDB and PostgreSQL where required by business context.