Member of Technical Staff, Data Infrastructure
The Role
We seek experienced engineers to architect and scale the core infrastructure behind distributed training pipelines and petabyte-scale data catalogs. You'll work directly with researchers to accelerate experiments, develop new datasets, improve infrastructure efficiency, and enable key insights across our data assets.
Key Responsibilities
- Design, build, and operate scalable, fault-tolerant infrastructure for LLM research: distributed compute, data orchestration, and storage across modalities.
- Develop high-throughput systems for data ingestion, processing, and transformation — including training data catalogs, deduplication, quality checks, and search.
- Build systems for web crawling, data ingestion, and real-time data processing to support model training operations.
- Develop tools and frameworks for efficient data storage, retrieval, and versioning across distributed systems.
- Ensure data collection adheres to privacy regulations.
Qualifications
- BS/MS/PhD in Computer Science, Machine Learning, or a related field (or equivalent experience).
- 3+ years of experience building data processing pipelines at scale, particularly with AI/ML applications.
- Strong proficiency in Python and experience with data processing frameworks (Apache Spark, Beam, Airflow).
- Familiarity with synthetic data generation techniques and data augmentation strategies.
- Familiarity with web scraping, crawling technologies, and Common Crawl datasets.
- Solid understanding of machine learning fundamentals and experience with ML frameworks (PyTorch, TensorFlow).
- Experience with SQL and NoSQL databases for managing structured and unstructured data.
Preferred Skills
- Experience with large language models and understanding of tokenization, embeddings, and model architectures.
- Experience managing human annotation workflows and quality control processes.
- Experience with vector databases and embedding-based retrieval systems.
- Knowledge of data privacy regulations and ethical AI practices.
- Experience with distributed computing and large-scale data storage systems (HDFS, S3, BigQuery).