Member of Technical Staff - Web Crawl Engineer
Posted 3
views
Summary
Builds and operates web-scale crawling infrastructure: URL discovery, prioritization, scheduling, content extraction, and dataset delivery. Works with distributed systems like Ray, Spark, Beam, or Flink, plus HTML parsing/browser rendering, and builds observability and reliability tooling for petabyte-scale crawl pipelines.
You will build and operate systems for web-scale crawling, URL discovery, scheduling, content extraction, and dataset delivery. You will optimize crawl quality and coverage, develop specialized crawlers, create monitoring and reliability systems, analyze performance, and resolve production issues.
Responsibilities
- Build and operate web-scale crawling infrastructure
- Design and optimize URL discovery, prioritization, scheduling, and crawl orchestration
- Develop distributed crawlers that respect site constraints and operational requirements
- Build content extraction, rendering, parsing, and normalization systems
- Improve crawl coverage, freshness, efficiency, and quality through experimentation
- Design recrawling, change-detection, and incremental-update infrastructure
- Develop specialized crawlers for high-value domains and dynamic websites
- Analyze crawl performance and web coverage
- Build observability, monitoring, and reliability systems
- Debug production issues and improve crawling infrastructure
Requirements
- Experience building large-scale web crawling, search indexing, content acquisition, or internet-scale data collection systems
- Understanding of crawling architectures, URL frontier management, scheduling, and distributed crawl coordination
- Experience with Ray, Spark, Beam, Flink, or similar distributed systems
- Familiarity with content extraction, HTML parsing, browser automation, rendering systems, and web technologies
- Experience operating petabyte-scale data-processing systems
- Systems engineering skills in reliability, observability, performance optimization, and debugging
- Experience designing experiments to improve crawl quality, coverage, and efficiency
- Communication skills
Benefits
- Stock options
- Medical, dental, vision, and life insurance
- Annual wellness allowance
- Daily office lunch and dinner
- 22 weeks of paid parental leave
- Unlimited paid time off in the U.S.
- 30 vacation days in the U.K.
- Visa sponsorship support
- Regular off-sites, happy hours, and team celebrations