Senior Data Engineer – Data Platform & Web Scraping

Senior Data Engineer

Location: Worldwide (Remote)

Employment Type: Full-time

Experience Level: Mid-Senior / Senior

Salary: Up to 100k Year

Exclusive FitNext: No

About the Company

We’re partnering with a growing data platform that helps investors and business teams discover, research, and understand companies, investors, funding activity, and key people through reliable, enriched data.

The platform aggregates information from multiple sources to deliver accurate, actionable insights to its users.

The Challenge

We’re looking for a Senior Data Engineer to take ownership of our data platform, from architecture and implementation to production reliability.

You’ll help evolve existing data extraction and enrichment workflows into a scalable, event-driven platform capable of processing high volumes of data concurrently. You’ll build reliable pipelines that ingest information from APIs, webhooks, files, and web scraping, ensuring data remains accurate, traceable, and easy to reprocess.

This is an opportunity to take ownership of core data infrastructure and shape how a growing platform collects, transforms, and delivers data.

What You’ll Do

• Build Multi-Source Data Pipelines: Develop concurrent ingestion systems for provider APIs, webhooks, files, and web scraping, with appropriate rate limits and failure handling.
• Orchestrate Data Workflows: Build and maintain scheduled workflows, historical backfills, and event-driven processing using tools such as Airflow, Dagster, or Prefect.
• Design Scalable Data Infrastructure: Implement asynchronous processing, task queues, retries, dead-letter queues, and controlled replay of failed tasks.
• Build Reliable Data Layers: Preserve raw source payloads in Google Cloud Storage (GCS) and transform them using dbt into standardized and curated data layers.
• Ensure Data Integrity: Implement idempotent upserts, safe retries, and recovery mechanisms to prevent duplicate records and data corruption.
• Own Data Quality: Establish data contracts, automated validation, pipeline testing, monitoring, and freshness checks.
• Optimize Data Storage: Design and optimize PostgreSQL schemas, queries, and loading patterns to support reliable downstream products.
• Integrate Enrichment and AI: Incorporate third-party data enrichment and LLM-based structured extraction into production workflows.

Must-Have

• Experience building production data platforms from zero to one, owning architecture, implementation, deployment, and ongoing operations.
• Production experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal. Airflow experience is preferred.
• Hands-on production experience with dbt and Google Cloud Platform (GCP).
• Experience building event-driven systems using messaging or task queues, including asynchronous processing and failure recovery.
• Strong Python skills, including asynchronous HTTP requests and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
• Strong SQL and PostgreSQL skills, including data modeling, query optimization, and incremental data loading.
• Experience implementing data contracts, raw and curated storage layers, retention policies, and automated data quality checks.
• Hands-on web scraping experience with tools such as Scrapy, Playwright, Selenium, or Beautiful Soup, including concurrent scraping, rate limits, and failure handling.

Nice-to-Have

• Experience working in startups or small engineering teams, with strong ownership and the ability to adapt to changing priorities.
• Entity resolution experience, including matching company and people records across multiple sources and reconciling duplicates.
• Experience with LLM-based structured extraction or company and people data enrichment.
• Familiarity with BigQuery and Dataform.

Work Setup

• Location: Worldwide — fully remote, with no required location.
• Employment: Full-time.
• Working Hours: No required working-hours overlap.

Selection Process

The selection process will be open to shortlisted candidates.