Data Engineer
Summary
Data engineer owning the data layer behind Derin, Surgo Health's AI-powered conversational interview platform: building Python/SQL pipelines into a Databricks lakehouse, modeling survey data, and maintaining the Postgres serving layer. Heavy use of Databricks, Azure, Terraform, and AI coding tools like Claude Code and Cursor.
- Work AI-first: use AI agents (Claude Code, Cursor and others) throughout your day to design, build, debug and optimize, and help the team keep raising the bar on AI-assisted engineering
- Own the Databricks data platform behind Derin, from ingestion of interview and survey data to the analytics-ready tables our product, scientists and clients rely on
- Design and build reliable batch and incremental pipelines on Databricks (Lakeflow Spark Declarative Pipelines, formerly Delta Live Tables; Jobs; Delta Lake) and govern them in Unity Catalog
- Help move our production data layer to Databricks Lakebase (managed Postgres), including synced tables, migration and cutover, and keep the serving layer fast for our APIs
- Model survey and conversation data in Delta Lake so it is easy to access for downstream analytics products
- Make pipelines observable and trustworthy: data-quality checks (such as Lakeflow expectations), tests, Unity Catalog lineage, monitoring and alerting
- Turn scientists' Databricks notebooks and analyses into production-grade, reusable pipelines
- Build privacy and security into how we handle sensitive participant data, using Unity Catalog access controls, row filters and column masks, in a SOC2 and HIPAA-compliant environment
- Use infrastructure as code (Terraform, Databricks Asset Bundles) and CI/CD to deploy data workloads across environments
- Contribute to shaping our engineering culture and data platform as we grow
- You have at least 3 years of experience in data engineering, building production pipelines in Python and SQL
- You have hands-on production experience with Databricks (Spark, Delta Lake, Unity Catalog, Lakeflow or Jobs)
- You are strong with relational databases, particularly PostgreSQL (including Databricks Lakebase): schema design, query optimization, replication or CDC
- You have hands-on experience with Microsoft Azure (required) and with infrastructure as code and CI/CD
- You have modelled data for analytics and BI (dimensional or long/wide formats) and care about data quality and testing
- Experience serving data to LLM-powered features (text-to-SQL, RAG, Databricks metric views, Genie or other semantic layers) is a strong plus
- Experience with survey, research or healthcare data, or with regulated data (HIPAA, GDPR/KVKK), is a plus
- You are AI-native: you use agentic tools such as Claude Code and Cursor every day to design architectures and diagrams, write and review code, debug and improve performance. You know how to set them up (context, rules, MCP servers, subagents) and when not to trust their output. Surgo is an AI-driven company that uses AI for almost everything we do, not only code, and our interview process asks you to show how you work with it
- You hold a University degree in Computer Science or a related field
- You are independent, proactive, and able to deliver end-to-end with minimal supervision
- You are fluent in English, both written and spoken, and comfortable working remotely with substantial overlap with US Eastern and UK hours
- Fantastic, fun and collaborative colleagues; opportunity to solve tough challenges in healthcare
- Ability to be entrepreneurial and tremendous opportunities for growth
- Public Benefit Corporation with an impact and equity mission
- Competitive compensation
- Professional development opportunities