Senior Software Engineer - Data

We're hiring a Senior Data Platform Engineer to build and operate the data platform that makes CertifyOS the source of truth for provider data. You'll own the reliability and performance of our pipelines and storage, and work closely with engineering, product, and ML.

Responsibilities:

• Build and operate distributed real-time and batch pipelines in Python and PySpark (Kafka, Kinesis, SNS/SQS).
• Contribute to schema, storage, and partitioning decisions across OLTP, OLAP, and NoSQL stores (Aurora PostgreSQL, Redshift, MongoDB).
• Own data quality, contracts, and observability for the pipelines you build.
• Profile and tune pipelines and queries for throughput, latency, and cost.
• Contribute to entity resolution and MDM systems for provider data.
• Build the data foundations for AI/ML workloads.
• Partner with product, analytics, and ML teams on the data they depend on.
• Join the on-call rotation and turn incidents into systemic fixes.

Requirements:

• 5+ years building production data systems.
• Strong Python and hands-on PySpark.
• Solid distributed systems fundamentals: consistency, partitioning, idempotency, failure modes.
• Shipped performance and scale improvements to pipelines or queries.
• Experience across OLTP (Aurora, PostgreSQL, MySQL), OLAP (Redshift, Athena, Snowflake, ClickHouse), and NoSQL (MongoDB, DynamoDB, Cassandra).
• Event-driven architecture (Kafka, Kinesis, SNS/SQS).
• Data observability in practice: quality checks, lineage, monitoring, alerting.
• Sound data modelling fundamentals.
• Communicates technical trade-offs clearly.
• Experience handling sensitive data (PII/PHI).
• Fluent with AI-assisted development (Cursor, Claude Code, MCP).

Nice to Have:

• Entity resolution or MDM (matching, deduplication, survivorship).
• Change Data Capture (CDC) patterns.
• Serving ML workloads: feature pipelines and training/inference data readiness.
• Schema registries or data catalogs.
• Orchestration tools (AWS Glue, EMR, Managed Flink, Airflow/MWAA, dbt).
• Healthcare or other regulated environments.

Technologies and Tools:

• OLTP: Aurora, RDS PostgreSQL
• OLAP: Redshift, Athena
• NoSQL: MongoDB Atlas
• Streaming: Kafka (MSK), Kinesis, SNS/SQS
• Python and PySpark
• AWS, Docker, Kubernetes (EKS)
• GitHub Actions
• Sentry and data observability tooling
• Cursor, Claude Code, Codex