Remote Data Platform Engineer - Real-Time Analytics
Posted Updated
Description
Block Labs is seeking a Data Platform Engineer to own the end-to-end real-time data pipeline and build a scalable, multi-tenant data platform that powers real-time decisioning and predictive modelling across products.
You will design and operate platform services, focusing on reliability, cost efficiency, and production-grade pipelines, collaborating with data scientists, AI engineers, BI developers, and infrastructure teams.
Design, build, and maintain scalable data pipelines using AWS Glue (PySpark), or equivalent orchestration and transformation tools. Engineer and optimise the ClickHouse warehouse for sub-second query performance across all back-offices. Implement data contracts between back-office and the platform. Onboarding a new operator is a config change, not new tables, topics, or feature views. Build the feature-serving layer providing pre-computed features to AI agents at millisecond latency. Integrate with third-party databases, back-office APIs, and external systems (CRM, affiliates, acquisition platforms). Establish monitoring, alerting, and maintenance procedures including pipeline health checks, freshness monitoring, anomaly detection, and data contract SLA enforcement. Own CI/CD and infrastructure-as-code for data workloads. Collaborate with data scientists, agent engineers, BI developers, and infrastructure teams to translate data requirements into reliable, production-grade pipelines. 3+ years building and operating production data pipelines at scale. Experience with streaming and batch data processing (Kafka / MSK, Spark). Strong SQL and warehouse engineering with ClickHouse or equivalents. Proficient in PySpark / Spark Streaming. Experience with CI/CD and IaC (CloudFormation, Terraform, CDK). Data modeling for multi-tenant schemas and data contracts. Observability: data quality checks and monitoring.Skills
Sign in to read and join the discussion.