Principal Big Data Engineer (Contract Role)

Key Responsibilities

• Lead the architecture and development of scalable data warehouses and batch and real-time data pipelines (e.g. Flink-based streaming and Hive/HDFS-based batch processing).
• Own the end-to-end design and operation of a big data development platform, including multi-cluster resource management (e.g. YARN), access and permission controls, audit trails and unified authentication.
• Build and maintain real-time user-facing data services, such as transaction, billing and profit-and-loss analytics, with strict accuracy, latency and availability targets.
• Design reliable data ingestion and collection systems for external data sources, including primary/backup links, heartbeat monitoring, quota monitoring and fallback strategies.
• Define data models, tagging systems and feature datasets that support user segmentation, analytics and personalisation.
• Establish standards for data quality, observability, monitoring and governance, including unified monitoring dashboards and alerting.
• Plan and run fault drills and joint performance and stress tests to find bottlenecks, reduce operational risk and prepare the platform for peak-traffic events.
• Improve processing performance, resource efficiency and platform stability, and make sound scaling decisions.
• Provide technical guidance, conduct design and code reviews, and mentor engineers.
• Collaborate with business, product and engineering teams to translate requirements into dependable data solutions.

Job Requirements

Essential

• Bachelor's degree in Computer Science, Information and Computing Science, Mathematics, Statistics or a related field, or equivalent practical experience.
• At least 8 years of professional software or data engineering experience, including at least 4 years building and operating big data platforms in production.
• Hands-on experience in production with Flink for real-time processing, and with Hive, HDFS and YARN (or equivalent distributed storage, query and resource-management technologies).
• Proven experience designing and running a multi-tenant data or big data development platform, covering resource and cluster management, permissions, auditing and authentication.
• Strong proficiency in Java (including Spring / Spring Cloud microservices); working knowledge of Python, Scala or Go is an advantage.
• Strong SQL skills, including query optimisation, with experience in relational and distributed databases (e.g. MySQL, TiDB).
• Demonstrated experience building real-time data services for transactional or financial data, where accuracy and timeliness are critical.
• Track record of improving system reliability through monitoring, fault drills, failover design and performance or stress testing.
• Solid understanding of data modelling, ETL/ELT, batch and streaming architectures, and data quality practices.
• Ability to evaluate technical trade-offs, set engineering standards and mentor other engineers.

Advantageous

• Experience in payments or other regulated financial environment.
• Experience with Spark, Kafka, Trino or similar technologies.
• Experience with Redis, message queues, gRPC and distributed-systems patterns (locking, idempotency, consistency).
• Experience introducing micro-frontend or modular architectures for internal platforms.
• Experience building tagging or feature datasets for segmentation and recommendation use cases.