Senior Data Platform Engineer — Scale, Security & AI Ops
Posted Updated
Description
SP Group is seeking a seasoned Data Platform Engineer to operate and evolve its enterprise Data & AI platform, supporting mission-critical analytics, regulatory compliance, and AI-driven innovation across the business.
You will design, deploy, and manage large-scale Hadoop-based environments (Cloudera CDP, Hive, Spark), implement IaC with Terraform and Ansible, and partner with data engineers, scientists, and governance teams to ensure uptime, security, and performance.
Operate and evolve SP Group's enterprise data platform to support analytics, regulatory compliance, and AI-driven innovation. Provide platform support and partnership to data engineering, analytics, and business teams. Design, deploy, and manage large-scale data platform environments (Cloudera CDP, Hadoop ecosystem). Manage cluster lifecycle including provisioning, scaling, patching, and upgrades. Ensure uptime and reliability through proactive monitoring and maintenance. Plan and coordinate upgrades, patching, and scaling activities; minimize service disruption. Coordinate cross-team dependencies (infrastructure, network, security). Document platform architecture, processes, and operational procedures. Implement Infrastructure-as-Code using Terraform and Ansible. Design and maintain CI/CD pipelines for platform configuration. Automate provisioning, deployment, and operational workflows. Improve observability through monitoring, alerting, and logging practices. Implement security controls including Kerberos authentication, authorization (Ranger), and encryption. Ensure compliance with data governance, security policies, and regulatory requirements. Support auditability, access control, and data protection mechanisms. Collaborate with data engineers, data scientists, and business teams to enable data solution delivery. Partner with infrastructure, network, and security teams to ensure platform integration and stability. 5-10 years of experience in data platform engineering, big data operations, or infrastructure engineering. Strong hands-on experience managing big data platforms (Cloudera CDP, Hadoop ecosystem). Deep understanding of Hadoop components: HDFS, Hive, Spark, HBase, and related services. Experience with cluster operations, performance tuning, and troubleshooting. Proficiency in automation and scripting (Python, Bash, Terraform, Ansible). Experience with CI/CD pipelines and DevOps practices. Solid understanding of data platform security and access control concepts. Strong analytical and problem-solving skills with a proactive mindset. Ability to work in a fast-paced, cross-functional environment. Good communication skills with ability to translate technical topics to stakeholders. Ownership mentality with focus on platform reliability and continuous improvement. Wellness programmes Collaborative teamsSkills
Sign in to read and join the discussion.