Big Data DevOps Engineer
Summary
A consultancy DevOps engineer who designs, deploys and secures Linux-based Hortonworks Hadoop big-data clusters on AWS, builds CI/CD pipelines, automates infrastructure with Ansible/Terraform/Docker, and runs monitoring and ELK logging for a multi-tenant data platform serving a UK client program.
Salary: £43,000 - 83,000 per year
Requirements:- Strong DevOps expertise with CI/CD pipelines in a Big Data technology landscape.
- Hands-on experience with AWS, Hortonworks, S3, Java 8, HBase, Kafka, JanusGraph, HDF (NiFi), and related big data technologies.
- Strong hands-on skills in setting up Linux-based infrastructure.
- Experience with infrastructure automation tools such as Ansible, Terraform, SaltStack, Puppet, and Docker.
- Experience with AWS cloud setup and administration.
- Fluency in Python and Java.
- Expertise in CI tools such as Jenkins and Nexus.
- Experience with Java build systems, primarily Maven and Gradle.
- Experience supporting code versioning and branching strategies using GitHub.
- Experience with configuration management and backup/restore policies using specialized tools and custom Bash scripts.
- Experience with system monitoring solutions such as Zabbix, Prometheus, and Nagios.
- Experience with log management using Elasticsearch and Logstash (ELK).
- Experience with Hadoop cluster administration, maintenance, capacity planning, and optimization.
- Experience securing Hadoop infrastructure using Knox and Kerberos.
- Experience working in an Agile environment.
- Ability to clearly articulate the technology and business pros and cons of various technologies, platforms, and architectural options.
- Ability to document processes.
- Excellent communication skills, with a can-do attitude and a team-player mindset.
- Excellent written and verbal communication skills and the ability to work in complex and critical systems.
- Graduate with work experience.
- Hortonworks Certified Administrator (HDPCA).
- Eligibility for SC clearance: UK/EU citizen or indefinite leave to remain visa holder without leaving the UK for more than 6 months in the last 5 years.
- Represent us as a DevOps expert and demonstrate our value to clients, prospects, and partners to establish trust, win confidence, and drive more business.
- Design, deploy, and maintain complex Linux-based infrastructure.
- Set up highly secure Hortonworks Hadoop clusters in AWS cloud.
- Perform cluster administration, maintenance, upgrades, capacity planning, and optimization.
- Automate operations by setting up continuous integration and continuous deployment pipelines, code versioning, and application containerisation.
- Run and manage a data platform with multiple tenants and client applications for transactional and analytical use cases.
- Oversee the integration of development output into production-quality releases and support the live system.
- Work with other leads and developers, own and contribute to deliverables, and guide the team towards success.
- Identify areas where processes and standards can be improved and help implement changes.
- AWS
- Ansible
- Big Data
- Bash
- CI/CD
- Cloud
- DevOps
- Docker
- ELK
- ElasticSearch
- GitHub
- Gradle
- Hadoop
- Support
- Java
- Jenkins
- Kafka
- Linux
- Maven
- Nexus
- Prometheus
- Puppet
- Python
- Terraform
- Zabbix
More:
We are hiring for a new transformational program for a UK customer focused on big data and advanced visualization capabilities. We work in an Agile environment with teams co-located in London, and this is a consultancy role. Successful applicants will work at our clients, primarily based in and around London, with some exceptions. The role is permanent and offers a competitive salary.
last updated 36 week of 2026