Databricks Infrastructure Engineer

We are looking for an experienced Databricks Infrastructure Engineer to manage, maintain and optimise Databricks infrastructure hosted on AWS. The role will focus on Databricks platform administration, infrastructure automation, security, governance and operational support.

Key Responsibilities

Databricks & Infrastructure Management

• Design, deploy and maintain Databricks workspaces and clusters on AWS.
• Monitor and optimise Databricks platform performance and resource utilisation.
• Configure AWS services such as S3, IAM, VPC and security groups for Databricks environments.
• Manage infrastructure using Terraform or Databricks Asset Bundles (DAB).
• Support multiple datamarts and data processing environments.

CI/CD & Automation

• Develop and maintain CI/CD pipelines using GitLab and approved tools.
• Support automated deployment workflows for Databricks components.
• Implement code quality and security checks using tools such as SonarQube, Fortify and Nexus IQ.
• Develop scripts and automation using Python, Bash or PowerShell.

Security & Compliance

• Implement security and governance controls for Databricks environments.
• Support data classification, data protection and data retention requirements.
• Manage secrets, encryption mechanisms and key rotation.
• Support vulnerability assessment, remediation and security scanning activities.
• Assist with security monitoring, incident response and compliance activities.
• Support security testing and acceptance activities.

Platform Operations & Support

• Monitor system health, performance and operational issues.
• Configure notifications for data processing status and system errors.
• Support access management and user onboarding.
• Assist with training environment setup and maintenance.
• Provide technical guidance for data pipelines and platform-related issues.

Collaboration

• Work closely with Data Engineers, Data Analysts and Security teams.
• Collaborate with government stakeholders and technical teams.
• Support the implementation of secure and reliable data platforms.

Key Requirements

• Minimum 3 years of experience in Databricks platform administration, infrastructure or related engineering roles.
• Strong hands-on experience with Databricks, including workspace and cluster management.
• Experience working with AWS, particularly services such as S3, IAM, VPC and security groups.
• Experience with Terraform or Databricks Asset Bundles (DAB).
• Experience with GitLab and CI/CD pipelines is an advantage.
• Knowledge of security and compliance practices for cloud environments.
• Experience with containerisation and orchestration technologies is an advantage.
• Strong scripting skills in Python, Bash or PowerShell.
• Knowledge of security scanning and vulnerability management tools such as SonarQube, Fortify and Nexus IQ is an advantage.
• Understanding of encryption, secrets management and key management practices.
• Strong troubleshooting, communication and stakeholder management skills.

Preferred Certifications

• Databricks Certified Associate or higher.
• AWS Certified DevOps Engineer or relevant AWS certification.

EA Number : 11C4879