Databricks Infrastructure and DevOps Engineer

Job Role Name : Databricks Infrastructure and DevOps Engineer

Key Responsibilities

Infrastructure Management

• Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment
• Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)
• Configure and Optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks
• Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management

• Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem
• Configure GitLab for version control and automated deployment workflows
• Implement code quality gates using SonarQube for static code analysis
• Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment
• Oversee dependency scanning and management using Nexus IQ
• Coordinate with Cloud SCAPE for security compliance scanning and remediation

Security and Compliance

• Implement and maintain Automated Baseline Log Review (ABLR) systems
• Coordinate with Gov Tech Cyber Security Operations Centre (GCSOC) for security monitoring
• Manage Central Accounts Management (CAM) integration for user access control
• Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT) findings.
• Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements
• Implement and maintain Vulnerability Management System (VMS)

Platform Operations

• Monitor system performance and implement optimization strategies
• Manage secrets and encryption key rotation for data protection
• Configure email notifications for data processing status and error alerts
• Implement and maintain data retention policies
• Support SaaS whitelisting processes and approvals
• Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support

• Work closely with Data Engineers, Data Analysts and security technical team.
• Support multiple data marts
• Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines
• Collaborate with various government divisions and stakeholders
• Support training environment setup for user onboarding and skill development

Qualifications

Technical Skills

• 3+years of experience with Databricks platform administration and optimization
• Strong expertise in AWS services, particularly in GCC environment
• Proficiency in Infrastructure-as-Code tools, preferably Terraform
• Experience with GitLab CI/CD pipeline development and management
• Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)
• Understanding of government security frameworks and compliance requirements
• Experience with containerization and orchestration technologies
• Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance

• Experience with government security clearance processes
• Knowledge of data classification and handling procedures (Sensitive High)
• Understanding of encryption mechanisms and key management
• Familiarity with vulnerability management and remediation processes
• Experience with security monitoring and incident response procedures

Certifications(Preferred)

• Databricks Certified Associate or higher
• AWS Certified DevOps Engineer

Required Experience

• Minimum 3 years of DevOps/Infrastructure engineering experience
• Experience working in government environments, esp in GCC.
• Experience in agile development methodologies