Devops Engineer - Databricks

Job Description

· We are looking for an experienced Databricks Infrastructure & DevOps Engineer to manage, optimize, and secure enterprise Databricks environments hosted on AWS.

· The ideal candidate will have strong expertise in cloud infrastructure, DevOps automation, CI/CD pipelines, Infrastructure as Code (IaC), and security governance within highly regulated environments.

Key Responsibilities

· Infrastructure & Platform Management Design, deploy, and administer Databricks workspaces and clusters on AWS.

· Build and manage cloud infrastructure using Terraform and Databricks Asset Bundles (DAB).

· Configure and optimize AWS services including S3, IAM, VPC, Security Groups, and networking.

· Monitor platform performance, availability, scalability, and reliability. Manage secrets, encryption keys, and platform security configurations.

· Implement backup, retention, and disaster recovery strategies. DevOps & CI/CD Design, develop, and maintain CI/CD pipelines for Databricks deployments.

· Manage source control and automated deployments using GitLab.

· Implement code quality and security scanning using SonarQube, Fortify on Demand (FOD), Nexus IQ, and CloudSCAPE.

· Automate infrastructure provisioning and application deployment.

· Support release management and deployment automation across environments.

· Security & Compliance Ensure compliance with government security frameworks and data governance standards.

· Support vulnerability assessments, penetration testing (VAPT), and remediation activities.

· Manage identity and access controls through Central Accounts Management (CAM). Implement security monitoring, vulnerability management, audit logging, and compliance reporting.

· Coordinate security testing, System Security Acceptance Testing (SSAT), and regulatory compliance activities.

· Collaboration & Support Collaborate with Data Engineers, Data Analysts, Cloud Architects, and Security teams to support enterprise data platforms.

· Support multiple Databricks environments and data pipelines.

· Provide technical guidance on infrastructure, deployment automation, and platform optimization.

· Maintain technical documentation, operational procedures, and knowledge-sharing materials.

· Support environment provisioning, onboarding, and training activities.

Required Qualifications

· Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.

· Minimum 3 years of experience in DevOps, Cloud Infrastructure, or Platform Engineering.

· Hands-on experience administering the Databricks platform.

· Strong experience with AWS Cloud services.

· Proficiency in Terraform and Infrastructure as Code (IaC).

· Experience designing and managing GitLab CI/CD pipelines.

· Strong scripting skills using Python, Bash, or PowerShell.

· Experience implementing security best practices in enterprise cloud environments.

· Strong analytical, troubleshooting, and problem-solving skills.

· Excellent communication and stakeholder management abilities.

· Technical Skills Databricks Administration AWS Cloud (S3, IAM, VPC, Security Groups) Terraform Databricks Asset Bundles (DAB) GitLab CI/CD SonarQube Fortify on Demand (FOD) Nexus IQ CloudSCAPE Python / Bash / PowerShell Infrastructure as Code (IaC) Security Monitoring & Compliance Vulnerability

· Management Containerization & Orchestration

Preferred Qualifications

· Experience working in Government Cloud (GCC) or highly regulated environments.

· Knowledge of government security standards, data classification, and compliance frameworks.

· Experience supporting enterprise data platforms and Databricks environments.

· Familiarity with SSAT, VAPT, PDPIA, and security governance processes.

· Preferred Certifications-Databricks Certified Associate (or higher) AWS Certified DevOps Engineer AWS Certified Solutions Architect (Preferred)