Azure Databricks DevOps Engineer
Summary
Build and maintain Azure Databricks and cloud data pipelines using Terraform, CI/CD, and Azure DevOps; ensure scalable, secure, and high-performance data workflows.
About the job Azure Databricks DevOps Engineer
- Databricks Management: Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.
- Infrastructure Automation: Implement and manage IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.
- CI/CD Pipelines: Develop and manage CI/CD pipelines using tools like Azure DevOps, GitHub Actions, Jenkins to automate data pipeline deployments.
- Monitoring & Logging: Set up and maintain monitoring solutions using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.
- Security & Compliance: Implement best practices for security, governance, and compliance, including RBAC, encryption, and network security policies.
- Data Integration & Optimization: Work with Data Engineers to optimize ETL pipelines, ensuring high performance and cost efficiency.
- Collaboration: Work closely with Data Scientists, Engineers, and Business Analysts to enable efficient data workflows.
- Incident Management: Troubleshoot and resolve Databricks, Azure Kubernetes Services (AKS), and networking issues efficiently.
- Documentation & Best Practices: Maintain technical documentation and establish best practices for Azure Databricks and DevOps processes.
Required Skills & Qualifications
- Bachelors or Masters degree in Computer Science, IT, or a related field.
- 5+ years of experience in DevOps, Cloud Engineering, or Data Engineering roles.
- Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.
- Proficiency in Terraform, ARM Templates, or Bicep for infrastructure automation.
- Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.
- Knowledge of Python, Scala, or PowerShell for automation and scripting.
- Familiarity with containerization (Docker, Kubernetes, AKS).
- Solid understanding of Azure networking, security, and identity management (IAM, RBAC, AAD).
- Experience with monitoring and logging tools (Azure Monitor, Log Analytics, Prometheus, Grafana).
- Strong problem-solving skills and ability to work in an Agile/Scrum environment.
Preferred Qualifications
- Azure Certifications (e.g., AZ-400, AZ-104, DP-203, DP-900).
- Experience with Big Data processing frameworks (Apache Spark, Delta Lake).
- Understanding of ML Ops and working with machine learning models in Databricks.
- Knowledge of Event-driven architectures (Azure Event Hubs, Kafka, Service Bus).