Big Data Platform Engineer

Systems is looking for an experienced Big Data Platform Engineer to design, manage, and optimize Azure-based data platform infrastructure. The ideal candidate will have strong expertise in Azure Kubernetes Service (AKS), Azure DevOps, Docker, Infrastructure as Code (Terraform/Bicep), and enterprise-scale data platforms, ensuring platform reliability, security, and high availability.

Responsibilities

• Manage and support AKS-based Big Data platform infrastructure.
• Administer Apache NiFi, Azure Event Hub, and Azure Functions.
• Implement and maintain CI/CD pipelines using Azure DevOps.
• Monitor platform health and performance using Azure Monitor and Log Analytics.
• Troubleshoot platform, Kubernetes networking, and infrastructure-related issues.
• Implement infrastructure automation using Terraform and Bicep.
• Ensure platform security, governance, and compliance.
• Collaborate with Data Engineering teams to enable and optimize the platform.
• Perform platform upgrades, patching, capacity planning, and performance optimization.
• Support production incidents and drive platform reliability initiatives.

Requirements

• 8+ years of overall IT experience.
• 5+ years in Cloud Infrastructure Engineering.
• 4+ years working with Kubernetes platforms.
• 3+ years supporting enterprise-scale Data Platforms.
• 3+ years in DevOps/SRE practices.
• 2+ years working with Azure Data Services.
• Azure Kubernetes Service (AKS): Deployment, administration, upgrades, autoscaling, RBAC, networking, storage, security, and troubleshooting.
• Kubernetes Networking: Services, Ingress, Load Balancers, DNS, Network Policies, Service Mesh (Istio preferred), and CNI Architecture.
• Azure Networking: VNet, Subnets, Private Endpoints, NSGs, UDRs, Azure Firewall, and VPN Connectivity.
• Docker & Containerization: Docker, Azure Container Registry (ACR), image optimization, security scanning, and lifecycle management.
• Azure DevOps: YAML Pipelines, Git, CI/CD, ARM Templates, Bicep, and Terraform.
• Monitoring & Observability: Azure Monitor, Log Analytics (KQL), Application Insights, and AKS monitoring.
• Azure Event Hub administration.
• Azure Functions deployment and operations.
• Apache NiFi administration and performance tuning.
• Infrastructure as Code (Terraform, Bicep, and ARM Templates).

Good-to-Have Skills

• GitOps (ArgoCD/Flux) and Platform Engineering.
• Grafana, Prometheus, and OpenTelemetry.
• Microsoft Defender for Cloud, Kubernetes Security, and Container Security.
• Azure Data Factory and Synapse Analytics.
• Multi-region architecture, Disaster Recovery, and Landing Zone implementation.
• Scripting using PowerShell, Bash, and Python.