Advance Python Programming, Automation Frameworks
Summary
Builds and maintains Python-based automation for data platform operations, integrating ServiceNow, Databricks, Dremio, and Azure services to streamline workflows and disaster recovery.
Advance Python Programming, Automation Frameworks JD\:
- Automate common ServiceNow request workflows related to data platform access, onboarding, approvals, fulfilment, and operational requests.
- Build or enhance self-service applications that integrate with ServiceNow and raise requests on behalf of users.
- Develop automated backup scripts and scheduled jobs for platform components such as Databricks Unity Catalog/metastore, Dremio metadata, configuration, ADLS objects, and other platform assets.
- Build tested restore scripts and runbooks for Dremio metadata, Databricks views, catalog objects, platform configurations, and other recoverable assets.
- Support ADLS disaster recovery planning, backup validation, failover readiness, and recovery testing.
- Responsible for both development and strong expertise to debug and fix any post production issues
- Create reusable scripts, notebooks, templates, and automation patterns for Databricks workloads, views, jobs, and pipelines.
- Work with platform users to review and optimize Databricks pipelines, improve performance, reduce cost, and promote reusable engineering patterns.
- Develop and maintain Terraform modules for Azure/GCP infrastructure used by the data platform.
- Build and maintain CI/CD pipelines using GitHub Actions/GitHub runners for deploying infrastructure, scripts, configurations, and platform automation.
- Help reduce repetitive operational work by identifying automation opportunities and implementing robust, reusable solutions.
- Own automation tasks end-to-end, from requirements gathering through development, testing, deployment, and support handover.
- Leads\: Role is 70% of time in Design, Architect, Coding and 30% of time in team mentoring, guidance, coordination
- Required Technical Skills\:
o Strong scripting experience using Python, Bash, PowerShell, or similar languages.
o Ability to build production-ready automation scripts.
o Experience integrating with ServiceNow APIs, automating ticket creation, request fulfilment, approvals, and workflow-driven processes.
o Good working knowledge of Databricks workspaces, jobs, notebooks, clusters, SQL warehouses, Unity Catalog/metastore concepts, views, and pipeline optimization.
o Responsible for both development and strong expertise to debug and fix any post production issues for their developed code
o Experience or willingness to learn Dremio administration, metadata backup/restore, source configuration, reflections, spaces, folders, and operational support.
o Experience with Azure Data Lake Storage, storage accounts, RBAC, managed identities, private networking, and disaster recovery concepts.
o Ability to design, automate, test, and document backup and restore processes for data platform metadata, configurations, and platform assets.
o Strong Terraform knowledge, including module development, state management, variables, providers, workspaces, and reusable infrastructure patterns.
o Strong Azure experience required.
o Ability to work with REST APIs, authentication mechanisms, secrets, tokens, and service principals.
- Nice to have skills
o Azure DevOps, Azure Functions, Logic Apps, Key Vault, Event Grid, Automation Accounts.
o Delta Lake, Unity Catalog permissions, job orchestration, DLT, cluster policies, cost/performance optimization.
o Dremio REST APIs, reflection management, source automation, backup/restore testing, Kubernetes/AKS deployment knowledge.
o AKS/GKE experience, Helm, kubectl, YAML-based deployments.