IN_Manager_Cloud Solution Architect_D&A_Advisory_Mumbai
Posted Updated
Description
Job Description & Summary:
PwC India is looking for a highly skilled Databricks Manager with hands-on experience in designing, developing, and managing scalable big data and analytics solutions using the Databricks platform. This role focuses on deploying, integrating, and optimizing the Databricks ecosystem to enable efficient data processing, advanced analytics, and enterprise-wide data transformation initiatives.
Job Position Title: IN_Manager_Cloud Solution Architect_D&A_Advisory_Mumbai
Responsibilities:
- Lead the architecture, design, and implementation of data engineering solutions on the Databricks platform to support big data analytics and AI/ML workloads.
- Develop and maintain scalable ETL/ELT pipelines using Apache Spark on Databricks for processing large volumes of structured and unstructured data.
- Collaborate with business stakeholders, data scientists, and engineers to translate data requirements into robust, performant Databricks solutions.
- Optimize cluster performance, cost, and resource utilization within Databricks environments.
- Manage data ingestion, transformation, and integration workflows connecting Databricks with data lakes, data warehouses, and enterprise applications.
- Implement data governance, security, and compliance best practices in the Databricks platform.
- Oversee monitoring, debugging, and troubleshooting of Databricks applications and data workflows.
- Mentor and lead a team of data engineers and analysts, driving best practices in coding, deployment, and operational support.
- Create and maintain detailed technical documentation including architecture diagrams, data flow diagrams, and runbooks.
- Stay updated with the latest Databricks product features, ecosystem tools, and industry trends to continuously improve solution quality and capabilities.
Mandatory skill sets:
- Extensive hands-on experience with Databricks platform, Apache Spark, and big data processing frameworks.
- Strong expertise in building and managing ETL/ELT workflows using Databricks and Spark SQL.
- Proficiency in programming languages such as Python, Scala, or SQL for data engineering tasks.
- Experience in integrating Databricks with cloud storage (e.g., AWS S3, Azure Data Lake Storage) and data warehouses.
- Knowledge of data modeling, data partitioning, and performance optimization techniques in big data environments.
- Understanding of data security, encryption, and governance frameworks relevant to cloud data platforms.
- Proven leadership experience managing data engineering teams and delivering complex projects on schedule.
- Familiarity with version control (Git), CI/CD pipelines, and agile development methodologies.
Preferred skill sets:
- Experience with cloud platforms such as AWS, Azure, or GCP and their respective Databricks offerings.
- Knowledge of containerization, orchestration (Docker, Kubernetes), and infrastructure as code (Terraform, ARM).
- Exposure to machine learning workflows and MLflow in Databricks.
- Prior consulting experience delivering enterprise-scale data solutions and transformations.
Years of experience required: 8 - 12 Years
Education qualification: BE, B.Tech, MCA, MBA, M.Tech
Skills
Sign in to read and join the discussion.