Associate Architect - Databricks

JOB PURPOSE

Looking for an experienced Data & Cloud Architect to define and drive enterprise-scale data architecture leveraging Databricks Lakehouse, Snowflake, Apache Iceberg, and modern cloud-native technologies. The ideal candidate will lead architecture decisions, platform strategy, governance frameworks, and modernization initiatives.

JOB RESPONSIBILITY

Key Responsibilities

Enterprise Architecture

• Define enterprise data platform architecture and roadmap.
• Design scalable Lakehouse and Data Mesh architectures.
• Establish standards, patterns, and governance frameworks.
• Lead modernization and migration strategies.

Solution Architecture

• Architect solutions using Databricks, Snowflake, Iceberg, and Cloud services.
• Design multi-cloud and hybrid-cloud data ecosystems.
• Define data sharing and interoperability strategies.
• Enable AI/ML and GenAI workloads on modern data platforms.

Data Platform Strategy

• Develop metadata, catalog, lineage, and governance solutions.
• Design enterprise-scale security and compliance frameworks.
• Establish platform observability and FinOps practices.
• Optimize storage and compute costs.

Architecture Governance

• Conduct architecture reviews and technical assessments.
• Define reference architectures and reusable frameworks.
• Drive NFRs including scalability, resiliency, security, and performance.

Leadership & Stakeholder Management

• Collaborate with business, product, security, and engineering leaders.
• Present architecture recommendations to executive stakeholders.
• Lead multiple delivery teams across strategic programs.

QUALIFICATION

• B.Tech degree/MTech/MCA with computer science background or equivalent experience.

EXPERIENCE

• 10+ years of total and minimum 3 years of relevant experience in Databricks.
• Proven experience architecting enterprise-scale Data Lakehouse and Data Platform solutions using Databricks and Snowflake.
• Deep expertise in Cloud Architecture (AWS/Azure/GCP), including security, governance, scalability, and resiliency frameworks.
• Extensive experience defining enterprise data strategies, modernization roadmaps, and migration approaches.
• Strong knowledge of Apache Iceberg, Delta Lake, metadata management, and Data Mesh architectures.
• Experience designing multi-cloud and hybrid-cloud data ecosystems supporting analytics, AI/ML, and GenAI workloads.
• Proven leadership in defining architecture standards, governance models, and reference architectures.
• Expertise in cost optimization, performance engineering, and cloud FinOps implementation.
• Experience influencing executive stakeholders and leading architecture reviews across multiple programs.
• Strong capability in mentoring architects, leads, and engineering teams.
• Excellent communication, presentation, and strategic decision-making skills.

SKILLS AND COMPETENCIES

• Expert knowledge of Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and Performance Optimization.
• Strong proficiency in Python, PySpark, Spark SQL, and distributed data processing.
• Extensive hands-on experience with Snowflake, including Snowpipe, Streams, Tasks, Data Sharing, and Query Optimization.
• Strong expertise in Apache Iceberg, open table formats, schema evolution, partitioning, and time-travel capabilities.
• Experience designing and implementing large-scale Data Engineering and ETL/ELT Pipelines.
• Hands-on experience with AWS/Azure/GCP cloud-native data services.
• Strong SQL and data modeling skills across OLTP and analytical environments.
• Experience with Real-Time Data Processing using Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Knowledge of Data Governance, Metadata Management, Data Quality, and Security frameworks.

Good to Have

• Experience with Terraform, CloudFormation, or Infrastructure as Code (IaC).
• Knowledge of Docker, Kubernetes, and containerized data platforms.
• Experience with CI/CD tools such as GitHub Actions, Jenkins, Azure DevOps, or GitLab.
• Exposure to AI/ML, MLOps, and GenAI data platform integration.
• Knowledge of Data Mesh, Data Fabric, and modern enterprise data architectures.