Technical Anchor - AWS Data Engineering

Role Overview:

Looking for a Technical Anchor to lead the architecture, engineering, and delivery of our enterprise Data Lakehouse platform. In this hybrid leadership and hands-on role, you will be the chief authority for Databricks on AWS, steering our technical strategy and mentoring a team of data engineers.

Key Responsibilities:

- Design and implement scalable cloud data architectures utilizing the Databricks Lakehouse platform on AWS (utilizing Medallion architecture - Bronze, Silver, Gold)

- Oversee the development of robust, real-time streaming and batch pipelines using PySpark, SQL, Delta Lake, and Delta Live Tables

- Architect access controls and cross-account integrations using Databricks Unity Catalog, ensuring strict compliance with AWS security standards (IAM, KMS, VPC Peering/PrivateLink)

- Set engineering guidelines, run code reviews, and mentor mid-level/senior data engineers while advising product owners and client stakeholders

- Continuously monitor and optimize Spark performance (Photon engine, serverless clusters, and pipeline efficiency) to manage and reduce compute costs

What we are looking for:

- 6+ years of dedicated data engineering experience with at least 4+ years of hands-on expertise building production solutions on Databricks on AWS

- Highly proficient in PySpark / Apache Spark, Python, and complex relational SQL

- Strong command of the AWS ecosystem (S3, EMR, Athena, Glue, CloudWatch) alongside infrastructure-as-code automation

- Direct, practical experience setting up and managing Databricks Unity Catalog across multiple environments

- Proven track record of guiding development teams, leading design reviews, and clearly communicating technical architecture to non-technical stakeholders