Advisor - Cloud Data architect
Posted Updated
Description
Job Summary
Responsibilities
Required
Good to have
We are seeking an experienced and visionary Lead Data Architect to design, build, and optimize our next-generation enterprise data platform on Google Cloud Platform (GCP). This pivotal role will be responsible for defining the architectural roadmap, leading the technical design, and ensuring the successful implementation of robust, scalable, and secure data solutions that support our strategic business objectives. The ideal candidate will have deep expertise in modern data warehousing, data lake, and data lake house concepts, with hands-on experience in advanced GCP data services, PySpark / Databricks and related open-source technologies
Responsibilities
- Design end-to-end data architectures, including data ingestion, transformation, storage, processing, and consumption layers.
- Lead the selection and integration of appropriate GCP services (e.g., BigQuery, Dataproc, Cloud Storage, Dataflow) and open-source technologies.
- Develop architectural standards, patterns, and best practices for data platform development and operations.
- Design and drive the implementation of robust, scalable, and efficient data pipelines using PySpark on Dataproc for large-scale data processing.
- Utilize data transformation tools like dbt (data build tool) for efficient and version-controlled data modeling and transformation within BigQuery.
- Implement and manage data storage solutions leveraging Apache Iceberg for open table formats, enabling schema evolution and time travel capabilities.
- Design and enforce data quality frameworks, processes, and tools to ensure data accuracy, completeness, and consistency across the enterprise.
- Collaborate with data goverce teams to define policies and ensure compliance with regulatory requirements (e.g., HIPAA, GDPR).
- Implement monitoring and alerting solutions for data pipelines and platform health
- Integrate and configure Starburst (Trino) for federated query capabilities across diverse data sources and BigLake Metastore for unified metadata management
- Provide technical leadership, mentorship, and guidance to data engineers and other technical teams.
- Collaborate closely with business stakeholders, US team and application development teams to understand data needs and deliver solutions.
- Communicate complex technical concepts effectively to both technical and non-technical audiences.
- Stay abreast of emerging technologies and industry trends in data architecture, GCP services, and open-source data platforms
- Provide regular updates on the tasks, status and risks to project manager
Required
- Bachelors degree or higher from a reputed university
- 12 to 18 years of progressive experience in data architecture, data engineering or similar role
- 3+ years of hands-on experience designing and implementing large-scale data platforms on Google Cloud Platform (GCP).
- Demonstrated expertise with BigQuery, Dataproc, Cloud Storage, and other relevant GCP data services.
- Strong proficiency in PySpark for data processing and transformation for both and streaming patterns
- Experience in streaming tools like Kafka, Pub-Sub, spark streaming etc.
- Extensive experience with modern data warehousing principles, data lakes, and Data lakehouse architectures.
- Hands-on experience with open table formats, specifically Apache Iceberg.
- Experience implementing data goverce frameworks, data quality tools, and metadata management solutions.
- Strong understanding of security best practices in a cloud data environment.
- Experience with Infrastructure as Code (e.g., Terraform) and CI/CD pipelines for data solutions is a plus
- Excellent analytical and problem-solving skills with a keen attention to detail
- Strong communication and interpersonal skills, with the ability to explain complex technical concepts to both technical and non-technical audiences
Good to have
- Experience with federated query engines like Starburst (Trino) and unified metadata solutions like BigLake Metastore is highly desirable.
- Proven experience with data transformation tools such as dbt (data build tool).
- Experience in processing streaming data using Kafka / Pub-Sub
- Healthcare industry experience
- Experience in Agile