Data Engineer - Google Cloud Platform
Summary
Builds and maintains scalable data pipelines, data models, and analytics platforms on Google Cloud Platform using BigQuery, dbt Cloud, and Apache Airflow, with Terraform-driven infrastructure and CI/CD. Requires 4-9 years of data engineering experience, including mandatory GCP, dbt, and Airflow skills.
Role Overview :
We are looking for an experienced GCP Data Engineer to design, develop, and optimize scalable cloud-based data solutions. The role involves building and maintaining data pipelines, data models, and analytics platforms using GCP, dbt, and Apache Airflow.
Responsibilities :
- Design, develop, and maintain scalable data pipelines and workflows on Google Cloud Platform (GCP).
- Build and optimize data models using dbt Cloud, including models, macros, tests, and Slowly Changing Dimensions (SCDs).
- Develop and manage Apache Airflow workflows, optimizing orchestration, scheduling, and pipeline performance.
- Work with BigQuery to develop complex queries, optimize performance, and manage data processing costs.
- Implement data modelling practices using fact and dimension tables, SCDs, and appropriate schema design.
- Develop data quality checks, validation frameworks, and monitoring mechanisms to ensure data accuracy and reliability.
- Manage version control and CI/CD workflows using GitHub, Bitbucket, and relevant deployment tools.
- Support infrastructure automation using Terraform or CloudFormation.
- Apply cloud security and IAM best practices to protect data and manage access.
- Build and maintain cross-cloud data pipelines where required.
- Collaborate with cross-functional teams to deliver reliable and scalable data engineering solutions.
Requirements :
- 4 - 9 years of experience in data engineering, with strong hands-on expertise in GCP.
- Mandatory experience with GCP, dbt, and Apache Airflow.
- Strong proficiency in BigQuery, including query tuning, performance optimization, and cost management.
- Hands-on experience with dbt Cloud, including data models, SCDs, tests, macros, and governance.
- Strong SQL skills and proficiency in Python for data processing and pipeline development.
- Experience designing and optimizing Airflow workflows and data orchestration processes.
- Knowledge of data modelling concepts, including fact and dimension tables and SCDs.
- Experience with GitHub or Bitbucket, CI/CD tools, and infrastructure-as-code tools such as Terraform or CloudFormation.
- Understanding of data quality frameworks, cloud security, and IAM best practices.
- Exposure to cross-cloud data pipelines is an advantage.
- B.E./B.Tech in Computer Science, Information Technology, or a related discipline is preferred.