Senior GCP Data Engineer
Summary
Senior GCP Data Engineer who designs, builds, and optimizes Google Cloud data architecture and pipelines — batch and real-time ETL/ELT, data lakes/warehouses, and governance on BigQuery, Dataflow (Beam), Dataproc, Pub/Sub, and Composer (Airflow), with Terraform IaC and CI/CD. Leads technical design and enforces data modeling and security best practices.
Job description:
We are looking for an accomplished and highly motivated Senior GCP Data Engineer to design, build, and optimize our Google Cloud Platform data architecture and pipelines. This expert will lead large-scale data solution implementation, ensure data quality, and promote engineering best practices.
What We're Looking For:
- 6+ years of hands-on experience with Google Cloud Platform (GCP) data and analytics services.
- Experience using SQL, Python, or other relevant technology.
- Core Services: BigQuery (expert level), Cloud Dataflow (Apache Beam), Cloud Storage.
- Processing: Dataproc (Apache Spark/Hadoop), Pub/Sub.
- Orchestration: Cloud Composer (Apache Airflow).
- Deep understanding of Data Warehousing, Data Lake concepts, and advanced data modeling techniques (Star/Snowflake schema, Data Vault).
- Experience with Infrastructure as Code (IaC) tools like Terraform or Cloud Deployment Manager.
- Strong knowledge of version control (Git), software development lifecycle, and Agile/Scrum methodologies.
What You'll Do:
- Architect, design, and implement robust, scalable, and fault-tolerant data ingestion and ETL/ELT pipelines for batch and real-time processing on GCP.
- Define and enforce data modeling standards (Dimensional, 3NF, Data Vault, etc.) for optimal performance in BigQuery and other storage solutions.
- Lead the technical design and deployment of Data Lakes and Data Warehouses using best-in-class GCP services.
- Develop and maintain complex data pipelines using Python (Pandas, PySpark) and/or Scala/Java with a strong focus on Apache Beam/Dataflow and Dataproc/Spark.
- Write, optimize, and tune advanced SQL queries for efficiency and cost reduction, particularly within BigQuery.
- Implement data orchestration and workflow management using Cloud Composer (Apache Airflow).
- Ensure optimal performance, cost efficiency, and reliability of all data solutions through continuous monitoring and tuning.
- Implement data governance and data quality frameworks, including data validation, lineage tracking, and metadata management (e.g., Data Catalog).
- Establish and enforce security best practices on GCP, including IAM, data encryption (at rest and in transit), and proper network access controls.
- Drive DevOps principles: implement and maintain CI/CD pipelines for data solutions using tools like Cloud Build, Jenkins, or GitLab.
Skills
- Agile
- Airflow
- Analytics
- BigQuery
- CI/CD
- Cloud
- Cloud Composer
- Data Governance
- Data Ingestion
- Data Lake
- Data Modeling
- Data Pipelines
- Data Quality
- Data Vault
- Data Warehousing
- Dataproc
- DevOps
- ELT
- ETL
- GCP
- Git
- GitLab
- Hadoop
- IAM
- Infrastructure as Code
- Java
- Jenkins
- Metadata Management
- pandas
- PySpark
- Python
- Scala
- Scrum
- Snowflake
- Spark
- SQL
- Terraform
- Vault
- Version Control
Sign in to read and join the discussion.