GCP Data Engineer
Summary
Designs and builds scalable GCP data pipelines using BigQuery, Dataflow, and Pub/Sub to ingest, transform, and deliver data for analytics and reporting.
We are seeking a skilled GCP Data Engineer to design, develop, and optimize scalable data pipelines and cloud-based data solutions on Google Cloud Platform (GCP). The ideal candidate should have strong experience in data engineering, ETL development, big data technologies, and cloud-native services within GCP.
Key Responsibilities
- Design, build, and maintain scalable ETL/ELT data pipelines using GCP services.
- Develop and manage data ingestion, transformation, and integration processes from multiple data sources.
- Implement and optimize data solutions using BigQuery, Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Composer.
- Build and maintain data warehouses, data lakes, and data marts on GCP.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Ensure data quality, governance, security, and compliance standards.
- Monitor, troubleshoot, and optimize data pipeline performance and costs.
- Automate deployment processes using CI/CD tools and Infrastructure as Code (Terraform preferred).
- Support real-time and batch data processing requirements.
Required Skills & Qualifications
- Bachelor's degree in Computer Science, Information Technology, or related field.
- 4+ years of experience in Data Engineering.
- Strong experience with Google Cloud Platform (GCP) services:
- BigQuery
- Dataflow
- Pub/Sub
- Cloud Composer (Airflow)
- Cloud Functions
- Strong SQL and Python programming skills.
- Experience building ETL/ELT pipelines.
- Hands-on experience with Apache Spark, Hadoop, or similar big data technologies.
- Knowledge of data modeling, data warehousing, and dimensional modeling.
- Experience with version control systems such as Git.
- Understanding of CI/CD and DevOps practices.
Preferred Qualifications
- GCP Professional Data Engineer Certification.
- Experience with Terraform and Infrastructure as Code (IaC).
- Familiarity with Kafka, Airflow, or distributed data processing frameworks.
- Experience working in Agile/Scrum environments.
- Exposure to machine learning data pipelines is a plus.
Key Technologies
- Cloud: GCP
- Data Warehouse: BigQuery
- Programming: Python, SQL