Data Engineer - BigQuery
Summary
Designs and builds scalable GCP data pipelines using Dataflow, BigQuery, and Airflow to ingest, process, and store data for analytics and operations.
Position: Data Engineer - BigQuery
Work Setup: Hybrid (3X RTO)
Location: Metro Manila & Cebue
Summary:
We are seeking a highly skilled Senior Data Engineer to design, build, and optimize our next-generation data platform on Google Cloud Platform (GCP). In this role, you will be the backbone of our data architecture, responsible for developing robust batch and streaming data pipelines, handling complex file ingestions, and ensuring the absolute reliability of our data ecosystem.
Roles & Responsibilities
- Pipeline Development: Design, develop, and maintain scalable, high-performance batch and streaming data pipelines using GCP Dataflow (Apache Beam).
- Data Ingestion: Build robust ingestion frameworks to consume data from a variety of sources, including API endpoints, databases, and structured/unstructured file formats.
- Orchestration: Author, schedule, and monitor complex data workflows using Cloud Composer / Apache Airflow.
- Data Warehousing & Storage: Optimize data storage and retrieval in BigQuery and manage transactional data integrity within Cloud Spanner.
- Monitoring & Troubleshooting: Proactively monitor pipeline health, diagnose bottlenecks, and troubleshoot data quality or performance issues in production.
- Performance Tuning: Write and optimize complex, highly efficient SQL queries and manage schema evolutions.
Required Skills
GCP Ecosystem
- Strong, hands-on experience with Dataflow, Cloud Composer, and BigQuery. Working knowledge of Cloud Spanner is highly desirable.
Data Streaming
- Proven experience handling both batch and real-time/streaming data processing paradigms (e.g., Pub/Sub to Dataflow).
Languages
- Mastery of SQL (analytical and transactional) and proficiency in Python or Java (for Apache Beam/Airflow).
Operations
- Strong experience in monitoring, logging, and alerting (e.g., GCP Cloud Monitoring/Cloud Logging) and standard debugging practices.
Data Engineering Fundamentals
- Solid understanding of data modeling, ETL/ELT processes, and handling various file formats (Avro, Parquet, JSON, CSV).