Data Engineer
Summary
Designs and builds scalable GCP-based data pipelines using Dataflow, Pub/Sub, and BigQuery to ingest, process, and store batch and streaming data reliably.
We are seeking a highly skilled Senior Data Engineer to design, build, and optimize our next-generation data platform on Google Cloud Platform (GCP). In this role, you will be the backbone of our data architecture, responsible for developing robust batch and streaming data pipelines, handling complex file ingestions, and ensuring the absolute reliability of our data ecosystem.
Pipeline Development
Design, develop, and maintain scalable, high-performance batch and streaming data pipelines using GCP Dataflow (Apache Beam).
Data Ingestion
Build robust ingestion frameworks to consume data from a variety of sources, including API endpoints, databases, and structured/unstructured file formats.
Orchestration
Author, schedule, and monitor complex data workflows using Cloud Composer / Apache Airflow.
Data Warehousing & Storage
Optimize data storage and retrieval in BigQuery and manage transactional data integrity within Cloud Spanner.
Monitoring & Troubleshooting
Proactively monitor pipeline health, diagnose bottlenecks, and troubleshoot data quality or performance issues in production.
Performance Tuning
Write and optimize complex, highly efficient SQL queries and manage schema evolutions.
Required Skills
GCP Ecosystem
- Strong, hands-on experience with Dataflow, Cloud Composer, and BigQuery. Working knowledge of Cloud Spanner is highly desirable.
Data Streaming
- Proven experience handling both batch and real-time/streaming data processing paradigms (e.g., Pub/Sub to Dataflow).
Languages
- Mastery of SQL (analytical and transactional) and proficiency in Python or Java (for Apache Beam/Airflow).
Operations
- Strong experience in monitoring, logging, and alerting (e.g., GCP Cloud Monitoring/Cloud Logging) and standard debugging practices.
- Solid understanding of data modeling, ETL/ELT processes, and handling various file formats (Avro, Parquet, JSON, CSV).
Preferred
- Google Cloud Certified Professional Data Engineer.
- Experience with CI/CD pipelines for data infrastructure (Terraform, GitHub Actions).
- Familiarity with data governance and security best practices on GCP (IAM, KMS, DLP).