Gcp Data Engineer
Posted Updated 7
views
Description
Responsibilities:
- Design and implement complex data pipelines using Dataflow (Apache Beam) and Dataproc (Spark).
- Lead Cloud Composer (Airflow) orchestration design, optimizing DAG structure, dependencies, and retry strategies.
- Drive advanced Python and SQL implementations, including reusable frameworks and utility components.
- Build and standardize custom Dataflow / Beam templates and reusable pipeline patterns.
- Perform deep performance tuning for BigQuery (query optimization, partitioning strategies, cost optimization).
- Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers.
- Review code and design artifacts produced by engineers; provide technical guidance and mentoring.
- Own technical quality for assigned modules, including non-functional aspects (performance, reliability, scalability).
- Act as the first point of escalation for technical issues before involving architects.
- Collaborate with architects to understand design intent and ensure correct implementation.
Mandatory Technical Skills:
- Advanced GCP Knowledge
- Strong understanding of IAM strategy, service isolation, and project-level design considerations.
- Programming & Querying
- Advanced proficiency in Python and complex GoogleSQL.
- Orchestration
- Strong hands-on expertise with Cloud Composer / Airflow, including best practices and optimization.
- Data Processing
- Deep hands-on experience with Dataflow and Apache Beam.
- Strong working knowledge of Dataproc / Spark internals.
- Analytics Platform
- Advanced experience with BigQuery, including performance and cost optimization techniques.