Sr Data Engineer / Data Pipeline Architect
Description
Key Responsibilities
- Design and drive the architecture, migrating pipelines incrementally while ensuring existing outputs remain unchanged.
- Establish sustainable engineering processes for testing, code reviews, incident management, and releases.
- Combine architectural leadership with hands-on engineering using Airflow, Spark, dbt, BigQuery, and Python.
- Collaborate closely with and mentor a Data Engineer within the team, ensuring long-term ownership, reliability, and independent maintenance of the migrated pipelines.
Must-Have Requirements
- Minimum 5 years of relevant professional experience (5-10 years overall).
- Deep hands-on experience with Airflow, Spark, dbt, BigQuery, and Python.
- Proven experience technically leading at least one complete data pipeline migration from start to finish.
- Experience refactoring production data pipelines while verifying that their outputs remain unchanged.
- Experience establishing and implementing engineering processes for testing, code reviews, and incident management.
- Demonstrated mentoring experience with measurable outcomes.
Education
- Minimum Bachelor's degree (or equivalent).
Nice-to-Have
- Experience with encryption at rest, data retention, or data deletion compliance (e.g., GDPR).
- Experience with envelope encryption and key management solutions (e.g., GCP KMS, AEAD functions, or Tink).
- Knowledge of BigQuery column-level security, policy tags, and Data Catalog.
- Experience with Config Connector (KCC) for managing GCP resources through Kubernetes.
- Experience with event-driven architectures using Kafka.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.