Data Engineer with Modelling Experience (DE)
NewBe an early applicantSummary
Data engineer in London building and maintaining GCP data pipelines and data models (warehouse and graph) for a telecommunications platform. Day to day involves SQL and Python work with BigQuery, Dataform, Spanner and Airflow, plus Terraform and GitLab CI/CD for infrastructure and deployment.
Job Summary (List Format):
- Design, build, and maintain scalable data pipelines and transformation workflows in a GCP environment
- Implement and manage CI/CD pipelines using GitLab
- Deploy and maintain infrastructure using Terraform (Infrastructure-as-Code)
- Develop and orchestrate data workflows with Airflow (Google Cloud Composer)
- Model data at both data warehouse and graph database levels to support platform needs
- Design and manage SpannerDB schemas and queries
- Apply data governance and metadata management practices, especially with BigQuery Knowledge Catalog
- Collaborate with stakeholders and contribute to delivery of large-scale projects
- Write advanced SQL (including complex data transformations, UDFs, and BigQuery optimization)
- Use Dataform for SQL-based modelling and transformation
- Develop data engineering solutions in Python
- Apply best practices in data modelling (normalisation, denormalisation, SCD, medallion architecture, etc.)
- Manage and integrate cloud storage (GCS) including structure, access, and retention policies
- Utilize GitLab for source control, branching, merging, and repository maintenance
- Integrate and manage Terraform modules for repeatable infrastructure deployments
- Work with SpannerDB for relational modelling, key design, and querying
- Understand and implement event-driven messaging with Pub/Sub
- Handle GIS data formats and conversions, and use BigQuery GIS capabilities
- Possess domain knowledge in telecommunications (network topology, KPIs, telemetry, asset lifecycle)
- Demonstrate stakeholder management and engagement on large-scale projects
- (Nice to Have) Experience with Dataflow (Apache Beam) for data processing
- Design, build, and maintain scalable data pipelines and transformation workflows in a GCP environment
- Implement and manage CI/CD pipelines using GitLab
- Deploy and maintain infrastructure using Terraform (Infrastructure-as-Code)
- Develop and orchestrate data workflows with Airflow (Google Cloud Composer)
- Model data at both data warehouse and graph database levels to support platform needs
- Design and manage SpannerDB schemas and queries
- Apply data governance and metadata management practices, especially with BigQuery Knowledge Catalog
- Collaborate with stakeholders and contribute to delivery of large-scale projects
- Write advanced SQL (including complex data transformations, UDFs, and BigQuery optimization)
- Use Dataform for SQL-based modelling and transformation
- Develop data engineering solutions in Python
- Apply best practices in data modelling (normalisation, denormalisation, SCD, medallion architecture, etc.)
- Manage and integrate cloud storage (GCS) including structure, access, and retention policies
- Utilize GitLab for source control, branching, merging, and repository maintenance
- Integrate and manage Terraform modules for repeatable infrastructure deployments
- Work with SpannerDB for relational modelling, key design, and querying
- Understand and implement event-driven messaging with Pub/Sub
- Handle GIS data formats and conversions, and use BigQuery GIS capabilities
- Possess domain knowledge in telecommunications (network topology, KPIs, telemetry, asset lifecycle)
- Demonstrate stakeholder management and engagement on large-scale projects
- (Nice to Have) Experience with Dataflow (Apache Beam) for data processing