Data Engineer (Porto ou Lisboa, 3y-6y)

🚀 About the Role

We are looking for a Senior GCP Data Engineer to support a strategic data transformation and cloud migration initiative.

In this role, you will help migrate databases, ETL processes, and analytical workloads from SQL Server to Google Cloud Platform, designing a modern, scalable, and cost-efficient data ecosystem powered by BigQuery, Cloud Composer, and Python.

You will work closely with Database Administrators, Data Engineers, Analysts, Data Scientists, and business stakeholders to understand the existing environment, rebuild critical data flows, and ensure a smooth transition to the new cloud platform.

This is an excellent opportunity for someone who combines strong data engineering expertise with hands-on cloud migration experience and enjoys solving complex data, performance, and integration challenges.

Your Mission

As a Senior GCP Data Engineer, you will:

• Contribute to the migration of data and ETL processes from SQL Server to GCP.
• Design and build scalable data pipelines using modern cloud technologies.
• Develop optimized data models and queries for BigQuery.
• Automate migration, transformation, validation, and integration activities using Python.
• Guarantee data quality, integrity, performance, and operational reliability throughout the migration.
• Help establish engineering standards and best practices for the new GCP data platform.
• Support teams and end users during the transition to the cloud environment.

Key Responsibilities

Data & Database Migration

• Contribute to the design and execution of the migration strategy from SQL Server to Google Cloud Platform.
• Analyze existing databases, ETL workflows, stored procedures, dependencies, and data structures.
• Develop migration scripts and pipelines to transfer and transform data into BigQuery.
• Rebuild or adapt existing SQL Server processes for a cloud-native data architecture.
• Support migration planning, technical assessments, testing, reconciliation, and production deployment.
• Ensure data consistency and integrity before, during, and after migration.
• Identify migration risks, technical dependencies, and potential performance issues.

GCP Data Pipeline Development

• Design, develop, and maintain robust ETL and ELT pipelines on Google Cloud Platform.
• Replace or modernize existing data flows using scalable cloud services.
• Build reusable components and standardized pipeline patterns.
• Ensure pipelines are reliable, maintainable, observable, and performance-oriented.
• Implement appropriate retry, recovery, logging, and error-handling mechanisms.
• Support both batch processing and data integration use cases.

Workflow Orchestration

• Create, schedule, monitor, and maintain workflows using Google Cloud Composer.
• Design and manage DAGs for migration, transformation, and integration processes.
• Define dependencies, execution schedules, retries, alerts, and failure-handling processes.
• Troubleshoot orchestration issues and optimize workflow execution.
• Ensure reliable operationalization of pipelines across development, testing, and production environments.

BigQuery Engineering & Optimization

• Design and implement scalable data structures in BigQuery.
• Develop and optimize complex SQL queries for analytical and transformation workloads.
• Adapt T-SQL logic and existing SQL Server processes to BigQuery SQL.
• Apply BigQuery optimization techniques, including:Partitioning
• Clustering
• Query optimization
• Storage optimization
• Cost-efficient processing
• Monitor query performance and resource consumption.
• Contribute to the design of cloud data models aligned with business and analytical requirements.

Python Development & Automation

• Develop Python scripts for:Data migration
• Data transformation
• Data reconciliation
• Test data preparation
• Process automation
• Integration with GCP services and APIs
• Build reusable and maintainable Python components.
• Automate repetitive migration and operational activities.
• Support the development of validation and monitoring utilities.
• Apply software engineering best practices, including code reviews, version control, testing, and documentation.

Data Quality & Validation

• Implement data quality controls throughout the migration lifecycle.
• Develop automated reconciliation processes between source and target systems.
• Validate data completeness, accuracy, consistency, and integrity.
• Define tests for migrated tables, transformations, pipelines, and business rules.
• Investigate data discrepancies and coordinate their resolution.
• Ensure that the new platform produces reliable and trusted data for its users.

Monitoring, Support & Troubleshooting

• Monitor data pipelines, scheduled workflows, and production processes.
• Investigate and resolve pipeline failures, data issues, and performance bottlenecks.
• Support debugging and incident resolution during the migration and after production deployment.
• Perform root cause analysis and implement sustainable corrective measures.
• Improve platform observability, alerts, operational procedures, and support documentation.

Collaboration & Stakeholder Support

• Work closely with SQL Server administrators and existing technical teams to understand the current ecosystem.
• Collaborate with Data Analysts, Data Scientists, and business teams to gather data requirements.
• Ensure that the new GCP platform meets functional, analytical, and operational expectations.
• Participate in technical design sessions, code reviews, and testing activities.
• Communicate migration progress, risks, dependencies, and technical decisions clearly.
• Support knowledge transfer and the adoption of the new cloud data platform.

Continuous Improvement

• Stay informed about GCP data engineering capabilities and cloud migration practices.
• Identify opportunities to improve performance, scalability, cost efficiency, and maintainability.
• Propose new tools, patterns, and automation opportunities for the data platform.
• Contribute to engineering standards, documentation, and reusable development practices.
• Help establish a culture of technical excellence and continuous improvement.

✅ What We’re Looking For

Mandatory Requirements

• Strong professional experience in Data Engineering.
• Advanced knowledge of Google Cloud Platform, particularly its data services.
• Significant hands-on experience with BigQuery, including:Data modeling
• Schema design
• Complex SQL development
• Partitioning and clustering
• Query performance optimization
• Cost management
• Strong experience with Google Cloud Composer.
• Hands-on experience creating, scheduling, monitoring, and debugging DAGs.
• Excellent SQL skills and the ability to adapt existing SQL workloads to BigQuery.
• Advanced Python skills for data engineering, automation, and GCP integration.
• Experience designing and building ETL or ELT pipelines.
• Strong understanding of database and data migration methodologies.
• Ability to analyze legacy or on-premises environments and redesign them for the cloud.
• Strong analytical and troubleshooting capabilities.
• Good English communication skills, with a minimum B2 level.

🛠 Technical Stack

Google Cloud Platform

• Google Cloud Platform
• BigQuery
• Cloud Composer
• Cloud Storage
• Dataflow
• GCP APIs
• Database Migration Service

Data Engineering

• ETL
• ELT
• Batch Data Processing
• Data Integration
• Pipeline Orchestration
• Data Migration
• Data Reconciliation

Development

• Python
• SQL
• T-SQL
• Automation Scripting
• API Integration

Databases & Data Warehousing

• Microsoft SQL Server
• BigQuery
• Relational Databases
• Cloud Data Warehouses
• Data Modeling
• Data Vault

Performance & Cost Optimization

• BigQuery Partitioning
• BigQuery Clustering
• Query Optimization
• Storage Optimization
• Cloud Cost Management

Quality & Delivery

• Data Quality
• Data Validation
• Automated Testing
• Code Reviews
• Version Control
• CI/CD Practices
• Technical Documentation

⭐ Nice to Have

• Direct experience with Microsoft SQL Server, including:SQL Server architecture
• T-SQL
• Stored procedures
• ETL dependencies
• Basic database administration
• Experience with additional GCP services such as:Cloud Storage
• Dataflow
• Database Migration Service
• Experience using cloud-specific database migration tools.
• Knowledge of Data Warehousing principles and dimensional modeling.
• Experience with Data Vault methodology.
• Knowledge of Power BI or other data visualization tools.
• Experience with source control and collaborative development workflows.
• Familiarity with CI/CD pipelines for data engineering solutions.
• Previous involvement in large-scale on-premises to cloud migration programs.