Senior Data Engineer
Posted
Description
Who are we?
Quanteam, founding entity of RAINBOW PARTNERS, is a consulting firm specialized in Banking and Financial Services. With more than 1200 consultants across 10 countries, we combine business and IT expertise to support leading institutions in transformation projects.
Our core areas:
➡ Quantitative Finance
➡ Risk & Compliance
➡ Operations & Finance
➡ Transformation & Organization
➡ Information Systems
🔎 We are looking for a Senior Data Engineer to join our team in Porto.
📝 MAIN TASKS
- Design, build and operate real-time data products, from event ingestion to consumer-facing APIs.
- Develop stateful stream-processing applications using Apache Flink (DataStream API and Flink SQL), including windowing, event-time processing, watermarks, checkpointing and exactly-once delivery.
- Build and manage Kafka-based event streams, including topic design, schema management, schema evolution, partitioning and consumer groups.
- Design and implement REST and streaming APIs (e.g. WebSockets and Server-Sent Events) and the microservices behind them.
- Develop APIs with clear contracts, versioning, authentication and rate-limiting mechanisms.
- Deploy and operate services on AWS using containers, ECS/EKS, Lambda, API Gateway and Infrastructure as Code with Terraform.
- Optimize streaming jobs and services to meet latency, throughput, availability and cost targets.
- Integrate data products with the Group Data Platform, following data contracts, catalogue and access policies.
- Implement data quality, observability and alerting mechanisms, including metrics, logs, tracing, consumer lag and data freshness monitoring.
- Develop and improve CI/CD pipelines, automated testing and safe release practices.
- Provide production support, investigate incidents and implement permanent solutions.
- Produce technical documentation, solution designs and operational runbooks.
- Conduct code and technical design reviews, mentor engineers and promote real-time engineering best practices.
- Collaborate with platform teams, product owners and business stakeholders, challenging requirements and proposing efficient technical solutions.
🎯 YOUR PROFILE AND SKILLS
- Bachelor's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
- At least 7 years of experience in Data Engineering, including a minimum of 3 years building and operating real-time or streaming systems in production.
- Strong hands-on experience with Apache Flink in production, using Java, Scala or Python/PyFlink.
- Strong experience with Apache Kafka and event-driven architectures, including schema management (Avro, Schema Registry).
- Proven experience designing and developing REST APIs and microservices using Java and/or Python. Experience with streaming APIs is a plus.
- Solid experience with AWS services, particularly MSK, Kinesis, Managed Service for Apache Flink, Lambda, API Gateway, ECS/EKS, S3 and IAM.
- Experience with Terraform and Infrastructure as Code concepts.
- Knowledge of CI/CD practices and tools such as GitLab CI, GitHub Actions or equivalent.
- Strong SQL and data modelling skills, covering both streaming and batch data.
- Experience with automated testing, including unit, integration, contract and load testing.
- Knowledge of observability tools such as CloudWatch, OpenTelemetry, Prometheus and Grafana.
- Understanding of security principles for data services, including OAuth2/OIDC, authentication, authorisation, encryption and least-privilege access.
- Experience operating production services against SLAs and troubleshooting complex technical issues.
- Strong communication skills, with the ability to lead technical discussions and collaborate with engineering teams and business stakeholders.
- Strong analytical and problem-solving skills, with a proactive ownership mindset.
- Fluency in English is mandatory.
- Based in Portugal, with availability to work in Porto.
🎁 WHAT WE OFFER
- This role offers a gross annual salary between 45.000€ to 60.000€, defined according to the candidate’s experience and competencies
- Permanent contract (CTI).
- Meal allowance.
- Health Insurance.
- Extra holiday: Christmas Eve or New Year's Eve.
- Access to Quanteam Academy for continuous learning and certifications.
- Elevate Program to support professional development and career growth.
- Career progression plan.
- Ongoing follow-up and technical support provided by a Consultant Expert
Skills
- API
- API Gateway
- Authentication
- Avro
- AWS
- CI/CD
- CloudWatch
- Data Engineering
- Data Modeling
- Data Quality
- ECS
- EKS
- Event Driven Architecture
- Flink
- GitHub
- GitHub Actions
- GitLab
- Grafana
- IAM
- Infrastructure as Code
- Java
- Kafka
- Kinesis
- Lambda
- Microservices
- OAuth
- Observability
- OpenID
- OpenTelemetry
- Prometheus
- Python
- REST
- S3
- Scala
- SQL
- Terraform
- Test Automation
- Websockets
Sign in to read and join the discussion.