Python data platform engineer
Summary
Senior Python engineer building and optimizing an operational data mesh: containerized event-processing services on AWS ECS Fargate consuming Kafka (Confluent Cloud), serverless Lambda REST APIs, Kubernetes, and Snowflake. Also sets technical standards, reviews code, and acts as backup to the lead developer.
Senior Python Data Platform Engineer to join our integration engineering team powering our core Data Platform Service. Built on a tightly integrated modern stack consisting of AWS, Confluent Cloud, Kubernetes, and Snowflake, this platform serves as an operational data mesh. It delivers source- and consumer-aligned data products exposed via event streams, serverless REST APIs, and high-performance data stores to drive mission-critical, real-time transactional and operational workloads. In addition to hands-on software development, you will serve as a technical pillar within the team—guiding technical standards, conducting rigorous code reviews, and acting as the primary backup to the lead developer.
- Build, optimize, and maintain Python-based containerized event-processing services deployed as headless AWS ECS Fargate tasks consuming from Kafka.
- Design, implement, and maintain scalable AWS Lambda functions exposing clean REST APIs for data product consumption, implementing robust API versioning and alias routing strategies.
- Connect high-throughput event pipelines using Confluent Cloud (Kafka) to feed operational data stores and analytical engines like Snowflake.
- Partner closely with the lead developer as their primary backup, contributing to architectural decisions and enforcing high standards through active code reviews.
- Maintain reliable deployment pipelines using advanced Git workflows, automated testing, and continuous integration practices.
- 5+ years of relevant work experience
- Mastery of advanced object-oriented Python, classes, and metaprogramming.
- Deep experience with data classes, type validation, and algorithmic design patterns.
- Proven track record building REST APIs, AWS Lambda handlers, transformation services, and automation scripts.
- Strong expertise with Git workflows, merge requests, conventional commits, and CI/CD pipelines.
- Solid practical experience with database operations (both relational and non-relational), database design, and advanced SQL.
- Extensive experience handling semi-structured data structures (JSON, XML).
- Mandatory to be located in Portugal