Data Engineer (Python / AWS & AI)
Description
As the founding entity of RAINBOW PARTNERS, Quanteam is a consulting firm specialized in Banking, Finance, and Financial Services. Through our core human values – proximity, teamwork, diversity, excellence – our 1000 expert consultants, hailing from 35 different nationalities, collaborate across 13 international offices: Paris, Lyon, New York, Montreal, London, Brussels, Luxembourg, Geneva, Lisbon, Porto, Casablanca, Madrid and Singapore.
We are looking for a Data Engineer to join our Power Markets data team, working on critical pipelines that support energy trading and market operations across North America.
What you'll do
Build and maintain robust, production-grade data pipelines on AWS to process ISO market data (settlements, auctions, congestion rights). You'll work on incremental loads, data quality frameworks, and cloud-native orchestration in a fast-moving energy environment.
Must have
- Python (pandas/Polars, clean modular code, pytest)
- SQL — complex queries, CTEs, window functions — SQL Server/T-SQL preferred
- PySpark via AWS Glue or Databricks
- Incremental/change-only loads, idempotent reruns, partitioning
- Data quality: row reconciliation, schema validation, audit logging
- AWS: S3, Glue, Lambda, Step Functions, Batch, Athena, EventBridge, Redshift, CloudWatch, Secrets Manager
- Snowflake (Python connector, key-pair auth, data shares)
- Git + CI/CD (GitHub Actions or equivalent), Docker
Nice to have
- Databricks: Delta Lake, Unity Catalog, workflows
- Azure: Blob/ADLS Gen2, ADF, Key Vault, Entra ID
- Terraform / CloudFormation / CDK
- Plotly Dash or Streamlit
- Selenium / Playwright
- Energy market data knowledge (ISO, Yes Energy, etc.)
- AWS, Azure or Databricks certification
Skills
- Athena
- AWS
- Aws Glue
- Azure
- CDK
- CI/CD
- Cloud
- Cloud Native
- CloudFormation
- CloudWatch
- Data Pipelines
- Data Quality
- Databricks
- Delta Lake
- Docker
- Entra ID
- Eventbridge
- Git
- GitHub
- GitHub Actions
- Lambda
- pandas
- Playwright
- Plotly
- Polars
- PySpark
- pytest
- Python
- Redshift
- S3
- Selenium
- Snowflake
- SQL
- SQL Server
- Streamlit
- Terraform
- Unity
- Unity Catalog
- Vault
Sign in to read and join the discussion.