Data Engineer Sr. Associate
Description
The Senior Data & AI Engineer is the hands-on owner of CSBR's lending data lake on AWS and a full-stack builder of the data products around it. Reporting to the Data Lead, the Engineer runs the pipelines and lakehouse that feed portfolio, market, and credit-decision data; uses AI coding agents to ship the applications that put that data to work; and contributes to credit modelling and data-science research, working closely with the investment, credit, finance, and product teams and with platform engineering.
Data Lake & Pipelines
- Own the lending data lake end to end – a lakehouse of open-format tables on S3, Glue, and Athena – and the orchestrated pipelines on ECS Fargate that load it, from raw ingestion to analytical Gold tables and serving snapshots.
- Build and maintain ingestion from public and market sources (BCB, B3, CVM, IBGE, MTE, Receita Federal, PGFN) and from internal systems, partners, and fund administrators, including scraped and semi-structured data.
- Build the pipelines that bring data from CSBR's proprietary lending products into the lake, with stricter handling of personal data (PII): minimization, masking or pseudonymization, segregated access, and audit trails.
- Model the datasets behind FIDC portfolio valuation and covenant monitoring, market intelligence, vehicle pricing and risk, and credit-decision features, each with a table contract, lineage, and documentation.
Data Quality & Operations
- Guarantee quality and freshness with data contracts, automated quality checks, reconciliation against source systems, and alerting, and lead the fix when a pipeline or a number is wrong.
- Deliver every change through automated tests, infrastructure as code, and CI/CD, with gated deploys and auditable evidence, and keep Athena, Fargate, and storage costs under control.
Data Products & AI-Assisted Development
- Build and evolve the applications on top of the lake – the SQL query portal and its interface for AI assistants, the FIDC portfolio monitoring app, and the market-intelligence and vehicle dashboards – as web front ends and APIs running serverless on Lambda and Amplify.
- Develop LLM features on Amazon Bedrock – the portal's data assistant, covenant extraction from fund documents – with evaluation sets, cost tracking, and release gates.
Data Science & Modelling
- Prepare model-ready data – point-in-time features, labels, and reference populations – and run batch scoring and model monitoring as pipelines in the lake.
- Partner with the Data Lead on research and development of credit and employer-risk models: exploratory analysis, feature engineering, fitting, validation, and calibration, with tracked, reproducible experiments under the team's modelling protocol.
- Contribute to CSBR's long-horizon data-science research project, turning research prototypes into reproducible, production-grade code.
Collaboration & Special Projects
- Turn questions from the investment, credit, and finance teams into datasets, analyses, and tools, and explain the results clearly to non-technical colleagues.
- Set engineering standards for data work – pipeline guides, templates, code review – and mentor the colleagues who contribute to the lake.
- Support new lending products as they launch, and other initiatives as prioritized by the Data Lead.
Requirements
- Native or bilingual Portuguese and professional English, equivalent to CEFR B2 or above.
- Degree in Computer Science, Engineering, Statistics, Mathematics, or a related field, or equivalent experience.
- 5+ years in data engineering or closely related roles, including owning production pipelines on a cloud data lake or lakehouse; financial-services experience – banks, fintechs, lenders, or asset managers – preferred.
- Expert Python and SQL: tested, typed, well-packaged code, and analytical SQL over large, partitioned datasets in Athena or a comparable query engine.
- Production experience with workflow orchestration and lakehouse design: dependency-aware scheduling, partitioned and incremental loads, backfills, idempotent runs, open table formats, schema evolution, and layered Bronze/Silver/Gold modelling.
- Solid AWS for data workloads (S3, Glue, Athena, Lambda, ECS Fargate, DynamoDB, IAM), delivered through infrastructure as code and CI/CD.
- Ability to ship a web application end to end with AI assistance: API design, authentication, a component-based front end, and automated deployment.
- Working data-science foundation: statistics, supervised learning with linear and gradient-boosted models, validation and calibration, and reproducible experiment tracking.
- Understanding of credit and lending data – portfolios, cash flows, delinquency, provisioning – and of LGPD; familiarity with Brazilian public data (BCB, CVM, IBGE, Receita Federal) is a plus.
- Autonomy and clear communication: able to own a problem from question to production in a small team, document decisions, and explain data to business stakeholders.
- Desirable: credit scoring or credit-risk modelling; FIDC or securitization analytics; LLM applications on Amazon Bedrock, tool-using agents, and LLM evaluation; entity resolution or graph analysis on company data (CNPJ); SageMaker; AWS data or machine-learning certifications.
Skills
- AI
- Analytics
- API
- Api Design
- Athena
- Authentication
- AWS
- AWS Bedrock
- CI/CD
- Cloud
- Data Engineering
- Data Lake
- Data Quality
- Data Science
- DynamoDB
- ECS
- Fargate
- Feature Engineering
- IAM
- Infrastructure as Code
- Lakehouse
- Lambda
- Lgpd
- LLM
- Machine Learning
- Python
- S3
- SageMaker
- Serverless
- SQL
- Statistics
- Workflow Orchestration
Sign in to read and join the discussion.