Data Engineer
Summary
Hands-on Databricks data engineer building and optimizing pipelines (PySpark, Python, SQL, Delta Lake) that transform raw data into curated, business-ready datasets, with production deployments and support via Azure DevOps. 6-month contract with possible extension; hybrid, Brazil only.
Role Overview
We are seeking a hands-on Databricks Engineer to build, optimize, and support data pipelines that transform data from the Raw layer into curated, business-ready datasets.
The Raw data layer and underlying cloud infrastructure are managed by our Platform team. You will work closely with Data Engineering, Platform, and business stakeholders to develop reliable data solutions and support production workloads.
- Location: Brazil only
- Work Arrangement: Hybrid – 2–3 days onsite per week
- Contract Duration: 6 months initially, with possibility of extension
Key Responsibilities
- Design, develop, and optimize Databricks data pipelines using PySpark, Python, SQL, and Delta Lake to transform Raw data into curated and business-ready layers.
- Manage production deployments through Azure DevOps (ADO), including code reviews, testing, and release management.
- Track development tasks, progress, and deliverables using ADO Boards.
- Monitor, troubleshoot, and optimize data pipelines and jobs to improve performance, reliability, and cost efficiency.
- Implement data quality checks and support data governance practices using Unity Catalog.
- Collaborate with business stakeholders and the Platform team to translate business requirements into technical solutions.
- Maintain technical documentation, operational procedures, and production support runbooks.
Required Technical Skills
- Strong hands-on experience with Databricks, including notebooks, jobs, workflows, and cluster management.
- Proficiency in PySpark, Python, and advanced SQL.
- Solid understanding of Delta Lake, data modeling, and ETL/ELT development patterns.
- Experience supporting production data workloads, including debugging, troubleshooting, and data validation.
- Working knowledge of Git and CI/CD practices, preferably with Azure DevOps (Repos, Pipelines, and Boards).
Nice-to-Have Skills
- Experience with Unity Catalog, Delta Live Tables (DLT), and Databricks Asset Bundles.
- Databricks certification, preferably Data Engineer Associate or Professional.
- Exposure to retail or convenience store data, particularly loyalty program data.
- Experience in the fuel or petroleum industry is a plus.
Experience and Qualifications
- 5+ years of overall data engineering experience.
- 3+ years of hands-on Databricks experience in production environments.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
- Excellent written and verbal English communication skills.
Skills
Sign in to read and join the discussion.