Databricks Data Engineer | Senior

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Databricks Data Engineer based in Brazil.

We are seeking an experienced Senior Databricks Data Engineer to design, develop, and optimize scalable, high-performance data engineering solutions. In this role, you will leverage the Databricks ecosystem to build modern Lakehouse architectures and implement Medallion data models (Bronze, Silver, and Gold). You will be responsible for developing robust data pipelines, integrating multiple data sources, and enabling efficient data processing through technologies such as PySpark, Spark SQL, and Delta Lake. Working with Azure cloud services, you will ensure data quality, governance, security, and operational reliability across production environments. As a technical reference for the team, you will guide engineering best practices, support architectural decisions, and mentor other professionals. This is an opportunity to contribute to advanced data platforms while driving innovation, performance, and cost efficiency in a collaborative technology environment.

• Design, develop, implement, and continuously improve data engineering solutions using the Databricks ecosystem.
• Define and implement scalable data architectures based on Lakehouse principles and the Medallion architecture (Bronze, Silver, and Gold layers).
• Develop, optimize, and maintain data pipelines using Databricks, PySpark, Spark SQL, Apache Spark, and Delta Lake.
• Implement data ingestion, processing, and integration solutions, including Change Data Capture (CDC) and Structured Streaming.
• Build and manage data pipeline orchestration and system integrations using Azure Data Factory (ADF).
• Develop REST APIs, microservices, and automation solutions using Python.
• Establish and maintain data governance, security, lineage, and access control standards through Unity Catalog.
• Optimize data processing performance, scalability, and infrastructure costs through efficient resource allocation and workload management.
• Apply advanced performance tuning techniques, including Photon optimization, partitioning, shuffle management, and execution plan analysis.
• Implement and maintain observability, monitoring, and operational support practices for production data pipelines.
• Manage Databricks Workflows, CI/CD processes, Git repositories, and Declarative Automation Bundles.
• Identify and resolve complex technical challenges while making architectural decisions aligned with business and engineering requirements.
• Act as a technical reference for the data engineering team, mentoring professionals, sharing knowledge, and establishing engineering best practices.

Requirements:

• Extensive professional experience delivering and maintaining production-grade data engineering solutions using Databricks.
• Advanced proficiency in PySpark, Spark SQL, Delta Lake, and Apache Spark.
• Hands-on experience with Delta Live Tables (DLT) / Lakeflow Spark Declarative Pipelines, Change Data Capture (CDC), and Structured Streaming.
• Strong knowledge of Liquid Clustering, Auto Loader, and Delta table optimization techniques.
• Advanced understanding of Unity Catalog, including data governance, security, lineage, and access management.
• Proven experience designing and implementing scalable data architectures, particularly Lakehouse and Medallion architectures.
• Practical experience using Azure Data Factory (ADF) for data ingestion, orchestration, and system integration.
• Advanced Python programming skills, including the development of REST APIs, microservices, and automation solutions.
• Strong expertise in Spark Performance Tuning, including Photon, partitioning strategies, shuffle optimization, and execution plan analysis.
• Experience with Databricks Workflows, CI/CD pipelines, Git, and Declarative Automation Bundles.
• Knowledge of workload optimization, computational resource sizing, and cloud cost reduction strategies.
• Experience with observability, monitoring, troubleshooting, and production pipeline support.
• Ability to independently solve complex technical problems and make sound architectural decisions.
• Previous experience acting as a technical lead or reference, supporting, mentoring, and guiding other engineering professionals.
• Strong analytical thinking, problem-solving skills, autonomy, and effective communication and collaboration abilities.

Required and Preferred Certifications:

• Mandatory: Databricks Certified Data Engineer Associate.
• Strongly preferred: Databricks Certified Data Engineer Professional.
• Nice to have: Databricks Certified Associate Developer for Apache Spark.
• Additional certifications in AWS, Microsoft Azure, or Google Cloud Platform (GCP), particularly in Data Engineering and Solutions Architecture, are considered an advantage.

Benefits:

• Food and meal allowance: Food and/or meal vouchers to support daily expenses.
• Healthcare: Medical and dental insurance coverage.
• Pet healthcare: Health insurance for your pet.
• Remote work support: Home office allowance for remote positions.
• Welcome kit: Onboarding kit to help you get started in your new role.
• Life insurance: Additional financial protection and security.
• Mental health and wellness: Access to Zenklub, including psychological and nutritional support.
• Fitness benefits: TotalPass membership for physical activities and wellness.
• Professional development: Continuous training and certification programs with leading technology partners, including Adobe, Anthropic, AWS, Google, Microsoft, and Salesforce.
• Learning and development partnerships: Access to exclusive partnerships, discounts, and resources to support your professional growth.
• Digital content: Access to UOL content and resources.
• Maternity support: Special maternity kit.
• Extended parental leave: Enhanced maternity and paternity leave.
• Employee referral program: Opportunities to recommend talented professionals through an internal referral initiative.