Senior Data Engineer – Databricks Lakehouse Platform

This role is for professionals seeking a long-term career with us; subcontractor applications will not be considered.

We are seeking an experiencedSenior Data Engineerto design, build, and optimize modern

data platforms using theDatabricks Lakehouse Platform. The ideal candidate will have10+

years of data engineering experience

years of data engineering experience

and deep expertise inLakehouse

Architecture,Medallion Architecture, cloud-native data solutions, and enterprise-scale data

governance practices.

Key Responsibilities

Data Platform Architecture & Engineering

• Design and implement scalable, high-performance data platforms using the Databricks Lakehouse architecture.
• Lead the development of enterprise data solutions leveraging Bronze, Silver, and Gold layers following Medallion Architecture principles.
• Build and maintain batch and streaming data pipelines for large-scale data processing.
• Develop reusable data frameworks, accelerators, and engineering best practices.
• Optimize data storage, processing, and query performance across cloud environments.

Databricks Development

• Design, develop, and maintain Databricks workflows using PySpark, SQL, and Python.
• Implement and manage Delta Live Tables (DLT) pipelines.
• Utilize Unity Catalog for centralized governance, metadata management, access controls, and lineage tracking.
• Support advanced analytics, machine learning, and AI workloads on the Databricks platform.
• Monitor and troubleshoot production data pipelines and platform performance.

Data Governance & Quality

• Establish and enforce enterprise data governance standards and policies.
• Implement data quality frameworks, validation rules, reconciliation processes, and monitoring solutions.
• Ensure compliance with security, privacy, regulatory, and audit requirements.
• Drive metadata management, lineage, cataloging, and data stewardship initiatives.
• Collaborate with business and governance teams to improve trust and usability of enterprise data assets.
• Design and deploy cloud-native data solutions on Azure, AWS, or GCP.
• Implement scalable and secure cloud architectures aligned with enterprise standards.
• Optimize cloud resource utilization, performance, and cost management.
• Integrate cloud-native services with Databricks-based solutions.

DevOps & Automation

• Automate deployment, testing, monitoring, and operational processes.
• Apply Infrastructure as Code (IaC) practices using tools such as Terraform or similar.
• Promote engineering excellence through code reviews, testing strategies, and release management processes.
• Provide technical leadership and mentorship to junior and mid-level engineers.
• Collaborate with architects, analysts, data scientists, product owners, and business stakeholders.
• Contribute to platform roadmaps, architectural decisions, and technology evaluations.
• Lead technical discussions and establish best practices across the data engineering organization.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related field.
• 10+ years of experiencein Data Engineering, Data Warehousing, or Big Data platforms.
• 5+ years of hands-on experience with Databricksin enterprise environments.
• Strong expertise in:
• Lakehouse Architecture
• Medallion Architecture
• Unity Catalog
• Data Governance and Data Quality frameworks
• Data Modeling and Data Warehousing concepts
• Extensive experience with at least one major cloud platform:
• Amazon Web Services (AWS)
• Advanced programming experience with:
• Python
• SQL
• PySpark
• Strong understanding of CI/CD methodologies and DevOps practices.
• Experience implementing Infrastructure as Code (Terraform).
• Expertise in performance tuning, optimization, and troubleshooting of large-scale data pipelines.
• Strong understanding of security, access controls, and compliance requirements in cloud environments.

Preferred Qualifications

• Databricks certifications such as 'Databricks Certified Data Engineer Professional'
• Cloud certifications (Azure, AWS, or GCP).
• Experience with real-time streaming technologies and event-driven architectures.
• Experience supporting AI/ML and Generative AI data platforms.
• Familiarity with data observability tools and modern data quality frameworks.
• Experience implementing enterprise metadata management and data lineage solutions.

Hiring Process

We aim to complete the interview process in October-November. The targeted onboarding is January 2027 or earlier.