Senior Data Engineer (Databricks and Python)

About the Opportunity
We are seeking a highly motivated Senior Data Engineer to join an enterprise technology team working on modern, AI-enabled data platforms. This role is ideal for an experienced engineer who enjoys building scalable data solutions, collaborating with cross-functional teams, and driving engineering best practices in a cloud-native environment.
You will work on enterprise-grade applications and data processing solutions using modern technologies including Azure, Databricks, Python, and microservices.
Key Responsibilities
Define and promote engineering best practices, coding standards, and development processes.
Conduct code reviews to maintain high-quality, maintainable software.
Design, develop, and maintain scalable Apache Spark applications.
Build efficient and reliable data processing and transformation pipelines.
Write clean, well-documented, and maintainable Python code.
Optimize applications for performance, scalability, and efficient data handling.
Collaborate with engineering, product, and business stakeholders to deliver high-quality solutions.
Troubleshoot and resolve technical issues to ensure system reliability.
Create and maintain technical documentation, workflows, and development standards.
Communicate effectively with internal teams and external stakeholders while supporting project delivery.
Required Qualifications
5+ years of professional software engineering or data engineering experience.
Strong experience with Apache Spark using Databricks and/or Azure Synapse/Fabric.
Advanced Python programming skills, including data manipulation with Pandas, Polars, or similar libraries.
Experience working with Parquet, Delta Tables, and other columnar storage formats.
Solid understanding of large-scale data processing, transformation, and analytics workflows.
Experience designing scalable applications using microservices architecture.
Strong analytical, troubleshooting, and problem-solving skills.
Excellent organizational, communication, and time management abilities.
Ability to work independently while collaborating effectively within distributed teams.
Preferred Qualifications
Experience with Microsoft Azure cloud services, including Data Lake, Blob Storage, Service Bus, Redis, and related services.
Familiarity with FastAPI.
Experience with Docker and Kubernetes.
Exposure to modern AI technologies, RAG pipelines, or large language model (LLM) applications is a plus.
Technology Stack
Azure Cloud
Apache Spark / Databricks
Python
Azure SQL
MongoDB
.NET 8 / ASP.NET Core
Microservices Architecture
GitHub Enterprise
Angular
Docker & Kubernetes
AI/LLM technologies (LangChain, LangGraph, RAG)