Lead / Senior Data Engineer (Azure Databricks)

About the Role
We are looking for an experienced Lead / Senior Data Engineer to join a high-performing technology team delivering innovative enterprise and AI-driven solutions within a global professional services environment.
The team designs, develops, and deploys modern technology platforms and intelligent tools that support complex business and operational processes. You will work alongside experts in software engineering, data, AI, change management, and project delivery on initiatives ranging from solution design and development to deployment and continuous improvement.
This is an exciting opportunity to work on large-scale, modern cloud and data engineering projects using Azure, Databricks, Spark, Python, and AI-powered technologies.
Technology Stack
Azure Cloud
Azure Databricks
Apache Spark / PySpark
Microservices Architecture
.NET 8
ASP.NET Core
Python
MongoDB
Azure SQL
Angular
GitHub and AI-assisted development tools
LangGraph
LangChain
RAG Pipelines
Multimodal LLMs
Your Responsibilities
Define and promote engineering best practices and coding standards across the project
Conduct thorough code reviews to ensure high code quality and adherence to established standards
Work independently while collaborating closely with cross-functional and distributed teams
Provide technical guidance and clear direction to team members
Support the coordination of day-to-day technical activities and delivery priorities
Communicate regularly with business and project stakeholders
Design, develop, and maintain robust, scalable, and high-performance Spark applications
Write clean, maintainable, and efficient code following modern software engineering principles
Optimize applications and data-processing workflows for performance and scalability
Ensure efficient and reliable data handling across large-scale data platforms
Collaborate with engineering, product, data, and business teams to deliver high-quality solutions
Identify, investigate, and resolve complex technical issues
Create and maintain comprehensive technical documentation for code, processes, architectures, and workflows
Experience & Technical Skills
5+ years of hands-on experience in software development or data engineering
Practical experience developing SQL stored procedures and migrating legacy SQL logic into Spark SQL or PySpark environments
Extensive hands-on experience with PySpark and Azure Databricks
Strong knowledge of Delta Tables, cluster management, and workflow automation
Proven experience optimizing Apache Spark performance in production environments
Strong Python programming skills, particularly for complex data processing and manipulation using libraries such as Polars or Pandas
Solid understanding of columnar data-storage formats, particularly Parquet
Practical experience working with Delta Lake and modern lakehouse architectures
Strong expertise in data processing, transformation, and analytical workflows
Strong analytical and problem-solving skills with a detail-oriented mindset
Ability to balance engineering standards and structured processes with pragmatic delivery requirements
Solid understanding of microservices architecture and scalable distributed systems
Nice to Have
Experience working with Azure Cloud services or other major cloud platforms
Experience with cloud-based services such as messaging platforms, Data Lake, object storage, caching solutions, and related managed services
Familiarity with FastAPI
Experience with containerization and orchestration technologies such as Docker and Kubernetes
Exposure to AI-driven applications, RAG architectures, LLM orchestration frameworks, or modern generative AI solutions