Senior Data Engineer / Data Architect (Databricks)
Posted 3
views
Summary
Designs and builds production-grade data platforms, pipelines, and analytics/ML solutions on Databricks for enterprise customers in a consulting capacity, guiding clients on architecture and best practices. Core stack: Databricks, Apache Spark, Python/Scala, and major clouds (AWS, Azure, GCP) with CI/CD delivery.
What You Will Be Doing
Design and build production-grade data solutions and applications using Databricks.
Develop scalable data pipelines, ingestion frameworks and distributed data processing solutions.
Lead or contribute to the end-to-end architecture, design, build and deployment of enterprise data platforms.
Work directly with customers to understand business and technical requirements and translate them into workable solutions.
Guide customers on Databricks architecture, implementation and platform best practices.
Integrate Databricks solutions with customers' existing cloud, data and application environments.
Support the delivery of big data, analytics and, where applicable, ML/AI solutions.
Work closely with project managers, architects, engineering teams and customer stakeholders to ensure successful delivery.
Troubleshoot implementation issues and work with Databricks engineering/support teams where required.
Develop reusable frameworks, accelerators, reference architectures and technical documentation.
Participate in technical discussions, workshops and whiteboarding sessions with both technical and senior business stakeholders.
Key Requirements
Strong background in Data Engineering, Data Platforms, Big Data or Software Engineering.
Strong hands-on experience with Databricks and Apache Spark.
Proficiency in Python and/or Scala.
Good understanding of distributed computing and Spark architecture/runtime.
Experience designing and deploying production-scale data architectures and pipelines.
Experience with at least two major cloud platforms – AWS, Azure or GCP, with strong expertise in at least one.
Familiarity with CI/CD and production deployment practices.
Strong enterprise customer-facing and technical consulting experience.
Able to manage technical scope, timelines, stakeholders and delivery outcomes.
Strong communication, documentation and solution-design skills.
Databricks certification is preferred.
Experience with MLOps, ML/AI models or AI APIs is an advantage.