Senior Data Engineer / Data Architect (Databricks)
Posted 5
views
Summary
Designs and builds production-grade data pipelines and end-to-end enterprise data platforms on Databricks and Apache Spark, working customer-facing to translate business needs into cloud solutions on AWS/Azure/GCP. Also covers consulting, delivery management, troubleshooting, and building reusable frameworks; Python/Scala skills required.
What You Will Be Doing
- Design and build production-grade data solutions and applications using Databricks.
- Develop scalable data pipelines, ingestion frameworks and distributed data processing solutions.
- Lead or contribute to the end-to-end architecture, design, build and deployment of enterprise data platforms.
- Work directly with customers to understand business and technical requirements and translate them into workable solutions.
- Guide customers on Databricks architecture, implementation and platform best practices.
- Integrate Databricks solutions with customers' existing cloud, data and application environments.
- Support the delivery of big data, analytics and, where applicable, ML/AI solutions.
- Work closely with project managers, architects, engineering teams and customer stakeholders to ensure successful delivery.
- Troubleshoot implementation issues and work with Databricks engineering/support teams where required.
- Develop reusable frameworks, accelerators, reference architectures and technical documentation.
- Participate in technical discussions, workshops and whiteboarding sessions with both technical and senior business stakeholders.
Key Requirements
- Strong background in Data Engineering, Data Platforms, Big Data or Software Engineering.
- Strong hands‑on experience with Databricks and Apache Spark.
- Proficiency in Python and/or Scala.
- Good understanding of distributed computing and Spark architecture/runtime.
- Experience designing and deploying production‑scale data architectures and pipelines.
- Experience with at least two major cloud platforms – AWS, Azure or GCP, with strong expertise in at least one.
- Familiarity with CI/CD and production deployment practices.
- Strong enterprise customer‑facing and technical consulting experience.
- Able to manage technical scope, timelines, stakeholders and delivery outcomes.
- Strong communication, documentation and solution‑design skills.
- Databricks certification is preferred.
- Experience with MLOps, ML/AI models or AI APIs is an advantage.