Data Engineer (Databricks)
Summary
Builds lightweight analytics apps in Python (Streamlit) to support training and onboarding, integrates data from multiple sources, and maintains Databricks pipelines.
Data Engineer (Databricks)
Whitehall Resources currently require an experienced Data Engineer (Databricks) to work with a key client.
**Please note this role requires falls INSIDE IR35**
Responsibilities
- Design build, test, deploy and maintain lightweight analytics application, primarily using python and streamlit, to support training and onboarding
- Translate business and user requirements into intuitive application workflows, well-structured data models and transparent metrics.
- Work closely with Data engineers to ensure data pipelines and reusable data sets meet application and analytical needs
- Maintain application code in version control and contribute to testing, technical documentation and development best practices
- Develop and maintain reusable datasets, semantic layer, data quality checks, monitoring and reporting for the data used by the applications.
- Gather user feedback and iteratively improve application usability, accessibility and adoption
- Build dashboards and visualisations where they are the most appropriate way to communicate onboarding, adoption and business metrics
- Support onboarding of new users to the data platform, improving accessibility and usability
Required Skills and Experience
- Hands on experience developing data or analytic applications in Python using Streamlit, Dash or a similar framework
- Experience designing, testing and maintaining user-facing applications, including the use of version control
- Strong SQL skills and experience analysing and integrating data sets from multiple sources
- Experience with Databricks (or similar modern data platforms)
- Working experience with notebooks and Python data libraries and processing frameworks (e.g. Pandas, PySpark)
- Understanding of deployment, CI/CD, orchestration and application support practices.
“Nice To Have” Skills and Experience
- Experience building dashboards and visualisation using (Databricks SQL, Power BI, Tableau, or a similar tool)Familiarity with data quality frameworks and automated testing approaches
- Understanding of data engineering concepts (pipelines, orchestration, version control)
- Exposure to working in a product-oriented or data platform environment