Production Data Scientist – Azure Databricks & R
Posted Updated
Description
Calleo is seeking an experienced Data Scientist to modernise and operationalise a microsimulation capability for production deployment within Azure Databricks and Delta Lake. You will optimise Spark workloads, design scalable data pipelines, and implement automated testing and version control while collaborating with data scientists, engineers and business stakeholders.
The role requires strong R programming expertise and a solid understanding of software engineering principles, with security
Analyse, review and optimise a complex microsimulation model for production deployment. Refactor and improve code quality, performance and maintainability. Optimise Spark workloads and improve computational efficiency across large-scale processing environments. Develop and maintain scalable data pipelines using Azure Databricks, Delta Lake and associated technologies. Implement best-practice software engineering and data engineering standards, including automated testing, version control, monitoring and error handling. Work closely with data scientists, analysts, engineers and business stakeholders to ensure accurate and effective model implementation. Produce technical documentation, operational procedures and architecture artefacts. Strong experience building analytical solutions in R and deploying models to production. Experience designing testing frameworks, validation processes and repeatable scenario testing. Solid knowledge of Apache Spark and large-scale data processing environments. Understanding of software engineering principles including modular design, automated testing and version control. Excellent stakeholder engagement and collaboration skills.Skills
Sign in to read and join the discussion.