Senior Software Engineer, Data Science
Summary
Build and maintain AI evaluation infrastructure for LLM-driven features in Progress AI, focusing on prompt optimization, dataset quality, and model accuracy measurement.
Senior Software Engineer, Data Science at Dronedeploy.
About the role
You will lead the effort to ensure our AI systems deliver reliable results by managing the accuracy of LLM-driven features. This role focuses on building evaluation infrastructure, refining prompt lifecycles, and creating tools to measure and improve model performance across our Progress AI product suite.
Key facts
Location: Auckland, New Zealand
Engagement: 6-month full-time contract
Team: AI Accuracy team (reporting to the Senior Manager of Software Engineering for AI Insights/Inspection AI)
What you'll do
- Develop and maintain infrastructure for evaluation datasets and ground-truth quality assurance.
- Identify and resolve discrepancies between offline evaluation results and production accuracy.
- Manage the end-to-end prompt optimization lifecycle from initial selection to post-launch monitoring.
- Scale labeling frameworks and knowledge capture systems.
- Create automated evaluation pipelines and internal tools for error analysis and auditing.
- Influence model performance by connecting ontology and schema design to measurable accuracy outcomes.
Requirements
- Professional proficiency in Python for data analysis and tooling, specifically using pandas and NumPy.
- Experience with SQL, database management, and large-scale data modeling.
- Background in data science with a focus on LLM accuracy and visual reasoning.
- Experience building and debugging evaluation datasets using tools like Braintrust or Langfuse.
- Ability to design labeling systems and quantify data quality and label agreement.
- Knowledge of prompt engineering lifecycles and deployment strategies.
- Understanding of how schema and ontology design impacts downstream models and tooling.
- Ability to work in the Auckland office at least two days per week and maintain 9:00 a.m. to 5:00 p.m. business hours.
- Comfort using AI-assisted tools for coding and debugging while maintaining high standards for code quality and safety.
Nice to have
- Domain expertise in the construction or industrial inspection sectors.
Skills & tools
- Python (pandas, NumPy)
- SQL
- Braintrust
- Langfuse
- LLM evaluation and prompt engineering
- Data modeling and ontology design
Practical notes
- This position requires regular in-office presence in Auckland to collaborate with the Inspection and Progress AI teams.
- You may be required to participate in occasional on-call rotations.
- Domestic and international travel for team onsites may be necessary.
- Drone certification is not required.
- All staff must adhere to company information security policies to protect customer and corporate data.