AVP/VP, AI Engineer, Data Management Office
Summary
Leads AI agent design, testing, and deployment for a financial group, focusing on robust, safe workflows and LLM evaluation in Python and Azure.
Responsibilities
Design, build and validate agentic AI workflows, including multistep reasoning and tooluse orchestration. Analyse agent and model behaviour to assess robustness, safety, error propagation and endtoend decision quality. Integrate APIs, external tools, plugins and system connectors required for agent operations. Develop and execute evaluation approaches for LLMs and agentic systems, including scenario tests, benchmarks and automated pipelines. Assess retrieval components, vector databases, memory systems and workflow reliability. Apply Responsible AI principles and regulatory expectations to model behaviour, documentation and control standards. Partner with data science, engineering, architecture and risk teams to ensure safe and compliant AI deployment. Collaborate with other departments' data scientists to understand modelling intent, technical assumptions and workflow logic. Contribute to AI/ML proofofconcept (POC) initiatives to strengthen evaluation practices and support innovation.
Requirements
Strong understanding of machine learning and LLM architectures, including components used in agentic AI systems. Minimum 3 years of relevant experience in AI/ML engineering, model development, model evaluation, or related technical roles. Knowledge of multistep reasoning, planning loops, workflow orchestration and tooluse frameworks. Experience evaluating LLMs, agents and AI systems through scenariobased testing, automated evaluations and diagnostic tracing. Ability to assess robustness, stability, drift, failure recovery behaviours and unintended model actions. Handson experience with API integration, workflow automation and intersystem tooluse; familiarity with Power Automate, Power Apps, Sharepoint is a plus. Understanding of AI governance frameworks, lifecycle controls, Responsible AI principles and regulatory expectations. Strong prompting, debugging, observability and tracing skills to diagnose complex multiturn agent behaviours. Ability to collaborate across technical, risk, governance and business teams effectively. Proficiency in Python (especially pyspark, MLlib), Hadoop stack and modern AI/ML frameworks such as PyTorch, TensorFlow, Keras, scikit-learn; familiarity with ML lifecycle framework/tool such as MLflow; familiarity with RAG, vector databases, retrieval systems and orchestration libraries is a plus. Familiarity with Copilot Studio is a huge plus. Strong familiarity with Microsoft Azure, Azure Databricks and Microsoft Purview is a huge plus.
Design, build and validate agentic AI workflows, including multistep reasoning and tooluse orchestration. Analyse agent and model behaviour to assess robustness, safety, error propagation and endtoend decision quality. Integrate APIs, external tools, plugins and system connectors required for agent operations. Develop and execute evaluation approaches for LLMs and agentic systems, including scenario tests, benchmarks and automated pipelines. Assess retrieval components, vector databases, memory systems and workflow reliability. Apply Responsible AI principles and regulatory expectations to model behaviour, documentation and control standards. Partner with data science, engineering, architecture and risk teams to ensure safe and compliant AI deployment. Collaborate with other departments' data scientists to understand modelling intent, technical assumptions and workflow logic. Contribute to AI/ML proofofconcept (POC) initiatives to strengthen evaluation practices and support innovation.
Requirements
Strong understanding of machine learning and LLM architectures, including components used in agentic AI systems. Minimum 3 years of relevant experience in AI/ML engineering, model development, model evaluation, or related technical roles. Knowledge of multistep reasoning, planning loops, workflow orchestration and tooluse frameworks. Experience evaluating LLMs, agents and AI systems through scenariobased testing, automated evaluations and diagnostic tracing. Ability to assess robustness, stability, drift, failure recovery behaviours and unintended model actions. Handson experience with API integration, workflow automation and intersystem tooluse; familiarity with Power Automate, Power Apps, Sharepoint is a plus. Understanding of AI governance frameworks, lifecycle controls, Responsible AI principles and regulatory expectations. Strong prompting, debugging, observability and tracing skills to diagnose complex multiturn agent behaviours. Ability to collaborate across technical, risk, governance and business teams effectively. Proficiency in Python (especially pyspark, MLlib), Hadoop stack and modern AI/ML frameworks such as PyTorch, TensorFlow, Keras, scikit-learn; familiarity with ML lifecycle framework/tool such as MLflow; familiarity with RAG, vector databases, retrieval systems and orchestration libraries is a plus. Familiarity with Copilot Studio is a huge plus. Strong familiarity with Microsoft Azure, Azure Databricks and Microsoft Purview is a huge plus.