Test Analyst - AI

Senior AI Test Analyst

Initial 12-Month Contract (Strong Extension Potential)

Competitive Daily Rate

Brisbane Based | Hybrid Working Arrangements

Enterprise AI & Digital Transformation Program

Are you an experienced Test Analyst who understands the difference between using AI to support testing and testing AI itself ?

We're seeking a Senior AI Test Analyst to join a major AI portfolio focused on evaluating enterprise AI solutions used at scale. This role sits at the intersection of traditional software testing and AI evaluation, making it ideal for someone with strong QA foundations who can critically assess AI behaviour, reliability, safety, and performance in real-world scenarios.

This role is perfect for a Test Analyst who can evaluate AI as the product under test and enjoys investigating edge cases, challenging model behaviour, and identifying where AI systems can fail.

Key Responsibilities

• Design and execute test strategies across both traditional software solutions and AI-enabled products
• Develop AI evaluation frameworks, benchmark datasets, and structured testing approaches
• Assess AI outputs for accuracy, reliability, consistency, safety, and hallucinations
• Perform adversarial testing, prompt variation analysis, and AI guardrail validation
• Conduct integration, regression, exploratory, and non-functional testing across enterprise environments
• Facilitate UAT, defect triage, root cause analysis, and formal test reporting for governance activities

Key Skills

• 7+ years' experience in software testing, QA, or test engineering within complex enterprise environments
• Demonstrated experience testing AI-enabled products, LLM applications, or Generative AI solutions
• Strong understanding of risk-based testing, exploratory testing, and quality assurance methodologies
• Experience evaluating AI outputs using experimentation, investigation, and structured assessment techniques
• Ability to analyse failures across prompts, retrieval layers, application logic, and underlying AI models
• Strong stakeholder engagement skills with the ability to communicate complex findings to technical and non-technical audiences

Highly Desirable

• Experience testing LLM-based applications or enterprise AI platforms in production environments
• Knowledge of hallucination testing, prompt injection testing, model drift, and AI safety evaluation
• Exposure to AI evaluation tools such as DeepEval, Promptfoo, Ragas, TruLens, Langfuse, or Arize
• Python, automation, API testing, and data analysis experience
• ISTQB CT-AI certification or equivalent AI testing credentials
• Higher education, Azure OpenAI, Canvas/LTI, or WCAG accessibility testing experience

This is a fantastic opportunity for a naturally curious tester who enjoys experimentation, challenging assumptions, and uncovering unexpected behaviours. If you've tested AI products and know how to measure quality beyond a simple pass/fail outcome, we'd love to hear from you.

To apply or learn more, reach out at ***email_hidden*** or View phone number on au.fitly.work .

Whizdom is a 100% Australian-owned recruitment company and a Veteran Employer of Choice (highest tier), proudly supporting government and commercial organisations across Australia.