Pass/fail testing is built on same-input-same-output determinism. AI agents violate that assumption on every run. This guide explains the golden dataset methodology, the five metrics every release must report, and why red-team failures are your most valuable QA asset.
Every engineering team that has shipped a non-trivial AI agent has hit the same wall: the QA team runs the test suite, everything passes, and the agent still does something unexpected in production. The root cause is not that the tests were poorly written. The root cause is that the testing methodology was designed for a different class of system entirely.
Traditional software QA is built on determinism. You define expected outputs for given inputs. The test runner compares actual to expected. Green means the behavior is consistent. This methodology works beautifully for APIs, databases, business logic, and user interfaces. It fails completely for language model agents, because the fundamental property it depends on — same input produces same output — does not hold for any system with a temperature parameter above zero.
This is not a solvable problem within the traditional QA paradigm. It requires a different paradigm: evaluation over distributions, behavioral policies instead of output assertions, and continuous adversarial probing as the primary quality signal. This guide explains that paradigm and provides the five concrete metrics that every agent release should instrument and report.
Non-determinism in AI agents means that for any given input, the output — and the tool call sequence that produces it — varies across runs. This is not a bug; it is an intentional property of language models that enables generalization. But it makes assertion-based testing useless for behavioral validation. You cannot write "assert output == expected_output" when expected_output changes on every invocation.
The replacement methodology is evaluation over distributions. Instead of testing whether a specific input produces a specific output, you test whether the agent's behavior — its tool usage patterns, its output quality, its error rates — stays within a defined policy envelope across a large sample of inputs. This is fundamentally statistical quality assurance: you are asking "what is the probability that this agent behaves correctly?" not "did this agent produce the right answer?"
Golden datasets are the foundation of this approach. A golden dataset is a curated set of input examples with defined behavioral expectations — not exact expected outputs, but policy constraints like "must not call the delete_record tool on this input" or "output must contain a citation to a real document" or "decision must be escalated to human if confidence is below 0.7." Policy-as-code evaluation frameworks like DeepEval allow these constraints to be expressed as testable metrics and run at scale.
A golden dataset is a curated collection of representative inputs paired with behavioral policy constraints rather than exact expected outputs. Policy-as-code evaluation runs the agent against the golden dataset and measures whether behavioral metrics (hallucination rate, tool misuse rate, etc.) fall within defined thresholds. Unlike pass/fail tests, golden dataset evaluation produces statistical quality signals that are valid even when individual outputs vary across runs.
Based on production deployment patterns and the OWASP LLM Top 10 (2025) risk taxonomy, five metrics together provide a complete picture of agent release quality. Each metric captures a distinct failure mode; reporting all five prevents teams from gaming any individual metric at the expense of overall agent safety.
| # | Metric | Definition | Threshold (Example) | Tool |
|---|---|---|---|---|
| 1 | Hallucination Rate | % of outputs containing factual claims not supported by retrieved context or tool results | < 2% for high-stakes domains | DeepEval (Faithfulness) |
| 2 | Tool Misuse Count | Count of unauthorized tool invocations per 1,000 requests (calls outside the agent's defined tool policy) | = 0 in pre-prod gate | Promptfoo, custom policy |
| 3 | Cost / Successful Task | Total token + API cost divided by tasks completed within policy constraints (not just completed) | Set per use case, trend alert | LLM provider telemetry |
| 4 | Escalation Rate | % of requests routed to human review; too high = agent undertrusted; too low = missing edge cases | 5–20% typical range | Agent tracing (OTLP) |
| 5 | Adversarial Completion Rate | % of red-team adversarial inputs that completed a task the agent should have refused or escalated | < 1% for regulated domains | PyRIT, Promptfoo |
Cost per successful task is the metric most commonly omitted from early eval frameworks. It measures efficiency at the task level, not the token level — an agent that completes a task in 12 steps when 3 would suffice is burning budget and introducing unnecessary latency and failure surface. Tracking this metric across releases also surfaces model degradation: a new model version that increases cost per successful task by 40% is likely generating more tool call loops, which is a behavioral regression even if hallucination rate holds steady.
Red-teaming for AI agents means running systematic adversarial probes against the agent — prompts designed to trigger prompt injection, jailbreaking, excessive agency, data leakage, and the other failure modes in the OWASP LLM Top 10 (2025). Per OWASP, LLM01 (Prompt Injection) is the highest-priority risk for deployed agents, and LLM06 (Excessive Agency) directly addresses the tool misuse failure mode measured in metric #2 above.
The critical insight is that red-team failures are not merely bugs to be fixed — they are the highest-value inputs for your golden dataset. When PyRIT discovers a prompt injection vector that causes your agent to call a write-access tool without authorization, that adversarial input becomes a golden dataset entry with the behavioral constraint "must not invoke write-access tools on this class of input." The next release is tested against that constraint, and the test remains in the suite permanently, ensuring the fixed behavior doesn't regress.
This creates a self-improving evaluation loop: each production incident or red-team finding strengthens the golden dataset, which raises the quality bar for the next release. Over time, the golden dataset becomes an accurate model of the agent's actual failure surface — which is far more valuable than a test suite that only covers the happy path.
The OWASP LLM Top 10 (2025) is the Open Worldwide Application Security Project's ranked list of the top security risks for large language model applications. For AI agents, the most relevant risks are LLM01 (Prompt Injection — malicious inputs hijacking agent instructions), LLM06 (Excessive Agency — agents taking unintended actions due to overly broad permissions or insufficient output validation), and LLM04 (Data and Model Poisoning — contamination of training or context data). All three require active red-teaming to detect before production deployment.
Pre-production gate evaluation answers the question: "Is this version of the agent safe to deploy?" It runs the full golden dataset eval suite, generates the 5-metric scorecard, and either blocks or permits the deployment based on whether all metrics meet their defined thresholds. This is a binary checkpoint: the agent either passes the gate or goes back for remediation.
Production shadow evaluation answers a different question: "Is this deployed agent behaving as expected in real traffic conditions?" Shadow eval instruments the production agent to capture a sample of live requests, runs the same metric calculations on live outputs (or near-live, with a delay buffer for human review of flagged samples), and surfaces behavioral drift as it happens. If the hallucination rate in production gradually climbs from 1.2% toward 2.5% over two weeks, shadow eval catches this before it becomes a customer-facing incident.
Both are necessary. Pre-production gate catches issues before they reach users. Shadow eval catches behavioral drift, distribution shift, and emergent failures that only appear under real-world input diversity. Neither alone is sufficient: a pre-prod gate without shadow eval misses post-deployment drift; shadow eval without a pre-prod gate exposes users to every new release before it's been validated.
These three tools address different layers of the evaluation stack and are complementary rather than competing. DeepEval provides the measurement layer: it implements quantitative metrics for hallucination, answer relevancy, contextual recall, and tool call correctness. You use DeepEval to run the golden dataset against a candidate agent version and produce the 5-metric scorecard.
Promptfoo provides the probing layer: it runs prompt variants, adversarial scenarios, and comparative evaluations across model versions or configurations. It is particularly useful for regression testing across deployments — ensuring that a model update or prompt change doesn't cause behavioral regression on any of the golden dataset scenarios.
PyRIT provides the attack layer: it automates the adversarial probing that populates your red-team golden dataset entries. PyRIT systematically probes for OWASP LLM Top 10 vulnerabilities and produces findings that, when remediated, become permanent golden dataset constraints. Running PyRIT before every major release is the closest thing to a penetration test that exists for LLM agents today.
The 5-stage eval pipeline above maps directly onto AgentTrust OS's three product functions. Trust Certify implements the pre-production gate: it runs golden dataset evaluation against a candidate agent version, produces the 5-metric scorecard, and issues a certification artifact that includes the policy constraints validated, the red-team scenarios run, and the metric thresholds achieved. Trust Runtime instruments the production shadow eval: it captures live request/response pairs, continuously measures the five metrics against production traffic, and triggers alerts when any metric breaches its threshold. Trust Audit provides the trace-and-explain layer: every agent decision is logged with a structured explanation that satisfies both engineering post-mortems and regulatory audit requirements.
Trust Certify implements the pre-production gate with the 5-metric scorecard built in. Trust Runtime runs continuous shadow eval in production. Get the eval methodology operationalized in your pipeline.
Start Free →