Agent eval acceptance cases: check the tool route, supported answer and grader separately

Start with the decision the answer must support

A support agent looks up the right order and returns a polished answer: the item is eligible because it arrived 12 days ago. If the order is marked final sale, the lookup was correct but the conclusion was not. An evaluator that checks only for a lookup or a mention of the return window can miss the defect.

OpenAI's agent-evaluation guidance distinguishes inspecting individual traces to diagnose workflow behavior from using repeatable datasets and evaluation runs to compare changes.[1] Its best-practices guide separately identifies functional correctness, tool selection and argument precision.[3] Those distinctions are useful even if your evaluation lives in a local case file rather than a vendor dashboard.

Build three checks: did the agent retrieve the right evidence; does its answer follow from that evidence; and does the grader agree with a defensible reference judgment? Here, a grader is a rule or model that labels an output. A reference judgment is the expected meaning and its evidence-based reason, not a sentence the agent must copy.

1. Freeze the task and evidence before scoring

Define a narrow objective: answer whether an order meets the stated criteria for requesting a return, or explain exactly what information is missing. The agent cannot approve a refund, process a return or change an order. Start with synthetic data and read-only tool fixtures, meaning fixed responses supplied by a test harness. This guide requires no paid API run, customer-data upload or production connection.

The following fictional contract is DEMO-SUPPORT-v1. Each caller is authorized to ask about the synthetic orders. POLICY-DEMO-v1 applies to standard physical items in DemoMarket. It permits return requests for unopened items at an elapsed delivery age of 0 through 30 days inclusive, and excludes final-sale items regardless of age. It grants no refund approval or execution authority. Delivery age is a supplied integer fixture field, not a calculation from live dates.

• lookup_order takes exactly one string argument, order_id. A successful fixture returns status=ok, order_id, region, product_type=standard_physical, unopened, final_sale, delivered_age_days and policy_id=POLICY-DEMO-v1. An unavailable response contains status=unavailable and the requested order_id, but no order facts.
• get_policy takes exactly one string argument, policy_id. For POLICY-DEMO-v1 it returns status=ok, region=DemoMarket, product_type=standard_physical, min_age_days=0, max_age_days=30, requires_unopened=true, excludes_final_sale=true and refund_execution_authorized=false. These fields encode only the fictional rule above.
• For a request naming an order, retrieve that order and POLICY-DEMO-v1; either call order is valid. Judge only this explicit read-only contract. For a request with no order identifier, ask for the ID without a speculative lookup. A region that remains unknown requires clarification before an eligibility conclusion.
• POLICY-DEMO-v1 is the only authoritative policy in this fixture set. In your own set, specify applicability and which approved version takes precedence. If competing evidence leaves that unresolved, hold the reference judgment instead of choosing whichever rule makes the answer pass.

Preserve the exact request, relevant conversation context, tool schema, fixed responses and reference rationale under a stable case ID. Record model identifier, prompt version, retrieval snapshot, application/runtime version, grader version and rubric version separately. Observable events and the final answer are enough for this procedure; it does not require the agent's private chain of thought. Remove credentials and unnecessary personal data from any later diagnostic record.

2. Build cases for ordinary answers and plausible mistakes

OpenAI recommends task-specific evaluation, ordinary and edge cases, and human feedback to calibrate automated scoring.[3] The small set below illustrates those categories. It is a worked design, not a validated sample size or an estimate of customer traffic. None of these cases has been run.

The next table completes each case's reference judgment. For cases that call tools, the literal arguments are order_id set to the named ID and policy_id set to POLICY-DEMO-v1. Judge equivalent final wording by its meaning. Do not make word-for-word matching the acceptance condition.

This set still lacks exact window boundaries, opened items, multilingual requests and other policies or products. Add those when they matter to your task; do not imply these examples cover them. Use permission-appropriate, minimized real cases later to check whether your chosen categories resemble actual requests. Until then, report coverage by case type rather than as production accuracy.

3. Preserve the run, then assign separate labels

Before running your own agent, choose which cases to repeat and how many attempts to preserve for each configuration. Use the same declared plan for comparisons. Keep every attempt, including timeouts and failed retrievals, rather than rerunning only the failures until a passing answer appears. The examples here remain unexecuted.

• Label the route PASS when observed calls and arguments satisfy the case contract. Label it FAIL for evidenced wrong or missing calls. Use INCONCLUSIVE if tool telemetry is missing; silence in an incomplete log does not prove that no call occurred. R3 passes only when a complete event record confirms no speculative call.
• Label the answer PASS when its meaning satisfies the reference using the returned evidence, FAIL when it contradicts or invents facts, and INCONCLUSIVE when missing evidence prevents a defensible judgment. If the tool explicitly reports unavailable, an honest limitation can pass the response-handling check while the order's eligibility remains unresolved. Do not report that as an eligibility determination.
• Record the automatic grader's label alongside those judgments. A failed route cannot be rescued by a fluent answer, and an unsupported answer cannot be rescued by a correct route. Keep a disagreement open for adjudication rather than averaging it away.

For a hypothetical R2 attempt, suppose the complete log shows the exact expected order and policy lookups, but the final answer says: It arrived 12 days ago, so it qualifies for a return. The route is PASS and the answer is FAIL: the response omits a conclusion-changing exclusion. That pair directs the correction toward policy interpretation or answer synthesis, not order-ID extraction. It is an illustrative diagnosis, not a reported model result.

4. Check the grader against labeled answers

Have a human reviewer label candidate answers against the fixed fixtures and rubric before comparing the automatic labels. Give the grader the request, applicable evidence and explicit criteria, not just the candidate text. OpenAI's guidance calls for checking agreement against human labels and identifies position bias, verbosity bias and disagreement among human experts as limitations.[3] Neither the grader nor the reference label is infallible.

For DEMO-SUPPORT-v1, the answer rubric checks that the response addresses the current request, applies all relevant eligibility conditions, distinguishes missing facts from negative facts, and avoids an unsupported refund claim. Clarification and honest unavailable-data handling are valid outcomes. A longer answer receives no automatic credit. Route scoring still uses the event record, not the answer's description of what it looked up.

Preserve each disagreement with the case ID, candidate answer, automatic label, reference label and cited fixture field or policy clause. Inspect false acceptance and false rejection separately. If the policy itself is ambiguous or two reviewers cannot settle applicability, mark the disputed reference INCONCLUSIVE and obtain an authoritative resolution before using it as a calibration target. R7 is resolvable under this example's explicit contract; a less explicit real contract may not be.

After revising the grader, check it on separately labeled held-out answers that were not used to tune its rules. Include concise correct answers, persuasive exception mistakes, necessary clarifications and honest retrieval failures. When comparing pairs, vary response order to inspect position sensitivity and keep length from substituting for correctness. Success on the calibration answers alone may mean the grader has learned those examples. Choose any operational agreement requirement for your task and risk; this guide supplies no universal threshold.

5. Repair the first evidenced defect and keep the case

Structured trace labels can help locate workflow errors rather than treating the final answer as the only diagnostic object.[2] Use the recorded defect to choose the next correction. Do not infer an unobserved internal reasoning failure from an output alone.

• If R6 calls DEMO-106, inspect current-request parsing and argument extraction. Keep the two-order conversation as a regression case.
• If the route retrieves an obsolete or inapplicable policy, correct retrieval/version selection first. A polished answer cannot compensate for the wrong authority.
• If R2 retrieves the right facts but ignores final sale, inspect answer synthesis and the instruction to apply every relevant condition.
• If the supported answer is correct but the grader rejects it, revise the grader's evidence or rubric. Do not make the agent give an unsupported answer merely to satisfy the grader.
• If the reference cannot be justified, resolve policy applicability or labeling disagreement first. Grader tuning against an unresolved reference hides the problem.

Keep case IDs stable and advance the fixture version when inputs, policies or reference meanings change. Preserve the previous evidence and label rather than silently overwriting it. Add resolved failures to the regression set, then rerun affected cases when the model, prompt, tools, retrieval source or policy changes; rerun grader validation when the rubric or grader changes. OpenAI recommends evaluation on changes and growing the case set over time.[3]

The resulting comparison should name the versions tested, included case types, repetition plan, case-level route and answer labels, grader disagreements and unresolved gaps. Report planned but unexecuted work as such. An overall average can hide a final-sale failure among easy cases; a small set of passes does not establish production reliability or rare-failure safety.

Platform and evidence limits

The official pages inspected on 10 October 2026 describe SDK-based application traces and existing Agent Builder workflows during a transition window.[1][2] They do not establish feature access for every account. This procedure needs observable application/tool events and fixed case files; it does not depend on a particular dashboard or promise continuing availability of a UI.

OpenAI's lifecycle notice dates the Evals platform deprecation announcement to 3 June 2026. Existing evals are scheduled to become read-only on 31 October, and the Evals dashboard and API are scheduled to shut down on 30 November 2026. Graders documented for eval workflows are included in that transition. Agent Builder has a separate notice and a scheduled 30 November shutdown.[4] These are announced schedules, not shutdowns already observed. Residual Evals API instructions in the guides should not be treated as a durable new-platform recommendation; the notices do not say that all SDK tracing is deprecated.

The four sources are one provider's methodological guidance and lifecycle notices. Their whole-page publication and update dates were not established; the dated deprecation entries are narrower evidence. The case contract, labels and maintenance procedure here are editorial proposals, not OpenAI requirements or independently measured findings. You can reuse the process, but must establish the authoritative policies, representative cases and acceptance needs for your own support task.