Define what “works” means before your AI pilot starts.

Updated October 14, 2026: added a practical test-case matrix and a repeatable case record covering routine work, missing evidence, denied actions, retries, and human review.

Start with the business decision

An AI pilot should answer a question you can act on: can this workflow move into a limited rollout, does it need another revision, or should you stop? “The assistant sounds helpful” does not settle that decision. A completed job, a checked output, and a workable review burden do.

For a hypothetical quote-intake pilot, the job might be to turn a customer’s email into a draft request with product, quantity, required date, and missing-information flags. Sales still approves the quote. The pilot does not get permission to choose prices or send commitments just because it can write a convincing response.

Separate release blockers from improvement targets

Some requirements should block release when a test violates them: exposing another customer’s information, sending without approval, or silently creating duplicate records. Other measures, such as drafting speed or editing effort, help you judge whether the workflow is worth adopting. Do not average a failed access-control check into an otherwise attractive quality score.

Choose the evidence before choosing the score

For each requirement, name what the reviewer will inspect. A saved record proves something different from a screenshot of a chat answer. If the task is to create a draft, inspect the draft. If it is to update a permitted field, compare the resulting record and audit entry with the approved request.

Anthropic’s agent-evaluation guidance distinguishes an agent’s recorded interaction from the final state it leaves behind. That distinction matters to a buyer: saying an action happened is not evidence that it happened. OpenAI’s evaluation guidance similarly starts with a defined objective, a dataset, and metrics tied to the task. Both sources are linked below.

Agree on the sample and the denominator

Have the people who do the work select permitted, sanitized examples. Cover routine requests, incomplete inputs, conflicting information, and cases the system should refuse or escalate. Keep a separate set for the acceptance review instead of repeatedly tuning against every example you will later use to judge the system.

Record the number of work items attempted, completed, corrected, rejected, and escalated. Say whether a reported success rate counts a human rescue as success. A percentage without that definition makes proposals hard to compare. A small sample with no observed failures is useful evidence for the cases tested; it is not proof that the workflow can never fail.

Agree the sample size and repeat-run plan with the technical owner based on the task’s variability and consequences. There is no universal “enough examples” number for every business. Repeat important scenarios when outputs vary, keep the environment consistent, and retain failures in the review record.

Turn each requirement into a runnable test case

A useful case states the starting conditions, input, allowed action, expected result, and evidence needed to decide. “Test missing information” is a topic. “Submit this email with no quantity, then confirm the draft leaves quantity empty and asks for it” is a check someone else can repeat. The examples below extend the hypothetical draft-only quote workflow; they are planning examples, not client test results.

Treat a correct refusal or request for clarification as a passing boundary check when that is the expected outcome. Track it separately from completed business work. An assistant that declines every request may protect a boundary while delivering no useful service. Review routine completion, appropriate escalation, and unnecessary escalation separately.

Make the evidence repeatable

Run these cases in an isolated test environment with synthetic or permitted sanitized inputs. Route outbound actions to test destinations. Record the workflow revision, model identifier, approved source version, permissions, and starting records. Reset test state between independent trials so a record left by one run cannot make the next run look successful. Test retry behavior within a single case, where preserving the earlier state is intentional.

Use exact checks for fields, record counts, permissions, and sends. For wording that allows several good answers, write a short rubric covering required facts, unsupported claims, and next steps. Have a knowledgeable reviewer resolve ambiguous judgments. A polished answer should never override evidence that the wrong record changed.

# AI workflow test-case record
Case ID / requirement: [identifier and acceptance rule]
Case group: [routine / clarification / denied / failure / correction]
Owner / reviewer: [roles]
Workflow revision / model identifier: [versions]
Source snapshot / permission setup: [identifiers]
Starting state: [records, queue, test destinations]
Input fixture: [sanitized file or exact message]
Steps: [actions to reproduce, including any simulated failure]
Expected output: [required facts, missing fields, or refusal]
Allowed state change: [records/actions that may change]
Prohibited side effects: [sends, duplicate jobs, unauthorized access]
Evidence to inspect: [saved output, before/after state, audit events]
Reset procedure: [restore starting state between independent runs]
Trial plan: [count and reason, agreed before execution]

## One record per trial
Trial ID / time: [identifier]
Actual result / evidence links: [observations]
Human intervention / review time: [what changed and duration]
Outcome: [pass / fail / blocked; explain]
Failure owner / fix reference: [if applicable]
Retest scope: [failed case plus affected previously passing cases]

Copy this pilot acceptance brief

Fill this in with your operations lead and the team building the system. Resolve the blank fields before a final acceptance review. This is an original DBAI planning template, not a completed evaluation or a contract.

# AI pilot acceptance brief

Workflow: [one bounded job]
Business owner: [name / role]
Technical owner: [name / role]
Acceptance decision date: [date]

## Scope
Trigger and permitted inputs: [sources and access boundaries]
Required output: [artifact or record, including required fields]
Allowed actions: [explicit list]
Human approval before: [actions or commitments]
Out of scope: [explicit exclusions]

## Evaluation
Dataset version: [identifier, permitted source, sample counts]
Case groups: [routine / incomplete / conflict / denied / failure]
Expected result for each case: [reference or checkable rule]
Separate acceptance set: [identifier and reviewer]
Repeat-run plan: [which cases, how many runs, why]

## Decision rules
Release blockers: [behaviors that must not occur in reviewed tests]
Quality target: [metric, threshold, denominator, exclusions]
Review-effort limit: [measure, threshold, comparison baseline]
Latency and operating-cost limits: [measure and threshold]
Evidence retained: [outputs, resulting state, logs, reviewer notes]

## Rollout decision
Decision: [limited rollout / revise and retest / stop]
Unresolved failures and owner: [list]
Rollout scope and permissions: [users, volume, allowed actions]
Rollback trigger and manual fallback: [condition and procedure]
Signed off by: [business and technical owners, date]

Check the workflow boundary, not just the AI response

DBAI’s DTF Tools case study describes artwork tools, pricing calculators, and a price-sheet builder in one portal. That separation gives a buyer a useful way to think about acceptance: image quality, calculation correctness, and exported output need different checks. A good-looking image cannot tell you whether a pricing calculation is correct.

End the pilot with a decision

Approve a limited rollout when the agreed checks pass and somebody owns operation, exceptions, and recovery. Keep the rollout within the authority you tested. A successful draft-only pilot does not establish that unattended sending is ready.

Revise when failures are understood and the next change is bounded. Document the failed requirement and rerun the affected checks, including checks that previously passed. Stop when the workflow cannot justify its review burden, necessary data is unavailable, or the required controls cannot be delivered within scope. Stopping can be the right result of a pilot.

When you bring DBAI a workflow, bring a few permitted examples and the result your team needs to trust. We can use those to define the acceptance brief before the build gets bigger.

Technical references