Define what “works” means before your AI pilot starts.

Start with the business decision

An AI pilot should answer a question you can act on: can this workflow move into a limited rollout, does it need another revision, or should you stop? “The assistant sounds helpful” does not settle that decision. A completed job, a checked output, and a workable review burden do.

For a hypothetical quote-intake pilot, the job might be to turn a customer’s email into a draft request with product, quantity, required date, and missing-information flags. Sales still approves the quote. The pilot does not get permission to choose prices or send commitments just because it can write a convincing response.

Separate release blockers from improvement targets

Some requirements should block release when a test violates them: exposing another customer’s information, sending without approval, or silently creating duplicate records. Other measures, such as drafting speed or editing effort, help you judge whether the workflow is worth adopting. Do not average a failed access-control check into an otherwise attractive quality score.

Choose the evidence before choosing the score

For each requirement, name what the reviewer will inspect. A saved record proves something different from a screenshot of a chat answer. If the task is to create a draft, inspect the draft. If it is to update a permitted field, compare the resulting record and audit entry with the approved request.

Anthropic’s agent-evaluation guidance distinguishes an agent’s recorded interaction from the final state it leaves behind. That distinction matters to a buyer: saying an action happened is not evidence that it happened. OpenAI’s evaluation guidance similarly starts with a defined objective, a dataset, and metrics tied to the task. Both sources are linked below.

Agree on the sample and the denominator

Have the people who do the work select permitted, sanitized examples. Cover routine requests, incomplete inputs, conflicting information, and cases the system should refuse or escalate. Keep a separate set for the acceptance review instead of repeatedly tuning against every example you will later use to judge the system.

Record the number of work items attempted, completed, corrected, rejected, and escalated. Say whether a reported success rate counts a human rescue as success. A percentage without that definition makes proposals hard to compare. A small sample with no observed failures is useful evidence for the cases tested; it is not proof that the workflow can never fail.

Agree the sample size and repeat-run plan with the technical owner based on the task’s variability and consequences. There is no universal “enough examples” number for every business. Repeat important scenarios when outputs vary, keep the environment consistent, and retain failures in the review record.

Copy this pilot acceptance brief

Fill this in with your operations lead and the team building the system. Resolve the blank fields before a final acceptance review. This is an original DBAI planning template, not a completed evaluation or a contract.

# AI pilot acceptance brief

Workflow: [one bounded job]
Business owner: [name / role]
Technical owner: [name / role]
Acceptance decision date: [date]

## Scope
Trigger and permitted inputs: [sources and access boundaries]
Required output: [artifact or record, including required fields]
Allowed actions: [explicit list]
Human approval before: [actions or commitments]
Out of scope: [explicit exclusions]

## Evaluation
Dataset version: [identifier, permitted source, sample counts]
Case groups: [routine / incomplete / conflict / denied / failure]
Expected result for each case: [reference or checkable rule]
Separate acceptance set: [identifier and reviewer]
Repeat-run plan: [which cases, how many runs, why]

## Decision rules
Release blockers: [behaviors that must not occur in reviewed tests]
Quality target: [metric, threshold, denominator, exclusions]
Review-effort limit: [measure, threshold, comparison baseline]
Latency and operating-cost limits: [measure and threshold]
Evidence retained: [outputs, resulting state, logs, reviewer notes]

## Rollout decision
Decision: [limited rollout / revise and retest / stop]
Unresolved failures and owner: [list]
Rollout scope and permissions: [users, volume, allowed actions]
Rollback trigger and manual fallback: [condition and procedure]
Signed off by: [business and technical owners, date]

Check the workflow boundary, not just the AI response

DBAI’s DTF Tools case study describes artwork tools, pricing calculators, and a price-sheet builder in one portal. That separation gives a buyer a useful way to think about acceptance: image quality, calculation correctness, and exported output need different checks. A good-looking image cannot tell you whether a pricing calculation is correct.

End the pilot with a decision

Approve a limited rollout when the agreed checks pass and somebody owns operation, exceptions, and recovery. Keep the rollout within the authority you tested. A successful draft-only pilot does not establish that unattended sending is ready.

Revise when failures are understood and the next change is bounded. Document the failed requirement and rerun the affected checks, including checks that previously passed. Stop when the workflow cannot justify its review burden, necessary data is unavailable, or the required controls cannot be delivered within scope. Stopping can be the right result of a pilot.

When you bring DBAI a workflow, bring a few permitted examples and the result your team needs to trust. We can use those to define the acceptance brief before the build gets bigger.

Technical references