When a workflow fails, know what happened before trying again.

A timeout does not tell you whether the job happened

Imagine a workflow that creates a CRM task, then sends an internal notification. The CRM saves the task, but the connection drops before the response arrives. If the automation starts over, it may create a second task. If it declares success, the notification may never arrive. This is a hypothetical example of an uncertain outcome, not a reported client incident.

The useful question is: what evidence can establish the destination state? A failed response and a failed business action are different things. Ask for a stable operation identifier, the destination record reference when available, and a way to reconcile an uncertain result before repeating a consequential write.

Choose a response for each kind of failure

Ask what “safe to retry” actually means

A retry should represent the same intended operation. Generating a fresh identifier for each attempt defeats duplicate protection. Reusing one identifier for two genuinely different requests can suppress legitimate work. Define the identity at the business-operation level, scoped to the correct account, and test simultaneous repeats as well as sequential ones.

Stripe provides a concrete API example: its idempotency mechanism can return the saved result for repeated requests with the same key. Its documentation also describes parameter checks and key retention limits. Those details are specific to that API; an integration cannot assume another destination has the same guarantees. Ask the builder to document the actual endpoint behavior, including what happens after its protection window expires.

A local “already processed” flag alone is not proof. There can be a gap between a destination committing a change and the workflow recording success. Where the destination cannot safely deduplicate or reconcile an action, define a manual review path for uncertain outcomes rather than promising exactly-once execution.

Bound retries, then make the exception visible

The retry policy needs an owner and an endpoint. Decide which errors qualify, how long to wait, how many attempts or how much elapsed time is acceptable, and what happens when that budget is exhausted. Avoid treating invalid input, missing permission, and temporary unavailability as the same problem.

AWS Step Functions documents retry controls, increasing delays, jitter, and catch paths. That is a useful reference for the questions to ask, not a requirement to use AWS. The business requirement is simpler: repeated failures must stop consuming effort and become visible to someone who can act.

A useful exception record says which job is affected, what completed, what remains uncertain, and which action the owner should take next. Keep the relevant identifiers and evidence within the permitted audience. A raw stack trace or an alert with no responsible person is not an operating process.

Recover the business process, not just the software run

If a record was created but a notification failed, recovery may mean retrying only the notification. If a wrong record was created, the correction may need a separate authorized action with its own audit history. A code rollback does not reverse an email, a customer commitment, or an external write.

Document which actions can be reversed, which need a compensating correction, and which require contacting a person. Before resuming an old job, recheck record versions, permissions, and approval validity. Recovery must not quietly act on information or authority that is no longer current.

Manual takeover also needs coordination. A staff member should be able to stop automated attempts for the affected job, make a documented correction, and mark the outcome. Test that a delayed retry cannot undo the correction or repeat the completed action.

Make the failure demonstration part of acceptance

Use approved test records. Ask the builder to show a repeated trigger, a saved write with a lost response, and a failure between two steps. Inspect the destination records alongside the workflow history. A green run status is useful only if the business outcome matches it.

Also test an exhausted retry budget, a permission change, and a manual correction followed by a delayed event. Record what the operator sees and what they must do. These are proposed acceptance scenarios, not claims about failure rates or guaranteed recovery times.

Copy this recovery brief

# AI workflow recovery brief
Workflow / business owner:
Trigger and stable operation identifier:
Permitted actions and required approvals:
Definition of business completion:

## For each external action
Destination and account scope:
Evidence that the action committed:
Duplicate protection and its documented limits:
Retryable conditions / maximum elapsed time:
Unknown-outcome reconciliation method:
Next step if reconciliation is inconclusive:

## Partial completion
Steps that can safely resume:
Actions requiring a compensating correction:
Irreversible actions and customer response:
Approval expiry / changed-record check:
Manual takeover and automation stop procedure:
Exception owner / escalation deadline:

## Demonstrate before rollout
Repeated trigger, including concurrent delivery:
Destination saves action but response is lost:
First step succeeds and second step fails:
Permission revoked / invalid input:
Recovery occurs after data or approval changes:
Manual correction followed by delayed retry:
Destination records and audit evidence reviewed:
Acceptance decision and unresolved limits: