Computer-use agent recovery playbook
Verify a browser postcondition before replaying an AI agent action
A lost tab, timeout, or missing screenshot does not prove the click failed. Inspect the state the action was supposed to create before the agent submits, purchases, publishes, books, or changes anything again.
INTENTPRECONDITIONACTIONUNKNOWN OUTCOMEFRESH SESSIONPOSTCONDITIONINTENTPRECONDITIONACTIONUNKNOWN OUTCOMEFRESH SESSIONPOSTCONDITION
Verify the world, not the last screenshot
A browser action is safe to replay only when the intended external effect is confirmed absent or the target provides an equivalent idempotency guarantee.
Browser automation often loses confirmation at the worst moment: after a form submission, publish click, booking confirmation, or checkout action. The page may navigate slowly, the session may close, a worker may restart, or a screenshot may never arrive. None of those failures establishes whether the target committed the action.
A postcondition describes the observable state that should exist after success. It may be a new record with a stable identifier, a changed status, a confirmation page, a provider event, a revision visible in a list, or the absence of an item from a pending queue. A robust agent records this expectation before it acts.
Recovery then opens a clean observation path and tests that expectation independently. If the postcondition exists, the action is complete and should not replay. If authoritative evidence proves it absent, the agent may retry under policy. If the evidence is inconclusive, the outcome remains unknown and high-impact replay should stop for review.
This method is different from automatic waiting before a click. Actionability checks help ensure that an element is visible and enabled at execution time. Postcondition verification proves what happened after the action crossed into the external system.
Postcondition taxonomy
Resource existence
A created booking, order, message, deployment, or account record exists with a stable identity tied to the original intent.
State transition
An existing resource moved from draft to published, pending to confirmed, enabled to disabled, or another expected status.
Observable consequence
A confirmation event, receipt, history entry, revision, or provider callback proves the effect even if the original page is gone.
Confirmed absence
An authoritative query shows no matching operation and no asynchronous work remains pending. “Not visible yet” is not always absence.
Freeze the replay
Stop automatic retries, competing workers, and any queued copy of the browser action. Preserve the run, attempt, workflow version, target URL, action input, timestamps, screenshots, network evidence, and trace context.
Restate the intended effect
Describe the result independent of the click: “revision a91c is published once” rather than “click Publish.” This gives recovery a stable target even if the interface has changed.
Assign or recover the operation identity used to correlate the effect, such as an order reference, draft ID, booking details, message fingerprint, or intended revision.
Load a fresh observation session
Open the target in a new or known-clean browser context. Restore only the credentials and tenant scope required to inspect state. Do not automatically replay navigation that itself creates side effects, and do not reuse a page whose stale client state could hide the committed result.
Query the strongest source
Prefer stable resource pages, history views, provider APIs, downloadable receipts, or event records over transient toast text. Search using the operation identity and a bounded time window.
- Match all material fields, not only a similar title.
- Check for pending asynchronous work.
- Record the observed URL, timestamp, and evidence source.
Classify the outcome
Mark the effect confirmed complete, confirmed absent, confirmed failed, or unknown. Complete requires evidence that the intended result exists. Absent requires authoritative evidence that no result or pending operation exists. A loading state, search miss, or changed selector may remain unknown.
Choose the recovery action
If complete, record the result and advance without clicking again. If absent, refresh any expired inputs and retry under the original intent identity. If failed, repair the cause or choose an approved alternative. If unknown, continue reconciliation or route to review.
High-impact actions should require stronger evidence and may need human authorization even after apparent absence.
Verify after any retry
After a permitted replay, run the postcondition check again and attach its evidence to the recovery receipt. Record whether the action was skipped, retried, or reviewed, and what final user-visible state was confirmed.
Four evidence classes
Strong
A provider operation ID, unique resource record, signed receipt, immutable event, or authoritative API response matches the intent.
Supporting
A history row, confirmation page, status badge, or revision list supports the conclusion but may require correlation with other fields.
Weak
A toast, visual disappearance, button state, URL fragment, or screenshot can be stale, transient, or shared by multiple outcomes.
Contradictory
Sources disagree, the same action appears twice, or the page and provider record show different states. Stop and reconcile rather than averaging evidence.
Worked examples
Publish button
The session closes after a click. Open the publication list, find the intended revision, verify its status and timestamp, and inspect the live URL. A missing success toast is irrelevant if the revision is already live.
intent: publish revision a91c once
postcondition: revision a91c status=live
action: skip replayAppointment booking
Search upcoming appointments by user, service, time range, and booking reference. Confirm whether a matching booking or pending hold exists before submitting the form again.
intent: one booking near 3:00 PM
postcondition: confirmed booking ID exists
action: complete or reviewCheckout
Inspect order history, payment authorization, email receipt, and cart state. A nonempty cart does not prove checkout failed; some sites retain carts after purchase.
intent: one order for cart fingerprint
postcondition: order and charge correlate
action: never blind retryBuild postconditions before actions
Use stable locators for evidence
Prefer accessible roles, labels, resource identifiers, and semantic state over layout position or brittle CSS chains. The postcondition should survive ordinary interface rearrangement.
Record a normalized intent fingerprint
Capture material inputs before acting so recovery can distinguish the intended result from a similar resource created by another user or attempt.
Separate observation tools
Create read-only inspection functions when possible. A generic browser replay that navigates through side-effecting steps can contaminate the evidence it is meant to collect.
Define freshness and authority
Every observation should record when it was made, how long it remains useful, and whether its source can authoritatively establish presence or absence.
Test delayed consistency
Some systems commit immediately but update lists later. Verification should use bounded polling and provider events rather than treating the first search miss as absence.
Applied personal-agent workflows
Text-message assistant
When a browser-backed task becomes unknown, the agent can tell the user verification is underway, ask them not to repeat it, and send the final evidence-backed receipt.
Computer-use cache
Cache observations separately from action results, attach freshness, and invalidate state after navigation or external change. Recovery can reuse evidence only within its declared scope.
Website-building agent
Verify repository commit, deployment record, and live revision before triggering another publish. Resume from health validation if the intended release already exists.
Pre-replay checklist
Automatic retries and competing workers are frozen.
The original user intent is still valid.
The intended effect is stated independently of the click.
A stable operation or intent fingerprint is available.
The observation session is fresh and correctly scoped.
The strongest available evidence source was queried.
Material fields and time windows match the intent.
Pending asynchronous work was considered.
Presence or absence is authoritative, not assumed.
Contradictory evidence is resolved or escalated.
Expired inputs and approvals are refreshed.
The replay reuses the original logical effect identity.
High-impact actions have required authorization.
A final postcondition check and receipt are planned.
What a postcondition cannot prove
A visible result may not establish every part of the intended effect.
A confirmation page can exist while payment later reverses. A published status may appear while the public edge still serves an older revision. A booking can exist with the wrong customer, timezone, or service. Verification must match the full intent, not a convenient success-shaped element.
Absence is especially difficult. Search results may lag, permissions may hide records, filters may be wrong, and the site may create a pending object under another view. Define which source is authoritative and how long to wait before absence can be trusted.
When no reliable postcondition exists, design the product to stop. Capture screenshots and network evidence, notify the user without implying failure, and route the action to a person. A manual check is cheaper than duplicating an irreversible purchase or submission.
Frequently asked questions
Is a success toast a valid postcondition?
It is supporting evidence, not usually the strongest proof. Toasts can disappear, be stale, or appear before asynchronous completion. Prefer a durable resource record, status page, provider event, or authoritative API.
How long should the agent wait before deciding an action is absent?
Use a bounded window based on the target’s consistency behavior and risk. Poll authoritative sources and pending queues. If absence cannot be established within the window, keep the outcome unknown rather than guessing.
Can screenshots prove completion?
Screenshots are useful evidence of what the browser displayed at a time, but they may not prove backend commitment or later state. Correlate them with stable identifiers and durable records.
What if the page changed after the original attempt?
Re-identify the postcondition semantically using the intended resource and stable fields. If the new interface prevents authoritative verification, stop and review rather than replay through unfamiliar controls.
Should a model judge whether the postcondition exists?
A model can interpret complex visual or textual evidence, but the system should preserve the source and confidence, apply deterministic matching where possible, and require review for high-impact ambiguity.
How does caching fit?
Cache read observations with source, scope, timestamp, and freshness. Invalidate after actions or known external changes. Never treat an old cached absence as proof that a recent action failed.
Primary references
W3C WebDriverBrowser automation commands, elements, navigation, and session behavior.
Playwright ActionabilityOfficial documentation on pre-action checks and assertions.
RFC 9110: HTTP SemanticsIdempotency and retry behavior at web boundaries.
OpenTelemetry Observability PrimerEvidence correlation across browser and backend services.
Inspect the result before repeating the action.
Super connects personal-agent workflows to messaging, computer use, and website operations where careful browser recovery prevents duplicate real-world effects.
Explore Super