It says “done.” The state disagrees.
The agent clicked Save. The right record, role, or workspace never changed. You need a checkable outcome.
BROWSER-AGENT REGRESSION TESTING
Turn failed browser-agent runs into regression tests.
Capture the browser path. Check the agreed outcome. Keep the failure as a case to rerun after your next model or application change.
A one-task pilot for teams building browser agents.
Start in an approved sandbox or test account.
TASK / ROLE-CHANGE-001
Change a member’s role.
Editor Viewer
Agent report Complete
“Role updated to Viewer.”
Checked outcome Fail
Still Editor after reload.
00:00
Start as Editor
00:06
Select Viewer → Save
00:12
Reload → still Editor First divergence
A failure worth keeping.Known start. Evidence. Instructions to rerun.
Illustrative pilot artifactExplore the case
BUILT AROUND YOUR TASK
Agent engineering
Evaluation & reliability
Tool-use & post-training
01 / THE GAP AFTER THE RUN
A finished run isn’t always a finished task.
When the final state is wrong, the useful question is what you can verify, reproduce, and test again.
A fresh run loses the failure.
A different session, permission, or starting state can hide the problem. The conditions matter as much as the clicks.
The lesson never becomes a test.
The trace gets reviewed. The incident gets closed. The same failure returns after the next model or application change.
02 / FROM ONE FAILURE TO A REUSABLE CASE
Keep the path. Check the result.
Explore a role change that looks successful until the page reloads. Each stage adds evidence your team can act on.
ROLE-CHANGE-001Update a test member’s role
Synthetic exampleworkspace.example / members
SANDBOX WORKSPACEWorkspace members
MemberRole
Test memberSynthetic account
EditorReload complete. Role remains Editor.
Observed browser state00:12
03 / OUTCOME CHECK FAILED
Find the first observable mismatch.
After a reload, the member still has the Editor role. The checked page state disagrees with the agent’s report.
Expected → ObservedViewer → Editor
An illustration of the proposed pilot deliverable, using synthetic data.
Download sample JSONPass, fail, or incomplete. When the agreed outcome cannot be checked, the verdict stays incomplete. A success message alone is not proof.
03 / FITS AROUND YOUR EXISTING WORKFLOW
Your agent. Your task. A clearer record.
Build on the runner and evidence you already have. Scope one outcome before you expand the evaluation set.
Your runner
Keeps running the agent.
The pilot adds a scoped capture and checking workflow around it.
Your trace
Shows what the agent did.
The artifact connects that path to starting conditions and a checked result.
Your next release
Needs a repeatable check.
The failure becomes a case to rerun against a new model, prompt, policy, or app.
The pilot preserves the first meaningful divergence for investigation. Automatic root-cause diagnosis and repair are outside its scope.
04 / START WITH ONE FAILED TASK
Bring the failure you keep rebuilding.
A focused pilot for teams that struggle to reproduce or grade a recurring browser-agent failure.
- The wrong record, workspace, or role gets changed.
- A save appears to work but does not persist.
- Navigation, timing, or permissions change the result.
- The agent reports success without enough evidence.
A SCOPED FIRST ENGAGEMENT
One task. Agreed conditions.
- 01Define the task together.
Agree on the starting state, allowed actions, success condition, and model and environment versions.
- 02Run in an approved environment.
Use a sandbox, synthetic workflow, or test account. Agree on access, redaction, retention, and permitted use first.
- 03Return a case your team can reuse.
Ordered actions, checked outcome, divergence evidence, and reproduction instructions with known reset requirements.
Pilot scope and acceptance criteria are agreed before work begins. Production customer data is not required.
BEFORE YOU JOIN
A few things to make clear.
What happens when I join the waitlist?
We’ll email you about pilot availability and next steps. If there’s a fit, we’ll scope one failing browser task, its starting conditions, and an observable success condition with your team before any run. Joining the waitlist does not reserve or guarantee a pilot place.
Does Samelogic run the agent?
The pilot is designed to work around your existing agent runner. Samelogic’s role is capture, evidence, outcome checking, and regression packaging. The runner and environment are agreed when we scope the task.
Does this automatically find the root cause?
No. The pilot identifies the first meaningful divergence and preserves evidence for your engineering or evaluation team to investigate. It does not promise automatic diagnosis or repair.
What if the outcome cannot be verified?
The verdict is incomplete. A click, a success message, or the agent’s own report is not enough to prove a task finished. We agree on observable conditions and the evidence your approved environment can provide before running the task.
Do you need production customer data?
No. Start with a synthetic workflow, test account, or approved sandbox. We scope access, retention, redaction, deployment requirements, and permitted use before any run.
What happens after the first task?
If the artifact proves useful for investigation or catches a regression, we can scope the next task family. Acceptance criteria and commercial scope are agreed before custom work begins.
AGENT RELIABILITY / PILOT WAITLIST
Make the next failure a test you keep.
Join the waitlist for the one-task pilot. We’re bringing teams on gradually, starting with a concrete browser task and an agreed way to check it.