agent-replay.frame by frame

A failing run is a sequence, not a moment.

Load a recorded agent run and step through it. agent-replay finds the frame that first contradicted what it was meant to do, walks backwards through the facts it consumed to show the path into it, and reports the smallest slice of the run that reproduces the failure. Everything after that frame is consequence.

A nine step run that posted an invoice with the wrong total. The agent read a second invoice along the way, which is the step everyone blames first.

One bar per step, in order. Bar height is that step's token cost, and the shaded span is the replay window.

Diverged herePath start or last inputOn the pathBystander

step 1 of 9900 tokens620 msarrow keys step through the run

CriticalA step contradicts what it was meant to do

The run first diverged at step 7, Summed the line items. The dependency path into it runs through 4 steps and starts at step 2, Opened the invoice. The last input it read was step 6, Chose the tax rate for the vendor. Replaying steps 2 to 7 reproduces it, which is 6 steps of the 9 steps recorded. 1 step inside that range is a bystander the failure does not need.

Steps recorded9
Replay window6
Bystanders1
Repeated calls0
Tokens to the failure13,500

The run produced: posted invoice 4417 with a total 1,740.00 too high

step 1ModelOutside the window

Read the request

The model produced something

This step declares no expectation, so it cannot be judged on its own.

Produces
goal
Uses
nothing
Share of the run6%

Path into the failure

4 steps

Set aside as context rather than followed: goal. A fact read by most of the run says nothing about which step went wrong.

Repeated calls

none

No tool was called twice with the same arguments, so the run did not loop.

How this reading is produced

Deterministic: the same trace always gives the same reading. It reports what the trace declares, and says so when the trace declares nothing.

  1. Divergence is the first step that contradicts what it was meant to do: an explicit failure, then an observation that disagrees with its own expectation, then the step the trace blames. A trace that declares none has no divergence.
  2. The path is walked backwards from the divergence through the facts it consumed. Each fact is attributed to the nearest earlier step that introduced it, so a later restatement does not hide the original.
  3. Context is set aside. Anything the first step produces is the request rather than an input, and any fact read by more than a third of the run, and by at least three steps, is context. Following those would pull the whole run into the path and make the window useless.
  4. The replay window is the smallest contiguous slice that contains the path and the divergence. Steps inside it that the failure does not depend on are bystanders, and they are the first thing to delete when filing a bug.
Replay your own trace
JSON with a steps array. Nothing leaves the browser.

Declare expected and observed on the step that went wrong, and produces and uses on every step. The path can only be followed through facts you name.