Benchmark #3 · after boot retry

done

16 scenarios × 1 · started Oct 29, 06:09 PM

Pass rate

88%

14 of 16 runs

Train

92%

11 of 12

Held-out

75%

3 of 4

Time per run

59s

average, all turns

Tokens per run

510

estimated from the text

By scenario

Scenario	Pass rate	Time	Tokens
"yes" accepts the offer Arkher just made train · 1/1	100%	42s	406
A private fact, answered on Cohere train · 1/1	100%	25s	18
A productive back-and-forth: the follow-up builds on the first answer train · 0/1	0%	68s	389
Draft the investor email itself, not a prompt train · 1/1	100%	44s	767
Moat for a VC, against three named competitors train · 1/1	100%	76s	1,151
Planned hires, then re-ordered to the user's priorities train · 1/1	100%	115s	2,245
Pricing and unit economics from the newest deck, no questions back train · 1/1	100%	74s	688
Projections from the newest deck, not the June plan train · 1/1	100%	43s	184
Revenue trend: the annualized growth as well as the summer dip train · 1/1	100%	42s	350
Signed or not: pending and verbal awards are not signed train · 1/1	100%	39s	94
Two companies with one name: keep Kestrel Robotics out train · 1/1	100%	71s	38
Write it as persuasive prose when asked train · 1/1	100%	56s	474
Pricing and unit economics (reworded) held-out · 1/1	100%	63s	613
Revenue trend (reworded) held-out · 1/1	100%	78s	415
Signed or not (reworded) held-out · 1/1	100%	59s	197
Two companies with one name (reworded) held-out · 0/1	0%	48s	125

Each run is also an attempt under the agent benchmark:after boot retry, so it shows on the dashboard and in Attempts. Checks are patterns, not judgement: open a failed attempt to read Arkher's answer.