Benchmark #2 · before boot retry

done

16 scenarios × 1 · started Oct 29, 05:46 PM

Pass rate

75%

12 of 16 runs

Train

83%

10 of 12

Held-out

50%

2 of 4

Time per run

57s

average, all turns

Tokens per run

433

estimated from the text

By scenario

Scenario	Pass rate	Time	Tokens
"yes" accepts the offer Arkher just made train · 1/1	100%	49s	370
A private fact, answered on Cohere train · 1/1	100%	51s	11
A productive back-and-forth: the follow-up builds on the first answer train · 1/1	100%	88s	300
Draft the investor email itself, not a prompt train · 1/1	100%	47s	554
Moat for a VC, against three named competitors train · 0/1	0%	99s	535
Planned hires, then re-ordered to the user's priorities train · 1/1	100%	123s	2,374
Pricing and unit economics from the newest deck, no questions back train · 0/1	0%	48s	615
Projections from the newest deck, not the June plan train · 1/1	100%	51s	190
Revenue trend: the annualized growth as well as the summer dip train · 1/1	100%	47s	406
Signed or not: pending and verbal awards are not signed train · 1/1	100%	37s	118
Two companies with one name: keep Kestrel Robotics out train · 1/1	100%	66s	50
Write it as persuasive prose when asked train · 1/1	100%	66s	592
Pricing and unit economics (reworded) held-out · 1/1	100%	63s	627
Revenue trend (reworded) held-out · 0/1	0%	11s	64
Signed or not (reworded) held-out · 0/1	0%	11s	39
Two companies with one name (reworded) held-out · 1/1	100%	61s	83

Each run is also an attempt under the agent benchmark:before boot retry, so it shows on the dashboard and in Attempts. Checks are patterns, not judgement: open a failed attempt to read Arkher's answer.