School

White-box improvement of Arkher. An engineer agent reads Arkher's code and the eval logs, diagnoses a failure, edits the code, and runs the same benchmark before and after. Each change is recorded here with its evidence and its measured effect, and waits for your verdict.

Latest pass rate
88%
benchmark #3
No answer
0
runs in the latest benchmark that got an HTTP error
Improvements
1
0 kept
Awaiting verdict
1
measured, not yet kept or reverted

Every benchmark, oldest first

passedanswer failed a checkno answer (HTTP error)
0%50%100%#1 smoke: 1 passed, 0 answer failed, 0 no answer, of 1100%#1smoke#2 before boot retry: 12 passed, 2 answer failed, 2 no answer, of 1675%#2before boo#3 after boot retry: 14 passed, 2 answer failed, 0 no answer, of 1688%#3after boot▲ #1

▲ #n under a column: improvement n was measured by that benchmark.

Each scenario in each benchmark

Scenario#1#2#3
Arkher: "yes" accepts the offer Arkher just made
Arkher: A private fact, answered on Cohere
Arkher: A productive back-and-forth: the follow-up builds on the first answer
Arkher: Draft the investor email itself, not a prompt
Arkher: Moat for a VC, against three named competitors
Arkher: Planned hires, then re-ordered to the user's priorities
Arkher: Pricing and unit economics from the newest deck, no questions back
Arkher: Projections from the newest deck, not the June plan
Arkher: Revenue trend: the annualized growth as well as the summer dip
Arkher: Signed or not: pending and verbal awards are not signed
Arkher: Two companies with one name: keep Kestrel Robotics out
Arkher: Write it as persuasive prose when asked
◆Arkher: Pricing and unit economics (reworded)
◆Arkher: Revenue trend (reworded)
◆Arkher: Signed or not (reworded)
◆Arkher: Two companies with one name (reworded)

◆ held-out scenario (never learned from)

Improvements

ImprovementStatusBeforeAfterChangeFeedbackUpdated
#1 · Retry chat calls the gateway refused before the function started
infra · arkher · by claude-code
measured75%88%▲ +13 pts0Oct 29, 06:15 PM