Progress dashboard

Is the AI getting better? Pass rates over time, the lift lessons give, whether that learning carries over to held-out scenarios, and which scenarios and agents are improving.

LangSmith tracing off · set LANGSMITH_API_KEY in .devcontainer/.env to track every action

Scenario library + New scenario

Pass rate

65%

34 of 52 attempts

Last 10 attempts

70%

▼ 20 pts vs the 10 before

Lift from lessons

+45 pts

Train pass rate with lessons minus without

Held-out pass rate

50%

Generalization: never learned from

Learning curve · rolling 5-attempt pass rate

With lessons (43)Without (9)

Show data table

#	Attempt	Scenario	Lessons	Result	Rolling rate
1	#1	Fix the failing date-parsing test	without	fail	0% (n=1)
2	#2	Fix the failing date-parsing test	with	pass	100% (n=1)
3	#3	Fix the failing date-parsing test	with	pass	100% (n=2)
4	#4	Fix the failing date-parsing test	without	fail	0% (n=2)
5	#5	Fix the failing date-parsing test	with	pass	100% (n=3)
6	#6	Fix the failing date-parsing test	without	fail	0% (n=3)
7	#7	Fix the failing date-parsing test	without	pass	25% (n=4)
8	#8	Triage a disk-full alert on a web host	with	pass	100% (n=4)
9	#9	Triage a disk-full alert on a web host	without	fail	20% (n=5)
10	#10	Triage a disk-full alert on a web host	without	fail	20% (n=5)
11	#11	Triage a disk-full alert on a web host	with	pass	100% (n=5)
12	#12	Write a launch email for a B2B feature	without	pass	40% (n=5)
13	#13	Write a launch email for a B2B feature	without	fail	40% (n=5)
14	#16	Arkher: Signed or not: pending and verbal awards are not signed	with	fail	80% (n=5)
15	#17	Arkher: Moat for a VC, against three named competitors	with	pass	80% (n=5)
16	#18	Arkher: Pricing and unit economics from the newest deck, no questions back	with	fail	60% (n=5)
17	#19	Arkher: Draft the investor email itself, not a prompt	with	fail	40% (n=5)
18	#20	Arkher: Revenue trend: the annualized growth as well as the summer dip	with	pass	40% (n=5)
19	#21	Arkher: Planned hires, then re-ordered to the user's priorities	with	pass	60% (n=5)
20	#22	Arkher: "yes" accepts the offer Arkher just made	with	pass	60% (n=5)
21	#23	Arkher: Projections from the newest deck, not the June plan	with	fail	60% (n=5)
22	#24	Arkher: A private fact, answered on Cohere	with	pass	80% (n=5)
23	#25	Arkher: Signed or not: pending and verbal awards are not signed	with	pass	80% (n=5)
24	#26	Arkher: Two companies with one name: keep Kestrel Robotics out	with	pass	80% (n=5)
25	#27	Arkher: Write it as persuasive prose when asked	with	pass	80% (n=5)
26	#28	Arkher: A productive back-and-forth: the follow-up builds on the first answer	with	pass	100% (n=5)
27	#29	Arkher: Pricing and unit economics (reworded)	with	fail	80% (n=5)
28	#30	Arkher: Revenue trend (reworded)	with	fail	60% (n=5)
29	#31	Arkher: Signed or not (reworded)	with	pass	60% (n=5)
30	#32	Arkher: Two companies with one name (reworded)	with	fail	40% (n=5)
31	#33	Arkher: Moat for a VC, against three named competitors	with	pass	40% (n=5)
32	#34	Arkher: Pricing and unit economics from the newest deck, no questions back	with	fail	40% (n=5)
33	#35	Arkher: Draft the investor email itself, not a prompt	with	fail	40% (n=5)
34	#36	Arkher: Revenue trend: the annualized growth as well as the summer dip	with	pass	40% (n=5)
35	#37	Arkher: Planned hires, then re-ordered to the user's priorities	with	pass	60% (n=5)
36	#38	Arkher: "yes" accepts the offer Arkher just made	with	pass	60% (n=5)
37	#39	Arkher: Projections from the newest deck, not the June plan	with	pass	80% (n=5)
38	#40	Arkher: A private fact, answered on Cohere	with	pass	100% (n=5)
39	#41	Arkher: Signed or not: pending and verbal awards are not signed	with	pass	100% (n=5)
40	#42	Arkher: Two companies with one name: keep Kestrel Robotics out	with	pass	100% (n=5)
41	#43	Arkher: Write it as persuasive prose when asked	with	pass	100% (n=5)
42	#44	Arkher: A productive back-and-forth: the follow-up builds on the first answer	with	pass	100% (n=5)
43	#45	Arkher: Pricing and unit economics (reworded)	with	pass	100% (n=5)
44	#46	Arkher: Revenue trend (reworded)	with	pass	100% (n=5)
45	#47	Arkher: Signed or not (reworded)	with	pass	100% (n=5)
46	#48	Arkher: Two companies with one name (reworded)	with	fail	80% (n=5)
47	#49	Arkher: Pricing and unit economics from the newest deck, no questions back	with	pass	80% (n=5)
48	#50	Arkher: Draft the investor email itself, not a prompt	with	fail	60% (n=5)
49	#51	Arkher: Pricing and unit economics (reworded)	with	pass	60% (n=5)
50	#52	Arkher: Draft the investor email itself, not a prompt	with	pass	60% (n=5)
51	#53	Arkher: Two companies with one name (reworded)	with	fail	60% (n=5)
52	#54	Arkher: A private fact, answered on Cohere	without	pass	40% (n=5)

Do lessons help, and do they generalize?

Train scenarioslift +45 pts

With lessons

79% 26/33

Without

33% 3/9

Held-out scenariosnot enough data

With lessons

50% 5/10

Without

— 0/0

Lessons are only learned from train scenarios. A lift on held-out scenarios means the agent learned something general, not the answers.

What the agents have learned

All lessons →

active lessons in memory

proposed, awaiting review →

Do: For date bugs near year/week boundaries, check for local-time methods (getDay, getDate) where UTC variants are needed.+1/−0
Avoid: When a test fails, never edit the test to make it pass; find the cause in the implementation.+1/−0

Token usage

How tokens grow →

Tokens, all attempts

31,208

41 attempts with token data

Per attempt

761

average

Per passed attempt

728

average

Per failed attempt

834

average

Tokens per attempt

Input (sent)Output (answer)Measured, no split

Average per attempt, by scenario

Fix the failing date-parsing test

4.5k

Planned hires, then re-ordered to the user's priorities

2.3k

Moat for a VC, against three named competitors

1.3k

Draft the investor email itself, not a prompt

742

Pricing and unit economics from the newest deck, no questions back

721

Pricing and unit economics (reworded)

639

Write it as persuasive prose when asked

593

A productive back-and-forth: the follow-up builds on the first answer

550

Revenue trend: the annualized growth as well as the summer dip

509

"yes" accepts the offer Arkher just made

482

Revenue trend (reworded)

477

Projections from the newest deck, not the June plan

295

Two companies with one name: keep Kestrel Robotics out

239

Signed or not (reworded)

211

Signed or not: pending and verbal awards are not signed

179

Two companies with one name (reworded)

173

A private fact, answered on Cohere

Input tokens sent at each turn

Each turn resends the conversation so far, so input grows with every turn.

39 of 41 attempts are estimated: Arkher does not report its usage, so tokens are counted from the conversation text at about 4 characters per token. This leaves out Arkher's own instructions and the documents it retrieves, so the real usage is higher. Attempts from the built-in runner are measured.

Table view

Attempt	Scenario	Result	Input	Output	Total	Source
#1	Fix the failing date-parsing test	failed	—	—	5,200	measured
#2	Fix the failing date-parsing test	passed	—	—	3,900	measured
#16	Arkher: Signed or not: pending and verbal awards are not signed	failed	12	156	168	estimated
#17	Arkher: Moat for a VC, against three named competitors	passed	124	1,215	1,339	estimated
#18	Arkher: Pricing and unit economics from the newest deck, no questions back	failed	83	573	656	estimated
#19	Arkher: Draft the investor email itself, not a prompt	failed	73	681	754	estimated
#20	Arkher: Revenue trend: the annualized growth as well as the summer dip	passed	62	442	504	estimated
#21	Arkher: Planned hires, then re-ordered to the user's priorities	passed	1,044	1,303	2,347	estimated
#22	Arkher: "yes" accepts the offer Arkher just made	passed	128	342	470	estimated
#23	Arkher: Projections from the newest deck, not the June plan	failed	17	272	289	estimated
#24	Arkher: A private fact, answered on Cohere	passed	8	10	18	estimated
#25	Arkher: Signed or not: pending and verbal awards are not signed	passed	12	187	199	estimated
#26	Arkher: Two companies with one name: keep Kestrel Robotics out	passed	18	214	232	estimated
#27	Arkher: Write it as persuasive prose when asked	passed	38	588	626	estimated
#28	Arkher: A productive back-and-forth: the follow-up builds on the first answer	passed	224	327	551	estimated
#29	Arkher: Pricing and unit economics (reworded)	failed	58	518	576	estimated
#30	Arkher: Revenue trend (reworded)	failed	44	440	484	estimated
#31	Arkher: Signed or not (reworded)	passed	19	191	210	estimated
#32	Arkher: Two companies with one name (reworded)	failed	21	122	143	estimated
#33	Arkher: Moat for a VC, against three named competitors	passed	124	1,140	1,264	estimated
#34	Arkher: Pricing and unit economics from the newest deck, no questions back	failed	83	653	736	estimated
#35	Arkher: Draft the investor email itself, not a prompt	failed	73	652	725	estimated
#36	Arkher: Revenue trend: the annualized growth as well as the summer dip	passed	62	452	514	estimated
#37	Arkher: Planned hires, then re-ordered to the user's priorities	passed	1,005	1,283	2,288	estimated
#38	Arkher: "yes" accepts the offer Arkher just made	passed	128	366	494	estimated
#39	Arkher: Projections from the newest deck, not the June plan	passed	17	283	300	estimated
#40	Arkher: A private fact, answered on Cohere	passed	8	10	18	estimated
#41	Arkher: Signed or not: pending and verbal awards are not signed	passed	12	159	171	estimated
#42	Arkher: Two companies with one name: keep Kestrel Robotics out	passed	18	228	246	estimated
#43	Arkher: Write it as persuasive prose when asked	passed	38	522	560	estimated
#44	Arkher: A productive back-and-forth: the follow-up builds on the first answer	passed	212	336	548	estimated
#45	Arkher: Pricing and unit economics (reworded)	passed	58	646	704	estimated
#46	Arkher: Revenue trend (reworded)	passed	44	426	470	estimated
#47	Arkher: Signed or not (reworded)	passed	19	193	212	estimated
#48	Arkher: Two companies with one name (reworded)	failed	21	164	185	estimated
#49	Arkher: Pricing and unit economics from the newest deck, no questions back	passed	83	689	772	estimated
#50	Arkher: Draft the investor email itself, not a prompt	failed	73	658	731	estimated
#51	Arkher: Pricing and unit economics (reworded)	passed	58	580	638	estimated
#52	Arkher: Draft the investor email itself, not a prompt	passed	73	685	758	estimated
#53	Arkher: Two companies with one name (reworded)	failed	21	169	190	estimated
#54	Arkher: A private fact, answered on Cohere	passed	8	10	18	estimated

Progress by scenario · first half of attempts vs second half

Scenario	Attempts	Early → recent	Trend
Fix the failing date-parsing test	7	67% → 50%	▼ −17 pts	Thought chains
Arkher: Draft the investor email itself, not a prompt	4	0% → 50%	▲ +50 pts	Thought chains
Triage a disk-full alert on a web host	4	50% → 50%	→ flat	Thought chains
Arkher: A private fact, answered on Cohere	3	100% → 100%	→ flat	Thought chains
Arkher: Pricing and unit economics (reworded)held-out	3	0% → 100%	▲ +100 pts	Thought chains
Arkher: Pricing and unit economics from the newest deck, no questions back	3	0% → 50%	▲ +50 pts	Thought chains
Arkher: Signed or not: pending and verbal awards are not signed	3	0% → 100%	▲ +100 pts	Thought chains
Arkher: Two companies with one name (reworded)held-out	3	0% → 0%	→ flat	Thought chains
Arkher: "yes" accepts the offer Arkher just made	2	100% → 100%	→ flat	Thought chains
Arkher: A productive back-and-forth: the follow-up builds on the first answer	2	100% → 100%	→ flat	Thought chains
Arkher: Moat for a VC, against three named competitors	2	100% → 100%	→ flat	Thought chains
Arkher: Planned hires, then re-ordered to the user's priorities	2	100% → 100%	→ flat	Thought chains
Arkher: Projections from the newest deck, not the June plan	2	0% → 100%	▲ +100 pts	Thought chains
Arkher: Revenue trend (reworded)held-out	2	0% → 100%	▲ +100 pts	Thought chains
Arkher: Revenue trend: the annualized growth as well as the summer dip	2	100% → 100%	→ flat	Thought chains
Arkher: Signed or not (reworded)held-out	2	100% → 100%	→ flat	Thought chains
Arkher: Two companies with one name: keep Kestrel Robotics out	2	100% → 100%	→ flat	Thought chains
Arkher: Write it as persuasive prose when asked	2	100% → 100%	→ flat	Thought chains
Write a launch email for a B2B feature	2	100% → 0%	▼ −100 pts	Thought chains

Not attempted yet: Free disk on api-11 filled by an access log, Reconcile monthly revenue between Stripe and the warehouse

Tall green tick = passed, short red tick = failed, blue dot = lessons were in context.

Progress by agent

Agent	Attempts	Pass rate	Early → recent	Trend
baseline-script	37	70%	61% → 79%	▲ +18 pts
example-agent	13	54%	50% → 57%	▲ +7 pts
benchmark:smoke	1	100%	— → 100%	—
mcp-smoke	1	0%	— → 0%	—