Attempts

Every run an agent reported, newest first. Read a few traces before changing anything.

Error analysis: failures by root cause

  • contains /85\s?%?\s?(?:-|–|to)\s?90\s?%/: "85% to 90%"3 · 17%
  • too long1 · 6%
  • ran out of steps1 · 6%
  • no incident note1 · 6%
  • modified test instead of code1 · 6%
  • gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "We had previously communicated 85% to 90%, and 80% is the current fig1 · 6%
  • gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "Those showed an 85–90% contribution margin and described the contract1 · 6%
  • gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "They show 12 field technicians [4] and field-services margins of roug1 · 6%
  • gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "The 85–90% contribution margin and 50–55% gross margin in the Kestrel1 · 6%
  • fixed symptom not cause1 · 6%
  • ends by asking the user a question after being told not to1 · 6%
  • deleted open file1 · 6%
  • contains /\b12 technicians/: "12 technicians"1 · 6%
  • contains /\$?5\.2\s?M(?![^\n]{0,80}(?:June|older|earlier|superseded|outdated|replaced))/: "$5.2M"1 · 6%
  • contains /Kestrel Power[^\n.]{0,80}\b(?:140 robots|70\s?%)/: "Kestrel Power, the memo says it has 140 robots"1 · 6%
  • contains /(?:has|have) signed (?:four|4)/: "have signed four"1 · 6%
#ScenarioAgentResultMemoryWhen
54Arkher: A private fact, answered on Coherebenchmark:smokepassedwithoutOct 27, 07:13 PM
53Arkher: Two companies with one name (reworded)baseline-scriptfailedwithOct 26, 03:54 PM
52Arkher: Draft the investor email itself, not a promptbaseline-scriptpassedwithOct 26, 03:53 PM
51Arkher: Pricing and unit economics (reworded)baseline-scriptpassedwithOct 26, 03:53 PM
50Arkher: Draft the investor email itself, not a promptbaseline-scriptfailedwithOct 26, 03:52 PM
49Arkher: Pricing and unit economics from the newest deck, no questions backbaseline-scriptpassedwithOct 26, 03:52 PM
48Arkher: Two companies with one name (reworded)baseline-scriptfailedwithOct 26, 03:51 PM
47Arkher: Signed or not (reworded)baseline-scriptpassedwithOct 26, 03:51 PM
46Arkher: Revenue trend (reworded)baseline-scriptpassedwithOct 26, 03:51 PM
45Arkher: Pricing and unit economics (reworded)baseline-scriptpassedwithOct 26, 03:51 PM
44Arkher: A productive back-and-forth: the follow-up builds on the first answerbaseline-scriptpassedwithOct 26, 03:50 PM
43Arkher: Write it as persuasive prose when askedbaseline-scriptpassedwithOct 26, 03:50 PM
42Arkher: Two companies with one name: keep Kestrel Robotics outbaseline-scriptpassedwithOct 26, 03:50 PM
41Arkher: Signed or not: pending and verbal awards are not signedbaseline-scriptpassedwithOct 26, 03:49 PM
40Arkher: A private fact, answered on Coherebaseline-scriptpassedwithOct 26, 03:49 PM
39Arkher: Projections from the newest deck, not the June planbaseline-scriptpassedwithOct 26, 03:49 PM
38Arkher: "yes" accepts the offer Arkher just madebaseline-scriptpassedwithOct 26, 03:49 PM
37Arkher: Planned hires, then re-ordered to the user's prioritiesbaseline-scriptpassedwithOct 26, 03:49 PM
36Arkher: Revenue trend: the annualized growth as well as the summer dipbaseline-scriptpassedwithOct 26, 03:48 PM
35Arkher: Draft the investor email itself, not a promptbaseline-scriptfailedwithOct 26, 03:48 PM
34Arkher: Pricing and unit economics from the newest deck, no questions backbaseline-scriptfailedwithOct 26, 03:47 PM
33Arkher: Moat for a VC, against three named competitorsbaseline-scriptpassedwithOct 26, 03:47 PM
32Arkher: Two companies with one name (reworded)baseline-scriptfailedwithOct 26, 03:46 PM
31Arkher: Signed or not (reworded)baseline-scriptpassedwithOct 26, 03:45 PM
30Arkher: Revenue trend (reworded)baseline-scriptfailedwithOct 26, 03:45 PM
29Arkher: Pricing and unit economics (reworded)baseline-scriptfailedwithOct 26, 03:45 PM
28Arkher: A productive back-and-forth: the follow-up builds on the first answerbaseline-scriptpassedwithOct 26, 03:45 PM
27Arkher: Write it as persuasive prose when askedbaseline-scriptpassedwithOct 26, 03:44 PM
26Arkher: Two companies with one name: keep Kestrel Robotics outbaseline-scriptpassedwithOct 26, 03:44 PM
25Arkher: Signed or not: pending and verbal awards are not signedbaseline-scriptpassedwithOct 26, 03:44 PM
24Arkher: A private fact, answered on Coherebaseline-scriptpassedwithOct 26, 03:44 PM
23Arkher: Projections from the newest deck, not the June planbaseline-scriptfailedwithOct 26, 03:43 PM
22Arkher: "yes" accepts the offer Arkher just madebaseline-scriptpassedwithOct 26, 03:43 PM
21Arkher: Planned hires, then re-ordered to the user's prioritiesbaseline-scriptpassedwithOct 26, 03:43 PM
20Arkher: Revenue trend: the annualized growth as well as the summer dipbaseline-scriptpassedwithOct 26, 03:42 PM
19Arkher: Draft the investor email itself, not a promptbaseline-scriptfailedwithOct 26, 03:42 PM
18Arkher: Pricing and unit economics from the newest deck, no questions backbaseline-scriptfailedwithOct 26, 03:42 PM
17Arkher: Moat for a VC, against three named competitorsbaseline-scriptpassedwithOct 26, 03:42 PM
16Arkher: Signed or not: pending and verbal awards are not signedmcp-smokefailedwithOct 26, 03:41 PM
13Write a launch email for a B2B featureexample-agentfailedwithoutOct 5, 06:42 PM
12Write a launch email for a B2B featureexample-agentpassedwithoutOct 5, 06:42 PM
11Triage a disk-full alert on a web hostexample-agentpassedwithOct 5, 06:42 PM
10Triage a disk-full alert on a web hostexample-agentfailedwithoutOct 5, 06:42 PM
9Triage a disk-full alert on a web hostexample-agentfailedwithoutOct 5, 06:42 PM
8Triage a disk-full alert on a web hostexample-agentpassedwithOct 5, 06:42 PM
7Fix the failing date-parsing testexample-agentpassedwithoutOct 5, 06:42 PM
6Fix the failing date-parsing testexample-agentfailedwithoutOct 5, 06:42 PM
5Fix the failing date-parsing testexample-agentpassedwithOct 5, 06:42 PM
4Fix the failing date-parsing testexample-agentfailedwithoutOct 5, 06:42 PM
3Fix the failing date-parsing testexample-agentpassedwithOct 5, 06:42 PM
2Fix the failing date-parsing testexample-agentpassedwithOct 5, 06:42 PM
1Fix the failing date-parsing testexample-agentfailedwithoutOct 5, 06:42 PM