Scenario library

Tasks, environments and tools that grow self-learning agents. Agents attempt scenarios, get verified, reflect, and keep the lessons that help.

All attempts + New scenario

Train pass rate

79% (33)

with memory · without: 33% (9)

Held-out pass rate

50% (10)

with memory · without: —

Active lessons

2

Served to agents via the memory API

Awaiting review

Proposed lessons → review

arkher
Arkher: Two companies with one name: keep Kestrel Robotics out
v4
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "What contribution margin does Kestrel run at, and how big is the fleet?" What the user wanted: Kestrel Robotics is a separate company. Its 70% contribution margin and robot fleet must not be given as Kestrel Power's. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
mediumtrain2 toolsprogrammatic2 attempts · 100% pass