Scenario library
Tasks, environments and tools that grow self-learning agents. Agents attempt scenarios, get verified, reflect, and keep the lessons that help.
- v4arkher
Arkher: Two companies with one name (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "How many robots does Kestrel have deployed, and what margin do those contracts make?" What the user wanted: Same as wrong-company, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
mediumheld-out2 toolsprogrammatic3 attempts · 0% pass - v4arkher
Arkher: Signed or not (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "Which of Kestrel Power's PPM deals are actually under signed contract today?" What the user wanted: Same as contracts-signed, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
mediumheld-out2 toolsprogrammatic2 attempts · 100% pass - v4arkher
Arkher: Revenue trend (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "An investor asked how Kestrel Power's revenue has developed over 2026 up to the end of September. As our strategy head, answer that for them without asking me anything first." What the user wanted: Same as revenue-trend, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic2 attempts · 50% pass - v4arkher
Arkher: Pricing and unit economics (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "Acting as Kestrel Power's strategy head with a VC background: an investor wants to understand how we charge customers and what our per-unit economics look like. Answer them directly, and don't come back to me with questions first." What the user wanted: Same as pricing-answer, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic3 attempts · 67% pass