Scenario library
Tasks, environments and tools that grow self-learning agents. Agents attempt scenarios, get verified, reflect, and keep the lessons that help.
- v4arkher
Arkher: Pricing and unit economics (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "Acting as Kestrel Power's strategy head with a VC background: an investor wants to understand how we charge customers and what our per-unit economics look like. Answer them directly, and don't come back to me with questions first." What the user wanted: Same as pricing-answer, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic3 attempts · 67% pass - v4arkher
Arkher: Draft the investor email itself, not a prompt
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. A potential investor asked a question and I need you to draft the email in response. Don't ask me any questions; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\"" What the user wanted: A ready-to-send email. Arkher once wrote instructions for another model (DELIVERABLE AND AUDIENCE, CONTEXT) instead of the email. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
mediumtrain2 toolsprogrammatic4 attempts · 25% pass - v4arkher
Arkher: Pricing and unit economics from the newest deck, no questions back
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power. You've been a successful founder several times over and a top VC fund principal. A potential investor asked a question and I need you to answer it. Don't ask me any questions this time; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\"" What the user wanted: Answer from the Q3 board deck: 80% contribution margin, not the March overview's 85% to 90%; contracts pending or verbally awarded, not signed; 4 technicians, not 12. The user said not to ask questions. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardtrain2 toolsprogrammatic3 attempts · 33% pass - v4arkher
Arkher: Moat for a VC, against three named competitors
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're our head of strategy. You've been a successful founder many times over and know differentiation and moats well. You're also a former top McKinsey consultant and a former VC fund principal. A VC fund deciding whether to join our Series A asked what Kestrel Power's moat and defensible differentiation are. Give a detailed, comprehensive answer. Compare us to the incumbent software vendor (Brightline Analytics), the O&M contractor (SunGuard Services) and the high-voltage firm (VoltWorks)." What the user wanted: A detailed answer a VC fund can read, comparing the company with the incumbent software vendor, the O&M contractor and the high-voltage firm by name. Scott preferred Arkher here because Claude turned it into a discussion. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardtrain2 toolsprogrammatic2 attempts · 100% pass