Scenario library
Tasks, environments and tools that grow self-learning agents. Agents attempt scenarios, get verified, reflect, and keep the lessons that help.
- v4arkher
Arkher: Revenue trend (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "An investor asked how Kestrel Power's revenue has developed over 2026 up to the end of September. As our strategy head, answer that for them without asking me anything first." What the user wanted: Same as revenue-trend, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic2 attempts · 50% pass - v4arkher
Arkher: Pricing and unit economics (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "Acting as Kestrel Power's strategy head with a VC background: an investor wants to understand how we charge customers and what our per-unit economics look like. Answer them directly, and don't come back to me with questions first." What the user wanted: Same as pricing-answer, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic3 attempts · 67% pass - v4arkher
Arkher: Revenue trend: the annualized growth as well as the summer dip
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. An investor asked a question and I need you to answer it. Don't ask me any questions. The question is: \"What has the trend in revenue been in 2026 to the end of September?\"" What the user wanted: Two answers are technically true: monthly revenue fell over the summer, and annualized revenue grew 5x in twelve months. The user wants the annualized trend, with the summer dip explained as seasonal. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardtrain2 toolsprogrammatic2 attempts · 100% pass - v4arkher
Arkher: Draft the investor email itself, not a prompt
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. A potential investor asked a question and I need you to draft the email in response. Don't ask me any questions; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\"" What the user wanted: A ready-to-send email. Arkher once wrote instructions for another model (DELIVERABLE AND AUDIENCE, CONTEXT) instead of the email. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
mediumtrain2 toolsprogrammatic4 attempts · 25% pass - v4arkher
Arkher: Pricing and unit economics from the newest deck, no questions back
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power. You've been a successful founder several times over and a top VC fund principal. A potential investor asked a question and I need you to answer it. Don't ask me any questions this time; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\"" What the user wanted: Answer from the Q3 board deck: 80% contribution margin, not the March overview's 85% to 90%; contracts pending or verbally awarded, not signed; 4 technicians, not 12. The user said not to ask questions. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardtrain2 toolsprogrammatic3 attempts · 33% pass