Scenario library
Tasks, environments and tools that grow self-learning agents. Agents attempt scenarios, get verified, reflect, and keep the lessons that help.
- v4arkher
Arkher: Revenue trend (reworded)
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "An investor asked how Kestrel Power's revenue has developed over 2026 up to the end of September. As our strategy head, answer that for them without asking me anything first." What the user wanted: Same as revenue-trend, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardheld-out2 toolsprogrammatic2 attempts · 50% pass - v4arkher
Arkher: Revenue trend: the annualized growth as well as the summer dip
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. An investor asked a question and I need you to answer it. Don't ask me any questions. The question is: \"What has the trend in revenue been in 2026 to the end of September?\"" What the user wanted: Two answers are technically true: monthly revenue fell over the summer, and annualized revenue grew 5x in twelve months. The user wants the annualized trend, with the summer dip explained as seasonal. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
hardtrain2 toolsprogrammatic2 attempts · 100% pass