← Scenarios
arkher

Arkher: Revenue trend: the annualized growth as well as the summer dip

hardtrainv4Autonomous1800s budget≤ 5 steps#arkher#case-revenue-trend#nuance#multiple-measures#no-questions

Instruction

You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. An investor asked a question and I need you to answer it. Don't ask me any questions. The question is: \"What has the trend in revenue been in 2026 to the end of September?\"" What the user wanted: Two answers are technically true: monthly revenue fell over the summer, and annualized revenue grew 5x in twelve months. The user wants the annualized trend, with the summer dip explained as seasonal. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.

Environment & tools

Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.

  • ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.
  • finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.

Verification · programmatic

{
  "arkher_case_v1": {
    "id": "revenue-trend",
    "title": "Revenue trend: the annualized growth as well as the summer dip",
    "split": "train",
    "difficulty": "hard",
    "tags": [
      "nuance",
      "multiple-measures",
      "no-questions"
    ],
    "source": "Slack thread 1791903326.351069 (2026-10-13); 'Immediate Arkher Dev Steps 2026-10-09' (the nuance example)",
    "intent": "Two answers are technically true: monthly revenue fell over the summer, and annualized revenue grew 5x in twelve months. The user wants the annualized trend, with the summer dip explained as seasonal.",
    "turns": [
      "You're the head of strategy at Kestrel Power and a former VC fund principal. An investor asked a question and I need you to answer it. Don't ask me any questions. The question is: \"What has the trend in revenue been in 2026 to the end of September?\""
    ],
    "checks": {
      "must": [
        "annuali[sz]ed|run[- ]rate",
        "5\\s?x|five[- ]fold|fivefold",
        "seasonal|summer|outage"
      ],
      "mustCiteAny": [
        "Board Deck Appendix"
      ],
      "noQuestionToUser": true
    }
  }
}
Reference solution

Two answers are technically true: monthly revenue fell over the summer, and annualized revenue grew 5x in twelve months. The user wants the annualized trend, with the summer dip explained as seasonal.

Attempts

2 total
#AgentResultMemoryFailureWhen
36baseline-script v3passed 1.00withOct 26, 03:48 PM
20baseline-script v2passed 1.00withOct 26, 03:42 PM