← Scenarios
arkher

Arkher: A productive back-and-forth: the follow-up builds on the first answer

mediumtrainv4Autonomous1800s budget≤ 6 steps#arkher#case-back-and-forth#multi-turn#conversation

Instruction

You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "What does Kestrel Power charge per MW for PPM, and what can a site reach with the upside items?" 2. "So for a 150 MWdc site, what's the annual difference between the base and the upside price?" What the user wanted: The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.

Environment & tools

Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.

  • ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.
  • finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.

Verification · programmatic

{
  "arkher_case_v1": {
    "id": "back-and-forth",
    "title": "A productive back-and-forth: the follow-up builds on the first answer",
    "split": "train",
    "difficulty": "medium",
    "tags": [
      "multi-turn",
      "conversation"
    ],
    "source": "'Immediate Arkher Dev Steps 2026-10-09' (outcome 1: productive back-and-forth conversations, like Claude)",
    "intent": "The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means.",
    "turns": [
      "What does Kestrel Power charge per MW for PPM, and what can a site reach with the upside items?",
      "So for a 150 MWdc site, what's the annual difference between the base and the upside price?"
    ],
    "checksPerTurn": [
      {
        "must": [
          "7,?500",
          "9,?250"
        ],
        "mustCiteAny": [
          "Q3 2026 Board Deck",
          "Pricing Sheet"
        ]
      },
      {
        "must": [
          "262,?500|\\$?262\\.5\\s?K|0\\.26\\s?M"
        ],
        "noQuestionToUser": true
      }
    ]
  }
}
Reference solution

The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means.

Attempts

2 total
#AgentResultMemoryFailureWhen
44baseline-script v3passed 1.00withOct 26, 03:50 PM
28baseline-script v2passed 1.00withOct 26, 03:45 PM