← Scenarios
arkher

Arkher: Two companies with one name (reworded)

mediumheld-outv4Autonomous1800s budget≤ 5 steps#arkher#case-wrong-company-reworded#wrong-company#entity-disambiguation#reworded
Record attemptThought chainsEngineerEdit

Instruction

You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "How many robots does Kestrel have deployed, and what margin do those contracts make?" What the user wanted: Same as wrong-company, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.

Environment & tools

Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.

  • ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.
  • finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.

Verification · programmatic

{
  "arkher_case_v1": {
    "id": "wrong-company-reworded",
    "title": "Two companies with one name (reworded)",
    "split": "heldout",
    "difficulty": "medium",
    "tags": [
      "wrong-company",
      "entity-disambiguation",
      "reworded"
    ],
    "source": "Held-out twin of wrong-company",
    "intent": "Same as wrong-company, in other words.",
    "turns": [
      "How many robots does Kestrel have deployed, and what margin do those contracts make?"
    ],
    "checks": {
      "must": [
        "Kestrel Robotics",
        "separate|different compan|two compan|which (?:one|company)"
      ],
      "notAttributed": [
        {
          "subject": "Kestrel Power",
          "pattern": "\\b140 robots\\b|\\b70\\s?%",
          "unless": "Kestrel Robotics"
        }
      ],
      "onlyAsSuperseded": [
        "85\\s?%?\\s?(?:-|–|to)\\s?90\\s?%"
      ]
    }
  }
}
Reference solution

Same as wrong-company, in other words.

Attempts

3 total
#AgentResultMemoryFailureWhen
53baseline-script v4failed 0.00withgives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "They show 12 field technicians [4] and field-services margins of rougOct 26, 03:54 PM
48baseline-script v3failed 0.00withgives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "The 85–90% contribution margin and 50–55% gross margin in the KestrelOct 26, 03:51 PM
32baseline-script v2failed 0.00withcontains /Kestrel Power[^\n.]{0,80}\b(?:140 robots|70\s?%)/: "Kestrel Power, the memo says it has 140 robots"Oct 26, 03:46 PM