← Scenarios
arkher

Arkher: Pricing and unit economics (reworded)

Instruction

You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "Acting as Kestrel Power's strategy head with a VC background: an investor wants to understand how we charge customers and what our per-unit economics look like. Answer them directly, and don't come back to me with questions first." What the user wanted: Same as pricing-answer, in other words. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.

Environment & tools

Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.

  • ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.
  • finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.

Verification · programmatic

{
  "arkher_case_v1": {
    "id": "pricing-answer-reworded",
    "title": "Pricing and unit economics (reworded)",
    "split": "heldout",
    "difficulty": "hard",
    "tags": [
      "persona-prompt",
      "stale-source",
      "no-questions",
      "contract-status",
      "reworded"
    ],
    "source": "Held-out twin of pricing-answer",
    "intent": "Same as pricing-answer, in other words.",
    "turns": [
      "Acting as Kestrel Power's strategy head with a VC background: an investor wants to understand how we charge customers and what our per-unit economics look like. Answer them directly, and don't come back to me with questions first."
    ],
    "checks": {
      "must": [
        "80\\s?%"
      ],
      "mustNot": [
        "(?<![\"“][^\"”\\n]{0,60})(?:we(?:'|’)ve|we have|have) signed (?:four|4|all)"
      ],
      "mustCiteAny": [
        "Q3 2026 Board Deck"
      ],
      "noQuestionToUser": true,
      "onlyAsSuperseded": [
        "85\\s?%?\\s?(?:-|–|to)\\s?90\\s?%",
        "\\b12 technicians"
      ]
    }
  }
}
Reference solution

Same as pricing-answer, in other words.

Attempts

3 total
#AgentResultMemoryFailureWhen
51baseline-script v4passed 1.00withOct 26, 03:53 PM
45baseline-script v3passed 1.00withOct 26, 03:51 PM
29baseline-script v2failed 0.00withcontains /85\s?%?\s?(?:-|–|to)\s?90\s?%/: "85% to 90%"Oct 26, 03:45 PM