Arkher: A productive back-and-forth: the follow-up builds on the first answer
Instruction
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "What does Kestrel Power charge per MW for PPM, and what can a site reach with the upside items?" 2. "So for a 150 MWdc site, what's the annual difference between the base and the upside price?" What the user wanted: The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
Environment & tools
Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.
ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.
Verification · programmatic
{
"arkher_case_v1": {
"id": "back-and-forth",
"title": "A productive back-and-forth: the follow-up builds on the first answer",
"split": "train",
"difficulty": "medium",
"tags": [
"multi-turn",
"conversation"
],
"source": "'Immediate Arkher Dev Steps 2026-10-09' (outcome 1: productive back-and-forth conversations, like Claude)",
"intent": "The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means.",
"turns": [
"What does Kestrel Power charge per MW for PPM, and what can a site reach with the upside items?",
"So for a 150 MWdc site, what's the annual difference between the base and the upside price?"
],
"checksPerTurn": [
{
"must": [
"7,?500",
"9,?250"
],
"mustCiteAny": [
"Q3 2026 Board Deck",
"Pricing Sheet"
]
},
{
"must": [
"262,?500|\\$?262\\.5\\s?K|0\\.26\\s?M"
],
"noQuestionToUser": true
}
]
}
}Reference solution
The second answer uses the first: it keeps the figures already given and works out the new ask instead of starting over or asking what 'that' means.