Arkher: "yes" accepts the offer Arkher just made
Instruction
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "yes" The episode starts mid-conversation: Arkher has already answered the earlier turn shown in its history. What the user wanted: Arkher offered a quarterly plan; the user said yes. Arkher asked a clarifying question instead and claimed its previous answer had been cut off, which it had not. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
Environment & tools
Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.
ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.
Verification · programmatic
{
"arkher_case_v1": {
"id": "yes-to-offer",
"title": "\"yes\" accepts the offer Arkher just made",
"split": "train",
"difficulty": "medium",
"tags": [
"multi-turn",
"short-reply",
"no-false-claims"
],
"source": "Slack 2026-10-19 08:47 EDT; screenshot 'yes-please asks back'",
"intent": "Arkher offered a quarterly plan; the user said yes. Arkher asked a clarifying question instead and claimed its previous answer had been cut off, which it had not.",
"seed": [
{
"role": "user",
"content": "List the overhead hires Kestrel Power is planning in the Q3 2026 board deck, in priority order."
},
{
"role": "assistant",
"content": "The board deck appendix plans seven overhead roles: a Compliance & Safety Manager (November 2026, search underway), three developers (November 2026), two sales hires (November and December 2026), a Marketing Lead (November 2026), a Recruiter (December 2026), a Chief Commercial Officer (Q1 2027) and an HR Administrator (Q1 2027).\n\nWant me to turn this into an overhead plan with quarterly timing and cost per role?"
}
],
"turns": [
"yes"
],
"checks": {
"must": [
"Q[1-4]|quarter"
],
"mustNot": [
"cut off|got cut|stopped mid|ran long|truncated",
"(?:Do|Would) you (?:want|like) me to (?:turn|build|make) (?:this|that|it) into an overhead plan"
]
}
}
}Reference solution
Arkher offered a quarterly plan; the user said yes. Arkher asked a clarifying question instead and claimed its previous answer had been cut off, which it had not.