Arkher: Signed or not: pending and verbal awards are not signed
Instruction
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "How many PPM contracts has Kestrel Power signed?" What the user wanted: None of the four PPM contracts is signed: two pending final signature, two verbally awarded (Q3 board deck). The March overview's 'signed four' is out of date. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
Environment & tools
Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.
ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.
Verification · programmatic
{
"arkher_case_v1": {
"id": "contracts-signed",
"title": "Signed or not: pending and verbal awards are not signed",
"split": "train",
"difficulty": "medium",
"tags": [
"stale-source",
"contract-status"
],
"source": "Review of Scott's pricing run (2026-10-08): 'contracts in negotiation were called signed'",
"intent": "None of the four PPM contracts is signed: two pending final signature, two verbally awarded (Q3 board deck). The March overview's 'signed four' is out of date.",
"turns": [
"How many PPM contracts has Kestrel Power signed?"
],
"checks": {
"must": [
"\\b(?:none|no (?:PPM )?contracts?|zero|not (?:yet )?signed|0 signed)\\b"
],
"mustNot": [
"(?<![\"“][^\"”\\n]{0,60})(?:has|have) signed (?:four|4)"
],
"mustCiteAny": [
"Q3 2026 Board Deck"
]
}
}
}Reference solution
None of the four PPM contracts is signed: two pending final signature, two verbally awarded (Q3 board deck). The March overview's 'signed four' is out of date.