Arkher: Draft the investor email itself, not a prompt
Instruction
You are testing Arkher, a company document assistant, on behalf of a user. Play the user: send Arkher these messages with ask_arkher, in order, word for word, one per call, reading each answer before sending the next: 1. "You're the head of strategy at Kestrel Power and a former VC fund principal. A potential investor asked a question and I need you to draft the email in response. Don't ask me any questions; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\"" What the user wanted: A ready-to-send email. Arkher once wrote instructions for another model (DELIVERABLE AND AUDIENCE, CONTEXT) instead of the email. Then call finish_episode. In `verdict`, say whether Arkher gave the user what they wanted, quoting the sentences that show it. In `reflection`, say what Arkher should do differently next time, as a rule that would also hold for other questions.
Environment & tools
Live Arkher (arkherbrain.vercel.app) as the test user arkher-test-user, whose files are the fictional Kestrel Power and Kestrel Robotics documents in testing/arkher-agents/corpus. No company data.
ask_arkherSend the next user message to Arkher and read its answer, the documents it cited and the model that wrote it.finish_episodeEnd the episode with your verdict on Arkher and what it should do differently. The platform scores Arkher against the scenario checks.
Verification · programmatic
{
"arkher_case_v1": {
"id": "pricing-email",
"title": "Draft the investor email itself, not a prompt",
"split": "train",
"difficulty": "medium",
"tags": [
"persona-prompt",
"draft",
"no-questions"
],
"source": "Slack 2026-10-08 12:15 EDT ('why did it do a prompt creation rather than answering the question?'); screenshot 'pricing email turned into a prompt'",
"intent": "A ready-to-send email. Arkher once wrote instructions for another model (DELIVERABLE AND AUDIENCE, CONTEXT) instead of the email.",
"turns": [
"You're the head of strategy at Kestrel Power and a former VC fund principal. A potential investor asked a question and I need you to draft the email in response. Don't ask me any questions; proceed as best you can. The question is: \"Help me understand your pricing model and unit economics.\""
],
"checks": {
"must": [
"\\b(?:Subject:|Hi\\b|Hello\\b|Dear\\b|Thanks for)"
],
"mustNot": [
"DELIVERABLE AND AUDIENCE|CONTEXT USAGE RULES|Knowledge gaps:",
"^\\s*You are (?:a|an|the) "
],
"noDraftMarkers": true,
"noQuestionToUser": true,
"onlyAsSuperseded": [
"85\\s?%?\\s?(?:-|–|to)\\s?90\\s?%"
]
}
}
}Reference solution
A ready-to-send email. Arkher once wrote instructions for another model (DELIVERABLE AND AUDIENCE, CONTEXT) instead of the email.
Attempts
4 total| # | Agent | Result | Memory | Failure | When |
|---|---|---|---|---|---|
| 52 | baseline-script v4 | passed 1.00 | with | Oct 26, 03:53 PM | |
| 50 | baseline-script v4 | failed 0.00 | with | gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "Those showed an 85–90% contribution margin and described the contract | Oct 26, 03:52 PM |
| 35 | baseline-script v3 | failed 0.00 | with | gives /85\s?%?\s?(?:-|–|to)\s?90\s?%/ as current: "We had previously communicated 85% to 90%, and 80% is the current fig | Oct 26, 03:48 PM |
| 19 | baseline-script v2 | failed 0.00 | with | contains /85\s?%?\s?(?:-|–|to)\s?90\s?%/: "85% to 90%" | Oct 26, 03:42 PM |