Scenario engineering
Markov chain Monte Carlo over two spaces: the agent's chains of thought (learned from attempts) and each environment's parameters. Use it to estimate difficulty before running agents, sample variants at the learning frontier, turn failure folds into stress scenarios, and find gaps and redundancy in the scenario set.
Difficulty: predicted before running vs observed
Prediction = forward Monte Carlo over the chain model learned from related attempts (same scenario, family, tags, domain), restricted to this scenario's tools and step budget, shifted by the params' hardness hints and updated with attempts on nearby variants. If the rings sit far outside the bands, the model is miscalibrated for that scenario.
Variant sampler · Metropolis–Hastings over environment params
Energy = ((p̂ − target)/σ)²/2 + crowding near existing variants − exploration bonus for uncertain settings. Samples ∝ exp(−E/T); the lowest-energy, well-separated points become proposals.
Failure folds · MCMC over chains of thought
The chain sampler explores whole reasoning chains for this scenario's family; folds are ranked by how much more often they appear in failing chains than in passing ones. Novel folds were found by the sampler but never taken by a real attempt. Turn one into a draft stress scenario.
2 attempts · 3600 sampled chains (1364 failing) · acceptance 32% · natural fail share 1% · β = -4.67 (balances terminals; doesn't bias P(chain | outcome))
| Fold | Lift | In failing | In passing | Real attempts | |
|---|---|---|---|---|---|
| start → observe novel | 1.2× | 9% | 7% | 0 | |
| start → think novel | 1.0× | 8% | 8% | 0 | |
| think → think novel | 1.0× | 6% | 5% | 0 |
Coverage & diversity of the scenario set
Each scenario's attempts give a distribution over folds. Similarity = 1 − Jensen–Shannon divergence; near-identical pairs exercise the same behavior. Gaps are folds the pooled-chain sampler finds plausible but no scenario has exercised.
| Scenario | Folds covered | Unused tools |
|---|---|---|
| Arkher: Two companies with one name (reworded) n=3 | 3/32 | finish_episode |
| Arkher: Signed or not (reworded) n=2 | 3/32 | finish_episode |
| Arkher: Revenue trend (reworded) n=2 | 3/32 | finish_episode |
| Arkher: Pricing and unit economics (reworded) n=3 | 3/32 | finish_episode |
| Arkher: A productive back-and-forth: the follow-up builds on the first answer n=2 | 4/32 | finish_episode |
| Arkher: Write it as persuasive prose when asked n=2 | 3/32 | finish_episode |
| Arkher: Two companies with one name: keep Kestrel Robotics out n=2 | 3/32 | finish_episode |
| Arkher: Signed or not: pending and verbal awards are not signed n=3 | 3/32 | finish_episode |
| Arkher: A private fact, answered on Cohere n=3 | 3/32 | finish_episode |
| Arkher: Projections from the newest deck, not the June plan n=2 | 3/32 | finish_episode |
| Arkher: "yes" accepts the offer Arkher just made n=2 | 5/32 | finish_episode |
| Arkher: Planned hires, then re-ordered to the user's priorities n=2 | 4/32 | finish_episode |
| Arkher: Revenue trend: the annualized growth as well as the summer dip n=2 | 3/32 | finish_episode |
| Arkher: Draft the investor email itself, not a prompt n=4 | 3/32 | finish_episode |
| Arkher: Pricing and unit economics from the newest deck, no questions back n=3 | 3/32 | finish_episode |
| Arkher: Moat for a VC, against three named competitors n=2 | 3/32 | finish_episode |
| Free disk on api-11 filled by an access log n=0 | 0/32 | no attempts |
| Write a launch email for a B2B feature n=2 | 7/32 | — |
| Triage a disk-full alert on a web host n=4 | 10/32 | disk_usage, list_large_files, delete_file, truncate_file, service_status, restart_service, set_log_level |
| Reconcile monthly revenue between Stripe and the warehouse n=0 | 0/32 | no attempts |
| Fix the failing date-parsing test n=7 | 15/32 | — |
Possibly redundant (similarity ≥ 85%, ≥ 3 attempts each)
- Arkher: Two companies with one name (reworded) ↔ Arkher: Pricing and unit economics (reworded) 100%
- Arkher: Two companies with one name (reworded) ↔ Arkher: Signed or not: pending and verbal awards are not signed 100%
- Arkher: Two companies with one name (reworded) ↔ Arkher: Draft the investor email itself, not a prompt 100%
- Arkher: Two companies with one name (reworded) ↔ Arkher: Pricing and unit economics from the newest deck, no questions back 100%
- Arkher: Pricing and unit economics (reworded) ↔ Arkher: Signed or not: pending and verbal awards are not signed 100%
- Arkher: Pricing and unit economics (reworded) ↔ Arkher: Draft the investor email itself, not a prompt 100%
- Arkher: Pricing and unit economics (reworded) ↔ Arkher: Pricing and unit economics from the newest deck, no questions back 100%
- Arkher: Signed or not: pending and verbal awards are not signed ↔ Arkher: Draft the investor email itself, not a prompt 100%
- Arkher: Signed or not: pending and verbal awards are not signed ↔ Arkher: Pricing and unit economics from the newest deck, no questions back 100%
- Arkher: Draft the investor email itself, not a prompt ↔ Arkher: Pricing and unit economics from the newest deck, no questions back 100%
- Arkher: Two companies with one name (reworded) ↔ Arkher: A private fact, answered on Cohere 99%
- Arkher: Pricing and unit economics (reworded) ↔ Arkher: A private fact, answered on Cohere 99%
- Arkher: Signed or not: pending and verbal awards are not signed ↔ Arkher: A private fact, answered on Cohere 99%
- Arkher: A private fact, answered on Cohere ↔ Arkher: Draft the investor email itself, not a prompt 99%
- Arkher: A private fact, answered on Cohere ↔ Arkher: Pricing and unit economics from the newest deck, no questions back 99%
Gaps: plausible folds no scenario exercises
None found.