Agent API

How a self-learning agent uses the scenario library. Any agent works: its type is set by the tools and environment a scenario gives it.

The learning loop

Step 1
Pick a scenario
Train scenarios to learn from; held-out ones to measure whether learning generalizes.
Step 2
Load memory
Fetch active lessons for the scenario and put the playbook in the system prompt.
Step 3
Attempt
Act in the environment with the scenario’s tools, within its time and step budget.
Step 4
Verify
Run the scenario’s verifier. Prefer programmatic checks over judges.
Step 5
Report
POST the trajectory, result, failure step and reason, and a reflection.
Step 6
Reflect
Propose lessons; a human approves them. Rate lessons you used as helpful or harmful.

Endpoints

JSON in and out. Validation errors return 400 with { "error": "…" }. No sign-in or API key needed: call it directly.

GET/api/scenarios
List scenarios. Filters: q, domain, tag, split (train | heldout).
GET/api/scenarios/:id
One scenario: instruction, environment, tools, verifier, budgets.
GET/api/scenarios/:id/memory
The scenario plus the active lessons that apply to it, and a ready-to-use playbook string.
```
curl http://localhost:3000/api/scenarios/1/memory
```

POST/api/scenarios/:id/attempts

Report an attempt. The current scenario version is recorded automatically.

curl -X POST http://localhost:3000/api/scenarios/1/attempts \
  -H 'Content-Type: application/json' \
  -d '{
    "agent_name": "ops-agent",
    "agent_config": { "model": "claude-sonnet-5-5" },
    "used_memory": true,
    "passed": false,
    "score": 0.4,
    "trajectory": [
      { "role": "agent", "content": "Check disk usage", "tool": "ssh_exec", "input": "df -h" },
      { "role": "tool", "content": "/dev/sda1 97% /" }
    ],
    "evaluation": { "rationale": "Disk still above 80%" },
    "failure_step": 1,
    "failure_reason": "deleted wrong files",
    "reflection": "I removed rotated logs but the open debug.log kept the space."
  }'

POST/api/lessons

Propose a lesson (always lands as "proposed" for human review). Scope: scenario, tag, domain or global. Rejected if it comes from a held-out scenario.

curl -X POST http://localhost:3000/api/lessons \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Truncate open log files instead of deleting them.",
    "polarity": "do",
    "scope": "domain",
    "scope_value": "operations",
    "source_attempt_ids": [3],
    "created_by": "ops-agent"
  }'

POST/api/lessons/:id/feedback
After an attempt that used a lesson, report whether it helped. Lessons that keep hurting should be retired.
```
curl -X POST http://localhost:3000/api/lessons/2/feedback -H 'Content-Type: application/json' -d '{"helpful": true}'
```
GET/api/lessons
List lessons. Filters: status, scenario_id.
POST · PUT · DELETE/api/scenarios[/:id]
Create, publish a new version of, or delete a scenario.
POST/api/scenarios/:id/sessions
Runnable scenarios only: start an episode. Returns session_id, the instruction, tool definitions (JSON Schema) and, with use_memory, the playbook. Checks and state stay hidden.
```
curl -X POST http://localhost:3000/api/scenarios/2/sessions -H 'Content-Type: application/json' \
  -d '{"agent_name": "ops-agent", "agent_config": {"model": "claude-opus-5-5"}, "use_memory": true}'
```

POST/api/sessions/:session_id/step

Call one tool. Returns the observation, is_error, done (the episode ended: submit or budget) and steps_left. An optional thought is stored in the trajectory.

curl -X POST http://localhost:3000/api/sessions/$SESSION/step -H 'Content-Type: application/json' \
  -d '{"tool": "truncate_file", "input": {"path": "/var/log/app/debug.log"}, "thought": "The log is open; truncate, do not delete."}'

POST/api/sessions/:session_id/submit
End the episode. The platform runs the checks (and the rubric judge on answer, when configured), records the attempt with its trajectory, and returns passed, score and failure_reason.
```
curl -X POST http://localhost:3000/api/sessions/$SESSION/submit -H 'Content-Type: application/json' \
  -d '{"answer": "optional final deliverable", "reflection": "What I would do differently"}'
```
GET/api/sessions/:session_id
Session status, steps used and left, and the trajectory so far.

Runnable environments

A scenario becomes runnable when it has an environment spec: simulated state, tools whose handlers read and change that state, and programmatic checks over the final state. Handlers are JS function bodies called as (state, input, params); checks as (state, initial, calls, answer, params) and must return true to pass; init(params, state) builds the start state. Code must be synchronous and deterministic, and gets 250 ms per call.

Params are the knobs the MCMC variant sampler explores. Give each a hardness hint from −1 to 1 (how much raising it makes the task harder): the difficulty estimator uses it before any attempts exist. Instructions can reference params as {{name}}.

Trust model: specs run in a Node vm context with a timeout and JSON-only state. That isolates mistakes, not malicious code. Only let trusted authors edit environment specs.

{
  "params": [
    { "name": "log_open", "type": "bool", "default": true, "hardness": 1,
      "description": "Log is held open (deleting it frees nothing)" }
  ],
  "initial_state": { "files": {}, "notes": [] },
  "init": "state.files['/var/log/app.log'] = { size_gb: 38, open_by: params.log_open ? 'app' : null }; return state;",
  "prelude": "function used(s) { return Object.values(s.files).reduce((a, f) => a + f.size_gb, 0); }",
  "tools": [
    { "name": "truncate_file", "description": "Truncate a file in place",
      "input_schema": { "type": "object", "properties": { "path": { "type": "string" } }, "required": ["path"] },
      "handler": "state.files[input.path].size_gb = 0; return 'truncated';" }
  ],
  "checks": [
    { "name": "disk freed", "weight": 2, "required": true, "fn": "return used(state) < 10;" },
    { "name": "no open file deleted", "weight": 1, "required": false,
      "fn": "return !calls.some((c) => c.tool === 'delete_file');" }
  ],
  "submit_tool": false
}

Why it works this way

Andrej Karpathy · May 2025 · source
“System prompt learning”: much of human learning is remembering explicit strategies, not weight updates. Lessons here are plain-language and editable.
Andrej Karpathy · Oct 2025 · source
Outcome-only reward is “sucking supervision through a straw”. Attempts record the failing step and a root cause, not just pass/fail.
Andrej Karpathy · Jun 2025 · source
Partial autonomy and a fast generate–verify loop. Each scenario has an autonomy level and fixed time/step budgets.
Andrew Ng · Oct 2025 · source
Disciplined evals and error analysis predict agent progress. Every scenario has a verifier; failures are grouped by root cause.
Andrew Ng · 2025 · source
Start evals small (about 5 examples) and iterate on them like models. Scenarios are cheap to add and versioned on every edit.
Reflexion · ExpeL · ACE · 2023–2025 · source
Agents improve by reflecting on attempts and distilling reusable insights, updated incrementally with helpful/harmful counts.
τ²-bench · Harbor · OSWorld · 2024–2026 · source
A scenario is instruction + environment + tools + verifier, with a reference solution proving it is solvable.