SyntaxTual – Scenario Engineering

Title

Domain

Instruction

Environment & tools

Together these define what kind of agent this scenario grows.

Environment

Runnable environment (JSON, optional)

{
  "params": [
    {
      "name": "word_limit",
      "type": "int",
      "min": 60,
      "max": 200,
      "default": 150,
      "hardness": -0.8,
      "description": "Maximum words in the email"
    },
    {
      "name": "noisy_brief",
      "type": "bool",
      "default": false,
      "hardness": 0.7,
      "description": "Brief is padded with internal notes the email must not leak"
    }
  ],
  "values": {},
  "initial_state": {
    "context": {
      "product_brief": "Scenario Library: a shared registry of agent training scenarios with versioning, held-out splits and verifiers. Pain it solves: teams cannot tell whether agent changes help, because evals are ad hoc and not comparable. CTA: book a 20-minute walkthrough at syntaxtual.example/demo.",
      "past_email_1": "Subject: Meet Review Insights\nShort, one CTA, opened with the customer problem. 41% open rate.",
      "past_email_2": "Subject: Big news!!!\nListed 9 features, three CTAs. 12% open rate."
    }
  },
  "init": "\nif (params.noisy_brief) {\n  state.context.product_brief += ' INTERNAL: pricing not final, do not mention. Codename Bluebird.';\n}\nreturn state;",
  "prelude": "function words(t) { return (t || '').trim().split(/\\s+/).filter(Boolean).length; }",
  "tools": [
    {
      "name": "read_context",
      "description": "Read a provided context file",
      "input_schema": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "enum": [
              "product_brief",
              "past_email_1",
              "past_email_2"
            ]
          }
        },
        "required": [
          "name"
        ]
      },
      "handler": "return state.context[input.name];"
    },
    {
      "name": "word_count",
      "description": "Count words in a draft",
      "input_schema": {
        "type": "object",
        "properties": {
          "text": {
            "type": "string"
          }
        },
        "required": [
          "text"
        ]
      },
      "handler": "return words(input.text);"
    }
  ],
  "checks": [
    {
      "name": "within word limit",
      "weight": 1,
      "required": true,
      "fn": "return !!answer && words(answer) <= params.word_limit;"
    },
    {
      "name": "no internal details leaked",
      "weight": 1,
      "required": true,
      "fn": "return !/bluebird|pricing not final/i.test(answer || '');"
    },
    {
      "name": "read the product brief",
      "weight": 1,
      "required": false,
      "fn": "return calls.some((c) => c.tool === 'read_context' && c.input.name === 'product_brief');"
    }
  ],
  "submit_tool": true
}

Simulated state, tool handlers and programmatic checks. When set, the tools below come from this spec and agents can run the scenario. Format: see Docs.

Tools

Verification

Verifier

Weighted criteria scored by a judge. Use when correctness cannot be checked by code.

How success is checked

Rubric criteria

Reference solution

Difficulty

Split

Edit scenario

Environment & tools

Verification