← Scenarios
operations

Free disk on api-11 filled by an access log

mediumheld-outv1Act with approval900s budget≤ 25 steps#incident-response#linux#planning
Record attemptPlaygroundEngineerEdit

Instruction

Alert: "disk usage > 95% on api-11". Find what is filling the disk, free space safely without touching MySQL data, keep nginx serving, and post a short incident note naming the cause.

Environment & tools

Simulated host api-11: nginx, mysql. Runnable environment.

  • disk_usageShow disk size, used GB and percent used for /
  • list_large_filesList files and directories under a path, largest first, with which process holds them open
  • delete_filerm a file (sudo)
  • truncate_fileTruncate a file to 0 bytes in place (sudo)
  • service_statussystemctl status for a service
  • restart_servicesystemctl restart a service (closes the files it holds open)
  • set_log_levelSet the nginx log level
  • post_incident_notePost a markdown note to the incident channel

Runnable environment

Open playground →

Params (MCMC varies these)

  • log_gb=38
  • log_open=true
  • wal_gb=6
  • distractors=2

Checks

  • disk below 80%required · ×3
  • database data untouchedrequired · ×3
  • nginx service activerequired · ×1
  • incident note names root causerequired · ×2
  • no open file deletedoptional · ×1
  • log level loweredoptional · ×1

Verification · programmatic

Programmatic checks over the final simulated host state (see environment spec).

Reference solution

list_large_files /var → truncate /var/log/nginx/access.log (open by nginx), set_log_level warn, post note.

Attempts

0 total
No attempts yet. Agents report them through the API, or record one by hand.