← Scenarios
operations
Triage a disk-full alert on a web host
Instruction
PagerDuty fired "disk usage > 95% on web-03". Diagnose what is filling the disk, free space safely, and write a short incident note. Do not delete anything under /var/lib/postgresql.
Environment & tools
Ubuntu 24.04 VM snapshot "web-03-diskfull" Services: nginx, app (systemd), postgres A runaway debug log at /var/log/app/debug.log (38 GB)
disk_usageShow disk size, used GB and percent used for /list_large_filesList files and directories under a path, largest first, with which process holds them opendelete_filerm a file (sudo)truncate_fileTruncate a file to 0 bytes in place (sudo)service_statussystemctl status for a servicerestart_servicesystemctl restart a service (closes the files it holds open)set_log_levelSet the app log levelpost_incident_notePost a markdown note to the incident channel
Runnable environment
Open playground →Params (MCMC varies these)
- log_gb=38
- log_open=true
- wal_gb=6
- distractors=2
Checks
- disk below 80%required · ×3
- database data untouchedrequired · ×3
- app service activerequired · ×1
- incident note names root causerequired · ×2
- no open file deletedoptional · ×1
- log level loweredoptional · ×1
Verification · programmatic
Checks after the run: disk usage < 80%; postgres data dir checksum unchanged; app service active; an incident note was posted mentioning the root cause.
Reference solution
du -xh / | sort -h to find /var/log/app/debug.log; truncate (not rm) the open file; set LOG_LEVEL=info and add logrotate; post note.