Skip to main content
SnowOps Labs

Reference

Incident commands

Inject reversible faults, run game days, and score the fix.

docs/reference/cli/incidents.md
On this page

labctl incident injects realistic, reversible production faults from incidents/ so you can practise diagnosing them. Each fault ships a detection check that confirms when you have actually fixed it. The fault contract is incidents/README.md.

labctl incident list                          # browse the fault library
labctl incident info oom-kill                 # details, upstream references, applyable snippets
labctl incident inject oom-kill               # break the lab on purpose
labctl incident status                        # run the detection check
labctl incident hint                          # next progressive hint (recorded)
labctl incident solution                      # full walkthrough (spoiler; asks to confirm)
labctl incident history                       # past runs: time-to-check, MTTR, hints, resolved-by
labctl incident resolve                       # escape hatch: undo the active fault
labctl incident resolve oom-kill              # works even if active state was lost

Game-day mode

labctl incident inject --random --silent      # surprise fault, name hidden
labctl incident inject --random --seed 42     # reproducible pick for a whole team
labctl incident inject --random --category network

inject flags:

Flag Default Meaning
--random off pick a random eligible fault
--seed — seed for --random, for reproducible team exercises
--category — restrict --random to workload, network, resources, storage or config
--silent off do not reveal which fault was injected
--force off inject even when another incident is active
--deploy-prereqs off build and deploy the target app if it is not running

Rules and timing

  • One active incident at a time; --force overrides.
  • Injection is gated on the fault's prerequisite apps being present.
  • resolve.sh always restores the lab, regardless of partial manual fixes.
  • The first incident status call timestamps time-to-check, the detection proxy. Resolution — the detection check passing, or the escape hatch — closes the run.
  • Each run is appended to .labctl/history/incidents.jsonl with MTTR, hints used, and whether it was resolved manually or via resolve. The escape hatch counts as a non-completion for challenge scoring.

On-call drills

Faults that declare an expectAlert — oom-kill, crashloop-bad-config and bad-deploy-rollout — arm a PrometheusRule when injected. Alertmanager routes the firing alert to the in-cluster pager (labctl service up pager), or to ALERT_WEBHOOK_URL if that was set when monitoring/metrics was installed.

incident status reports whether the page fired by querying Alertmanager at ALERTMANAGER_URL, defaulting to http://alertmanager.<DOMAIN_SUFFIX> through the ingress the metrics provider creates.

REST: GET /api/v2/incidents, POST /api/v2/incidents/{name}/inject[?silent&force], POST /api/v2/incidents/inject-random[?seed&category], GET /api/v2/incidents/status, POST /api/v2/incidents/hint, GET /api/v2/incidents/history, POST /api/v2/incidents/resolve[?name=]. Silent mode hides the fault's identity in API responses until it is resolved.

The incident field notes

One real Kubernetes failure a week — the symptom, the commands that found it, and the fix. Written from actual lab runs, not from memory.

You'll get the Kubernetes Incident Response Field Guide, plus occasional emails about new scenarios, posts and paid offerings such as courses and workshops. Unsubscribe any time.