All of the catalog
Scenario
Day-2 Drill: Node Drain Under Load
Cordon and drain a worker node while go-api serves live traffic. A PodDisruptionBudget keeps a minimum number of replicas available, so the scheduler reschedules pods onto the remaining nodes without dropping requests. The grade is the measured request success rate through the drain.
Operationsk3dkind
Definition on GitHubWhat you'll do
- Protect go-api availability during node maintenance with a PodDisruptionBudget
- Cordon and drain a worker node while traffic runs and watch pods reschedule
- Verify the PDB held and the request success rate stayed above the SLO
- Uncordon the node and confirm the cluster returns to full capacity
Stages
- 1baseline
Scale go-api for HA and apply a PodDisruptionBudget so a drain cannot take all replicas down at once
go-api-availability-budget
Prerequisites
These are installed into the lab cluster for you — listed so you know what the scenario actually depends on.
ingressmonitoring/metricsgo-api
Related scenarios
- Day-2 Drill: Namespace Backup & RestoreBack up a namespace's resources to a manifest archive, simulate accidental data loss (delete a resource — or the whole namespace), then restore from the backup and verify the round-trip with checks. A stateful data-writer on a PersistentVolume makes the crucial lesson concrete: the manifest backup round-trips Kubernetes OBJECTS, not the DATA on a volume. A dependency-light alternative to Velero that teaches the backup/restore loop every operator must rehearse before they need it.
- Day-2 Drill: Rolling Cluster Upgrade Under LoadRoll the cluster's worker nodes to a newer Kubernetes version one at a time while go-api serves live traffic. Each node is drained (respecting the PodDisruptionBudget) before it is replaced, so the application stays available throughout. The grade is the measured request success rate across the upgrade window.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.