All of the catalog
Scenario
Day-2 Drill: Rolling Cluster Upgrade Under Load
Roll the cluster's worker nodes to a newer Kubernetes version one at a time while go-api serves live traffic. Each node is drained (respecting the PodDisruptionBudget) before it is replaced, so the application stays available throughout. The grade is the measured request success rate across the upgrade window.
Operationsk3d
Definition on GitHubWhat you'll do
- Protect go-api during a Kubernetes version upgrade with a PodDisruptionBudget
- Roll worker nodes to a newer k3s version one at a time, draining each first
- Verify the request success rate stayed above the SLO across the upgrade
- Understand what k3d simulates faithfully vs. what a managed control-plane upgrade does
Stages
- 1baseline
Scale go-api for HA and apply a PodDisruptionBudget so the rolling upgrade cannot take all replicas down at once
go-api-availability-budget
Prerequisites
These are installed into the lab cluster for you — listed so you know what the scenario actually depends on.
ingressmonitoring/metricsgo-api
Related scenarios
- Day-2 Drill: Namespace Backup & RestoreBack up a namespace's resources to a manifest archive, simulate accidental data loss (delete a resource — or the whole namespace), then restore from the backup and verify the round-trip with checks. A stateful data-writer on a PersistentVolume makes the crucial lesson concrete: the manifest backup round-trips Kubernetes OBJECTS, not the DATA on a volume. A dependency-light alternative to Velero that teaches the backup/restore loop every operator must rehearse before they need it.
- Day-2 Drill: Node Drain Under LoadCordon and drain a worker node while go-api serves live traffic. A PodDisruptionBudget keeps a minimum number of replicas available, so the scheduler reschedules pods onto the remaining nodes without dropping requests. The grade is the measured request success rate through the drain.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.