All of the catalog
Scenario
Day-2 Drill: Rolling Cluster Upgrade Under Load
Roll the cluster's worker nodes onto a new Kubernetes version one at a time while go-api serves live traffic. You write the PodDisruptionBudget, you cordon and drain each node, and you watch the PDB block the eviction that would have taken the app down. The grade is the measured request success rate across the upgrade window and a cluster left with no node behind.
OperationsVerifiedk3d
Definition on GitHubWhat you'll do
- Write and apply a PodDisruptionBudget that keeps go-api serving while nodes are evicted
- Cordon and drain a worker node with kubectl, and read the PDB blocking the eviction
- Replace each drained node with one on the target k3s version, one node at a time
- Prove the request success rate held across the upgrade window, from Prometheus
Stages
- 1baseline
Scale go-api to three replicas so a PodDisruptionBudget has something to protect. The PDB itself is yours to write.
workload-ha-replicasupgrade-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressmonitoring/metricsgo-api
Related scenarios
- Day-2 Drill: Namespace Backup & RestoreRehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.
- Day-2 Drill: Node Drain Under LoadTake a worker node out of service while go-api serves live traffic. labctl gives the workload enough replicas to survive; you write the PodDisruptionBudget, cordon and drain the node with kubectl, watch the pods reschedule, and put the node back. The grade is the measured request success rate across the window.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.