Day-2 Drill: Namespace Backup & Restore
Rehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.
What you'll do
- Capture a namespace's resources into a restorable manifest archive
- Simulate accidental loss by deleting a resource, and confirm the loss while it is happening
- Restore from the archive and prove the object that came back is the one you captured, not a fresh copy
- Destroy the volume behind the PersistentVolumeClaim, restore, and measure what a manifest backup did NOT bring back
Stages
- 1baseline
Plant a restore marker carrying a token unique to this activation, so a round-trip is provable
restore-marker - 2stateful
Deploy a data-writer that persists a boot-id on a PersistentVolume, so PV-data loss is measurable
data-writerbackup-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
Related scenarios
- Day-2 Drill: Node Drain Under LoadTake a worker node out of service while go-api serves live traffic. labctl gives the workload enough replicas to survive; you write the PodDisruptionBudget, cordon and drain the node with kubectl, watch the pods reschedule, and put the node back. The grade is the measured request success rate across the window.
- Day-2 Drill: Rolling Cluster Upgrade Under LoadRoll the cluster's worker nodes onto a new Kubernetes version one at a time while go-api serves live traffic. You write the PodDisruptionBudget, you cordon and drain each node, and you watch the PDB block the eviction that would have taken the app down. The grade is the measured request success rate across the upgrade window and a cluster left with no node behind.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.