All of the catalog
Scenario
Autoscaling Under Load
Watch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
ScalabilityVerifiedk3dkind
Definition on GitHubWhat you'll do
- Declare metric-driven autoscaling for go-api with a KEDA ScaledObject
- Drive a traffic spike and watch replicas scale up on Prometheus RPS
- Confirm the tail did not pull away from the typical request while scaled, then watch it scale back on cooldown
Stages
- 1instruments
Install the replicas-vs-RPS dashboard the drill is read from
no-autoscaler-yetautoscaling-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
ingressmonitoring/metricsmonitoring/grafanaautoscaling/kedago-api
Related scenarios
- Chaos EngineeringRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
- Cost & Capacity: Right-Sizing Over-Provisioned Resourcesgo-api arrives reserving 2000m CPU and 1Gi of memory - forty times what it uses. Drive real traffic, read the gap between reserved and used on the dashboard and in OpenCost, then right-size the requests. Grading has two sides: a request over the waste ceiling fails, and so does one under the peak the workload actually reached, because under-provisioning is how workloads get throttled and OOM-killed.
- Day-2 Drill: Namespace Backup & RestoreRehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.