All of the catalog
Scenario
Cost & Capacity: Right-Sizing Over-Provisioned Resources
go-api arrives reserving 2000m CPU and 1Gi of memory - forty times what it uses. Drive real traffic, read the gap between reserved and used on the dashboard and in OpenCost, then right-size the requests. Grading has two sides: a request over the waste ceiling fails, and so does one under the peak the workload actually reached, because under-provisioning is how workloads get throttled and OOM-killed.
CostVerifiedk3dkind
Definition on GitHubWhat you'll do
- Read the waste: compare go-api's reserved CPU/memory against what it actually uses under load, on the dashboard and in OpenCost
- Right-size the requests down from 2000m/1Gi - above the observed peak, under the waste ceiling. Both bounds are graded
- Keep go-api healthy and un-throttled while doing it
- Know which knob does what: requests drive scheduling and billing, limits cap usage and are what actually throttles
Stages
- 1inflate
Over-provision go-api's CPU/memory requests 40×/32× (the 'before' state to observe in OpenCost)
workload-overprovisionedopencost-metricscost-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
cost/opencostmonitoring/metricsingressgo-api
Related scenarios
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
- Chaos EngineeringRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.
- Day-2 Drill: Namespace Backup & RestoreRehearse the loop every operator needs before they need it. Capture a namespace to a manifest archive with 'kubectl get -o json | jq', delete something that matters, and put it back with 'kubectl apply --server-side'. Then find the edge of what that buys you: a data-writer holds a boot-id on a PersistentVolume, and restoring its claim gives you back the object bound to a brand-new empty disk. The objects round-trip; the bytes do not. A dependency-light way to learn where a manifest backup stops and a volume snapshot has to start.