Secrets Management & Rotation
A workload mounts a Vault-backed secret that External Secrets keeps in sync. You rotate the value in Vault; it must reach the running pod through ESO and the kubelet with no restart and no redeploy. The grade is what the running container reads from disk.
What you'll do
- Compare two workloads consuming the same Vault-backed Secret - one as a file, one as an env var
- Rotate the value in Vault with the vault CLI and watch it flow through ESO to the pod
- Prove the rotation reached the file-mounted container with no restart and no redeploy
- Show the env-var container never sees it, and say why that is not a bug
- Read the rotation on ESO's own metrics rather than by polling kubectl
Stages
- 1consume
Seed a known baseline in Vault and stand up two consumers of the same Secret - one mounting it as a file, one taking it as an env var
secret-consumersseed-baselinerotation-dashboard
Prerequisites
--deploy-prereqs installs these into the lab for you. They are listed so you know what the scenario depends on.
Related scenarios
- Security & Compliance (Audit → Enforce)Take a cluster from 'policies exist' to 'policies bite'. Kyverno ships four Pod Security and best-practice ClusterPolicies in Audit mode, so nothing is blocked and every violation lands in a policy report. Your job is the part a real team owns: read the report, remediate the workload at the source, promote the policies to Enforce, prove the admission webhook now rejects a non-compliant Pod, and issue a real TLS certificate from the lab CA with cert-manager - on a namespace that is default-deny in both directions.
- Autoscaling Under LoadWatch KEDA scale go-api on Prometheus RPS: a spike drives it from 1 to several replicas, then cooldown brings it back. The flagship 'autoscaling actually works' demo, verified under traffic from the load generator.
- Chaos EngineeringRun the chaos engineering loop against go-api with Chaos Mesh: establish a steady state under load, form a hypothesis, inject a failure (pod kill, network delay, CPU and memory stress) and measure the blast radius on a Grafana dashboard. go-api ships as a single replica behind a PodDisruptionBudget, so it starts out with disruptionsAllowed=0: a node drain would block forever while a pod kill is still a full outage. Your job is to find that with an experiment and fix it, then prove the same experiment no longer hurts.