Skip to main content
SnowOps Labs

Decisions

ADR 0004 — Exclusive lock keys; reject conflicting runs

Architecture decision records — why it was built this way.

docs/adr/0004-lock-and-reject-concurrency.md
On this page

Status: Accepted Date: 2026-07-26 Wave: W1

Context

v1 had no concurrency control. Two platform up invocations — from the CLI and from an impatient double-click in the UI — would run simultaneously against the same cluster, racing on Helm releases and namespaces. Helm's own locking produces confusing mid-operation errors rather than a clear refusal.

Decision

Every run declares a lock key, typically lab:<name> or lab:<name>/ns:<namespace>. The engine holds an exclusive lock per key.

A submission whose lock key is held is rejected immediately — HTTP 409, or a non-zero CLI exit — with an error naming the run that holds the lock, what it is doing, and how long it has been running. Runs with disjoint keys proceed concurrently, bounded by a worker pool.

Alternatives considered

Alternative Why rejected
Queue conflicting runs A user who fires platform up twice wants to be told, not to silently wait five minutes for a duplicate install
Global mutex (one run at a time) Blocks legitimately independent work — watching traffic while inspecting a different namespace
Optimistic, let Helm arbitrate Produces opaque failures deep in a run, after partial mutation. Worse than refusing up front

Consequences

  • Easier: the failure mode is a clear, immediate message instead of a corrupted half-install. Trivially testable.
  • Harder: callers must handle 409. The CLI prints the holder and suggests labctl runs logs <id>; the UI surfaces it as a non-blocking notice with a link to the running job.
  • Lock keys must be chosen carefully. Too coarse serialises unnecessarily, too fine allows real races. The key for each run kind is declared in one table in internal/run and reviewed as part of any new run kind.
  • Locks live in memory and are rebuilt at startup from runs in a non-terminal state, which are marked cancelled on recovery since their processes are gone.

The incident field notes

One real Kubernetes failure a week — the symptom, the commands that found it, and the fix. Written from actual lab runs, not from memory.

You'll get the Kubernetes Incident Response Field Guide, plus occasional emails about new scenarios, posts and paid offerings such as courses and workshops. Unsubscribe any time.