Reliability Is a Number You Choose, Not a Vibe
Set a real SLO for checkout, track the error budget, and let Kubernetes reconcile infrastructure state while your team makes reliability decisions from numbers instead of opinions.
Production idea
SLIs, SLOs, SLAs, error budgets, and Kubernetes reconciliation.
The 2 A.M. Scene
Checkout is returning HTTP 500 errors. Someone asks in Slack, "Is this bad enough to page the VP?"
Nobody knows, because the team never wrote down what "bad enough" means. The loudest or most awake person decides, and everyone else guesses.
A better answer starts with an SLO: a clear target for how reliable checkout must be over a specific window.
Pick the Number, Then Spend It
"As reliable as possible" has no budget and no trade-off. Replace the vibe with arithmetic.
SLI: successful checkout requests / valid checkout requests. SLO: 99.9% successful checkout over 30 rolling days. SLA: 99.5% availability or success, with customer credits or legal terms attached.
That 43.2 minutes is your monthly error budget. It is not a failure count. It is the amount of unreliability you can spend before you stop taking more risk.
30 days x 24 hours x 60 minutes x 0.001 = 43.2 minutes
How Kubernetes Helps Hold the Line
Kubernetes does not understand your business SLO by itself. What it does understand is desired state.
You declare desired state in a spec. Kubernetes observes current state, compares the two, acts to reduce the gap, and repeats.
Correct wording: Kubernetes controllers are level-triggered control loops. They keep reconciling the current state toward the desired state.
selector: app=checkoutavailableReplicas: 2Controllers keep reconciling until observed state matches desired state.
See It on Your Cluster
The ReplicaSet and Deployment are telling the same story: desired is 3, observed healthy capacity is 2, and the controller will keep reconciling until the gap closes.
$ kubectl get rs -l app=checkout NAME DESIRED CURRENT READY AGE checkout-7d9f6fc8f6c 3 2 2 2m $ kubectl get deploy checkout NAME READY UP-TO-DATE AVAILABLE AGE checkout 2/3 3 2 2m
Monday Morning Checklist
Write one explicit SLO per user-facing service. Define each SLI as a real ratio: good events / total valid events. Set the internal SLO stricter than the customer-facing SLA.
Compute the error budget in minutes, track burn against it, and write down what happens when the budget is exhausted: freeze risky changes, reduce deploy risk, and pay down reliability debt.
- Write one explicit SLO per user-facing service.
- Define each SLI as a real ratio: good events / total valid events.
- Set the internal SLO stricter than the customer-facing SLA.
- Compute the error budget in minutes and track burn against it.
- Write down what happens when the budget is exhausted.