CS Fundamentals Trainer · Guides

Containers and Kubernetes

What a container really is (namespaces, cgroups and image layers, measured), and how Kubernetes keeps desired state: the control plane, scheduling by requests, rolling updates and probes, and autoscaling.

40 min read

"A container is a lightweight virtual machine" is the most common description and the least accurate. A container is an ordinary Linux process that the kernel has been told to lie to in three ways: about what it can see (namespaces), about how much it may use (control groups, or cgroups) and about which files it has (a layered root file system). Kubernetes then runs thousands of those processes across many machines, and its whole design is one idea repeated: you declare the state you want, and controllers keep working until reality matches.

This guide builds a container by hand with the kernel's own tools and measures what the limits do, then moves up to Kubernetes: its parts, its scheduler, rolling updates, probes and autoscaling. Everything in the first half was run in a Linux sandbox as root; the scripts are in the appendix. It follows on from operating systems, and its deployment topics connect to Git and CI/CD and microservices.

1. A container is a process with three disguises

MechanismWhat it limitsThe lie it tells
Namespaceswhat the process can see"you are process 1 on a machine called web-7f9, with only these processes, network devices and mounts"
cgroupshow much it can use"you have 50 MB of memory and a fifth of a CPU" (enforced, not just advertised)
Layered file systemwhich files it has"this is your root directory" (assembled from image layers plus a writable layer)

Diagram: Containers share the host's kernel; virtual machines each bring one

Because there is only one kernel, a container starts in milliseconds and has almost no overhead, and a vulnerability in the kernel is shared by everyone on the host; that is the main difference in isolation strength from a virtual machine.

Namespaces: what the process sees

A namespace gives a process its own view of one kind of resource: process ids (PID), network interfaces, mount points, host name (UTS), users, inter-process communication, and cgroups. I started a shell in a new PID namespace with unshare:

WhereProcesses visible
inside the new PID namespace4, and the shell sees itself as PID 1
on the host71

The process is the same kernel object the host sees under some other number; it simply cannot see the rest. A new UTS namespace let me change the host name to demo inside while the machine's name stayed vm. A container runtime creates a full set at once.

cgroups: how much it can use

Control groups account for and limit the resources of a set of processes. They are what turns "a container" from a view into a budget. I put a Python program in a cgroup with a 50 MB memory limit and had it allocate and touch 10 MB at a time:

allocated 10 MB
allocated 20 MB
allocated 30 MB
allocated 40 MB
Killed        exit status: 137

At the fifth block the kernel's OOM killer ended the process with SIGKILL, and the shell reported exit status 137, which is 128 + 9 (signal 9). That is what you see in Kubernetes as OOMKilled with exit code 137. Notice what did not happen: there was no exception the program could catch. Memory is a hard limit, and the process dies. Page cache, the memory of child processes, and in-memory file systems also count towards the limit.

CPU works differently: it is a rate, so exceeding it means being throttled (paused), not killed. I gave a busy loop a quota of 20 ms of CPU time per 100 ms period:

SetupLoop iterations per secondCPU time used in 3 seconds
no limit6.45 million2.94 s
quota of 20% of one core1.30 million0.62 s

The limited process got 21% of the time and ran at 20% of the speed. For a web service this appears as latency: a request that needs 50 ms of CPU may take 50 ms, or, if the quota is used up in that period, it waits for the next one. CPU throttling is a classic cause of mysterious p99 latency spikes in containers whose average CPU use looks low.

Two runtime consequences follow. First, programs must read the limit: a JVM or Go runtime that sizes its heap and thread pools from the host's memory and cores, not from the container's, will be killed or throttled (modern runtimes read the cgroup limits; check yours, see garbage collection). Second, set the memory limit above the real peak (including the runtime's overhead) and monitor how close you get.

Image layers: what files it has

A container image is a stack of read-only layers, each a set of file changes (the result of one build step), plus a thin writable layer on top. The kernel's overlay file system merges them. I made a lower layer with a.txt containing "base", then changed a.txt and added b.txt in the merged view:

LayerContent
lower (read-only)a.txt = base (untouched)
upper (writable)a.txt (the changed copy) and b.txt
merged viewa.txt = changed, b.txt = new

This is copy-on-write again (as with fork in the operating systems guide): the first write to a file copies it into the upper layer. It explains why many containers can share one image on disk, why starting a container is fast, and why data written inside a container is lost when it is removed unless it is on a mounted volume. Practical consequences for building images:

Docker is one way to build and run all this; the images follow the OCI standard and run under containerd or CRI-O as well, which is what Kubernetes uses.

2. Kubernetes: declare the state you want

One host can run containers by hand. To run hundreds across dozens of machines, with restarts, updates, scaling and networking, you need an orchestrator. Kubernetes is built on desired state: you submit objects that describe what you want ("four replicas of this image, reachable on this name"), and controllers run a loop forever: observe the actual state, compare it with the desired state, and act to close the gap.

Diagram: The parts of a Kubernetes cluster

The main objects

ObjectWhat it is
Podthe smallest unit: one or more containers sharing a network address and volumes, scheduled together
Deployment (and ReplicaSet)keeps N identical pods running and performs rolling updates
Servicea stable name and virtual IP that load-balances across the healthy pods matching a label selector (load balancing, DNS)
Ingress / GatewayHTTP routing from outside the cluster to Services
ConfigMap, Secretconfiguration and credentials injected as environment variables or files (secrets management). A Secret is only base64, so enable encryption at rest and restrict access
StatefulSetpods with stable identities and their own persistent volumes, for databases and similar
Job, CronJobrun to completion, once or on a schedule
HorizontalPodAutoscalerchanges the replica count from a metric
PersistentVolume(Claim)storage that outlives a pod

Pods are cattle, not pets: they are created, killed and rescheduled at any time and their IP addresses change, so nothing should depend on a particular pod. You address the Service, and keep state in volumes or external stores.

3. Scheduling: requests decide, usage does not

Each container declares requests (what the scheduler should reserve) and limits (what the cgroup enforces). The scheduler puts a pod only on a node with enough unrequested capacity, then scores the candidates (spreading, affinity, and so on). It works from requests, not from actual use.

I packed 200 pods with random CPU requests (244 CPUs in total) onto 8-CPU nodes:

PolicyNodes usedRequested CPU filled
first fit3198%
best fit3198%
spread (least loaded node first)3295%

The theoretical minimum is 30.5, so all three policies pack well; spreading costs one node and buys resilience, because a failing node takes fewer pods of the same service with it. The same pods actually used about 90 CPUs, 37% of what they requested. That gap is the usual cost problem: requests that are too high waste money (the nodes look full and are idle), and requests that are too low let the scheduler over-pack nodes, which then throttle or evict pods under load. Measure real use and set requests near the typical value and limits with headroom. The quality of service class follows from this: pods whose requests equal their limits are Guaranteed and are evicted last; pods with some requests are Burstable; pods with none are BestEffort and are the first to go when a node runs short.

3.1 Requests and limits: what each one really does

The two numbers are easy to confuse, and they do different jobs at different times. I measured them on real cgroups (Linux, cgroups v1, as root; scripts in the appendix) and the rest comes from the Kubernetes documentation.

CPUMemory
Requestused by the scheduler to place the pod; becomes a relative weight (cpu.shares) that matters only when the CPU is contendedused by the scheduler to place the pod; nothing is enforced at run time, but it decides who is evicted first when the node runs short
Limita hard cap (cfs_quota_us per cfs_period_us, a 100 ms period by default); the container is throttled, never killeda hard ceiling (memory.limit_in_bytes); beyond it the kernel's OOM killer kills the process (OOMKilled, exit status 137)

Kubernetes also derives a QoS class from these numbers (documentation): requests equal to limits for every container gives Guaranteed; some requests or limits gives Burstable; none gives BestEffort. Under node pressure BestEffort pods are evicted first, then Burstable pods using more than their requests, and Guaranteed pods last.

A CPU request is a weight, not a cap. Two spinning processes on one CPU with shares 1024 and 256 (a 4:1 request ratio) received 78% and 20% of the CPU. The one with the small share, running alone, received 99%: a request reserves a claim, and nothing more, when nobody else wants the CPU.

A CPU limit is a cap, even on an idle machine. A process alone on an idle CPU with a limit of 0.2 CPU (20 ms per 100 ms) got 21%, and was throttled in all 41 periods of the test.

Throttling shows up as latency, while average CPU looks fine. The usual picture is a service whose dashboards show 40 to 50% CPU. I ran a service with two worker processes and bursty traffic (page loads of three requests at a time; each request 50 ms of CPU; about 0.49 CPUs on average) on a machine with two cores, under different limits:

CPU limitCPU actually usedp50p99Periods throttled
none0.4955 ms194 ms
2.0 CPUs (not binding)0.4955 ms189 ms0 of 122
1.0 CPU0.4997 ms559 ms71 of 144 (49%)
0.6 CPU0.48306 ms1,834 ms151 of 184 (82%)
0.5 CPU0.47653 ms2,559 ms192 of 205 (94%)

The average use is the same in every row. A limit of 1.0 CPU is twice the average and still tripled the p99: a burst of three requests needs 150 ms of CPU, the quota allows 100 ms per period, so the rest waits for the next period. With a limit close to the average the queue never drains. This is the "average CPU looks fine, p99 is terrible" pattern: look at the throttling metrics (container_cpu_cfs_throttled_periods_total over container_cpu_cfs_periods_total in cAdvisor/Prometheus), not at average utilisation. A throttled pod also fails its own health probes by slowness: a readiness probe with a 1-second timeout would have flapped in the 0.6 and 0.5 rows.

Options when a service is throttled: raise the limit well above the burst you need; set a CPU request that matches real use and leave the CPU limit high or unset (many teams do this for latency-sensitive services, accepting that a noisy neighbour can then take more than its share when the node is busy; others keep limits for fairness in shared clusters, so this is a policy decision); and make the application's thread pools match the CPU it will really get (a runtime that sees all the host's cores will start too many threads).

A memory limit set from the average kills at the peak. I ran a program with a steady 40 MB working set and a 50 MB peak held for 0.2 s every 0.6 s (average about 57 MB):

Memory limitResult
64 MB (just above the average)killed by SIGKILL (exit status 137 in a shell) at the first peak; 0 of 6 peaks survived; the limit was hit 36 times before the kill
128 MB (above the peak of 90 MB)all 6 peaks survived; highest use 90 MB; the limit never hit

Memory is not compressible like CPU: there is no "throttling" that slows the process instead; at the limit the kernel first tries to reclaim memory (dropping page cache, which slows file I/O), and when it cannot, it kills. Set the memory limit from the peak, with headroom, and set the request equal to the limit for predictable placement and the Guaranteed class for the memory side. Managed runtimes need their own settings, because their heap is separate from the container's limit: for example -XX:MaxRAMPercentage for the JVM and GOMEMLIMIT for Go, leaving room for stacks, native memory and the runtime (garbage collection).

Memory pressure before the kill. Between "fine" and "killed" a workload slows down: page cache for files is evicted (databases and file-serving services lose their cached data and read from disk again), garbage collectors run more often, and with swap enabled, pages go to disk. In the kernel, pressure stall information (/proc/pressure/memory, cpu and io) says what fraction of time tasks waited for the resource. I could not reproduce the page-cache effect in my sandbox (reading a 300 MB file under a 100 MB limit ran at the same speed as under a 600 MB limit, so the sandbox evidently does not charge file cache to the limit); treat this paragraph as documentation, not as a measurement. Node-level pressure is different from a container reaching its own limit: when the node runs short, the kubelet evicts pods (status Evicted, message "The node was low on resource: memory"), chosen by the QoS order above; a pod killed for exceeding its own limit shows OOMKilled instead.

4. Rolling updates

A Deployment replaces pods gradually, under two rules that hold at every moment: at most replicas + maxSurge pods exist, and at least replicas - maxUnavailable pods are ready. Step through an update of four pods with maxSurge: 1 and maxUnavailable: 0, and then see what happens when the new version is broken:

Interactive animation: rolling update

In the model, an update with maxSurge 1 and maxUnavailable 0 took 12 steps and never had fewer than 4 ready pods: capacity is never reduced, at the price of one extra pod and a slower rollout. Other settings trade speed against capacity:

StrategySteps to finishFewest ready pods during the update
maxSurge 1, maxUnavailable 0124
maxSurge 2, maxUnavailable 163
maxSurge 0, maxUnavailable 1 (no spare capacity needed)133
maxSurge 0, maxUnavailable 4 (like the Recreate strategy)40

The case that proves the design is the broken release: when the new pods never pass their readiness check, a rollout with maxUnavailable 0 stalls with all four old pods still serving traffic and one new pod stuck. With maxUnavailable 1, three old pods keep serving and two new pods are stuck. Users see nothing wrong, and the rollout itself stays in progress until someone notices (so alert on a rollout that exceeds its progressDeadlineSeconds) and runs kubectl rollout undo. This works only if the readiness probe really tests the app.

Probes

Combine them with a graceful shutdown: on termination Kubernetes sends SIGTERM and waits (terminationGracePeriodSeconds) before SIGKILL; the app should stop accepting new work, finish in-flight requests and exit. Without it, every deploy drops requests (resilience patterns). A PodDisruptionBudget limits how many pods of a service can be down at once during node maintenance.

Troubleshooting walkthrough: OOMKilled, failing probes, Pending pods and stalled rollouts

When a release goes wrong, the useful habit is the same each time: status, then events, then logs, then metrics. The commands and messages below come from the Kubernetes documentation and are typical; I had no cluster in the sandbox, so I did not capture them from one, and the exact wording differs between versions. The two things I could test, the cgroup behaviour and the rollout rules, are measured above and below.

kubectl get pods -o wide                       # STATUS, READY (0/1?), RESTARTS, which node
kubectl describe pod <pod>                     # Events at the bottom; Last State; Conditions; Requests/Limits
kubectl logs <pod> --previous                  # logs of the container that just died, not the new one
kubectl get events --sort-by=.lastTimestamp    # what the cluster did, in order
kubectl top pod <pod> --containers             # current CPU and memory (needs metrics-server)
kubectl rollout status deployment/<name>       # waits; reports why it is stuck
kubectl rollout undo deployment/<name>         # go back to the previous revision

1. OOMKilled and CrashLoopBackOff

2. Pods that are Running but not Ready (failing readiness probes)

3. Pods stuck in Pending

4. A rollout that does not finish

kubectl rollout status keeps waiting and the Deployment shows Progressing until progressDeadlineSeconds (600 by default) passes, then reports ProgressDeadlineExceeded. Kubernetes does not roll back by itself; it stops advancing and leaves the mix of old and new pods in place, so kubectl rollout undo (or your pipeline's rollback) is a deliberate step. Three different problems look the same from outside, and I simulated them with the rolling-update rules (16 replicas, each requesting 1 CPU, maxSurge 4):

SituationResult
a cluster of 16 CPUs, fully requested, maxUnavailable 0, healthy new versionstalled: 16 old pods ready, 4 new pods Pending; no capacity for a surge pod, and no old pod may be removed first
the same cluster, maxUnavailable 4finished in 13 ticks (old pods are removed to make room)
one more node (24 CPUs), maxUnavailable 0finished in 12 ticks
the same 16 CPUs, but requests lowered so that 20 pods fitfinished in 12 ticks
24 CPUs, maxUnavailable 0, new version never passes readinessstalled: 16 old pods still serving, 4 new pods starting but never ready
24 CPUs, maxUnavailable 4, new version never passes readinessstalled, but with only 12 old pods ready and 8 new pods not ready: capacity already reduced

How to tell them apart: Pending new pods mean a capacity or scheduling problem (section 3 above); new pods that are Running but not Ready mean a probe or application problem (item 2); ImagePullBackOff or ErrImagePull means a wrong image name, tag or registry credentials; CrashLoopBackOff means the new version crashes (item 1). Prevent the stalls that are not bugs: leave spare capacity for the surge (or use a small maxUnavailable), set requests from measurements, gate the rollout on a readiness probe that tests the new version, use minReadySeconds so that a pod that is ready for one second and then fails does not count as healthy, and have the pipeline fail and roll back on ProgressDeadlineExceeded instead of waiting for a person.

5. Autoscaling and capacity

The Horizontal Pod Autoscaler computes the replica count from a metric, by the rule:

desiredReplicas = ceil( currentReplicas × currentMetric / targetMetric )

For example, 4 pods at 90% average CPU with a target of 60% gives ceil(4 × 90 / 60) = 6 pods. It acts on requests-relative CPU, so wrong requests mean wrong scaling, and it needs time to react (metrics delay, pod start-up, image pull), so it does not absorb a sudden spike by itself; keep headroom, and scale on a better signal (queue depth, requests per second) where CPU is a poor proxy. The cluster autoscaler adds nodes when pods cannot be scheduled. Vertical scaling (the VPA) adjusts requests rather than the count. See load balancing and the queueing arguments in resilience patterns for why running close to 100% utilisation is dangerous.

6. Stateful workloads, networking and security in brief

7. Common mistakes

8. Things to mention in an interview

Quick recap

Appendix: the experiment code

Four files. containers.sh is a shell script that must run as root on Linux with cgroups version 1 (on version 2 the control files are memory.max and cpu.max under /sys/fs/cgroup); it shows the PID and host-name namespaces, the memory and CPU limits and an overlay mount, and uses hog.py and spin.py. k8s.py uses only the Python standard library and simulates first-fit, best-fit and spread placement and the Deployment rollout rules. The process counts, exit status 137 and CPU figures are real measurements from a small sandbox and vary between machines; the scheduling and rollout results are simulations of the documented rules with random pod sizes, not a run of a real Kubernetes cluster.

Namespaces, cgroups and image layers by hand (shell)

#!/bin/bash
# Run as root on Linux with cgroups v1 (as in my sandbox); on cgroups v2 the file names differ (memory.max, cpu.max).
set -u

echo "--- 1. a new PID namespace: the process believes it is the only one"
unshare -p -f --mount-proc -- sh -c 'echo "inside: my pid is $$, processes visible: $(ps -e | wc -l)"'
echo "on the host: processes visible: $(ps -e | wc -l)"

echo "--- 2. a new UTS namespace: its own host name"
unshare -u sh -c 'hostname demo; echo "inside: $(hostname)"'; echo "outside: $(hostname)"

echo "--- 3. a memory limit of 50 MB (hog.py joins the cgroup and allocates 10 MB at a time)"
mkdir -p /sys/fs/cgroup/memory/t1 && echo 52428800 > /sys/fs/cgroup/memory/t1/memory.limit_in_bytes
python3 hog.py; echo "exit status: $?   (137 = 128 + 9, killed by SIGKILL)"
echo "limit hits (failcnt): $(cat /sys/fs/cgroup/memory/t1/memory.failcnt)"

echo "--- 4. a CPU quota of 20 ms per 100 ms period = 20% of one core"
mkdir -p /sys/fs/cgroup/cpu/t2
echo 20000 > /sys/fs/cgroup/cpu/t2/cpu.cfs_quota_us; echo 100000 > /sys/fs/cgroup/cpu/t2/cpu.cfs_period_us
python3 spin.py            # no limit
python3 spin.py t2         # inside the cgroup with the quota

echo "--- 5. image layers: a read-only lower layer and a writable upper layer"
mkdir -p ov/{lower,upper,work,merged}; echo base > ov/lower/a.txt
mount -t overlay overlay -o lowerdir=ov/lower,upperdir=ov/upper,workdir=ov/work ov/merged
echo changed > ov/merged/a.txt; echo new > ov/merged/b.txt
echo "lower a.txt: $(cat ov/lower/a.txt); upper layer holds: $(ls ov/upper | tr '\n' ' '); merged view of a.txt: $(cat ov/merged/a.txt)"
umount ov/merged; rmdir /sys/fs/cgroup/memory/t1 /sys/fs/cgroup/cpu/t2

hog.py: allocate memory until the limit kills the process

import os, sys, time
open("/sys/fs/cgroup/memory/t1/cgroup.procs","w").write(str(os.getpid()))
blocks = []
try:
    for i in range(40):
        blocks.append(bytearray(10*1024*1024)); 
        for j in range(0, len(blocks[-1]), 4096): blocks[-1][j] = 1
        print(f"allocated {10*(i+1)} MB", flush=True)
except MemoryError: print("MemoryError")

spin.py: measure the CPU a process really gets

import os, sys, time
if len(sys.argv) > 1: open(f"/sys/fs/cgroup/cpu/{sys.argv[1]}/cgroup.procs","w").write(str(os.getpid()))
n = 0; t = time.perf_counter(); user0 = os.times()
while time.perf_counter() - t < 3: n += 1
print(f"{sys.argv[1] if len(sys.argv)>1 else 'no limit':>8}: {n/3/1e6:6.2f} million loop iterations per second, cpu time used {os.times().user - user0.user:.2f} s of 3 s wall")

Scheduling and rolling-update simulations

import random

# ---- 1. scheduling: pods with CPU requests onto nodes of 8 CPUs
def pack(requests, capacity, policy, rnd):
    nodes = []                                      # remaining capacity per node
    for r in requests:
        fits = [i for i, free in enumerate(nodes) if free >= r]
        if policy == "first-fit": i = fits[0] if fits else None
        elif policy == "spread":  i = max(fits, key=lambda i: nodes[i]) if fits else None   # least loaded node first (the default Kubernetes scoring leans this way)
        elif policy == "best-fit": i = min(fits, key=lambda i: nodes[i]) if fits else None
        if i is None: nodes.append(capacity); i = len(nodes) - 1
        nodes[i] -= r
    used = sum(requests) / (len(nodes) * capacity)
    return len(nodes), used
rnd = random.Random(4)
reqs = [rnd.choice([0.25, 0.5, 1, 1.5, 2, 3]) for _ in range(200)]
print(f"200 pods requesting {sum(reqs):.0f} CPUs in total on 8-CPU nodes (never fewer than {sum(reqs)/8:.1f} nodes):")
for policy in ("first-fit", "best-fit", "spread"):
    n, u = pack(reqs, 8, policy, rnd); print(f"  {policy:>9}: {n} nodes, {u*100:.0f}% of requested CPU filled")
n, u = pack(sorted(reqs, reverse=True), 8, "best-fit", rnd); print(f"  best-fit, largest pods first: {n} nodes, {u*100:.0f}%")
# requests vs real use
use = [r * rnd.uniform(0.1, 0.6) for r in reqs]
print(f"the same pods actually use {sum(use):.0f} CPUs ({sum(use)/sum(reqs)*100:.0f}% of what they requested): requests decide placement, not use")

# ---- 2. rolling update of N replicas, following the Deployment controller's rules:
# total pods <= N + maxSurge, and ready pods >= N - maxUnavailable at every moment
def rolling(n=4, surge=1, unavailable=0, ready_after=2, new_works=True, max_ticks=60):
    pods = [{"v": "v1", "age": 99, "ready": True} for _ in range(n)]; min_ready = n; t = 0
    while t < max_ticks:
        t += 1
        for p in pods:
            if p["v"] == "v2" and not p["ready"]:
                p["age"] += 1; p["ready"] = new_works and p["age"] >= ready_after
        # scale up: create new pods while the total stays within N + maxSurge
        while len(pods) < n + surge and sum(p["v"] == "v2" for p in pods) < n: pods.append({"v": "v2", "age": 0, "ready": False})
        # scale down: delete old pods while the ready count stays at or above N - maxUnavailable
        for p in [p for p in pods if p["v"] == "v1"]:
            ready = sum(q["ready"] for q in pods)
            if ready - (1 if p["ready"] else 0) >= n - unavailable: pods.remove(p)
        min_ready = min(min_ready, sum(q["ready"] for q in pods))
        if len(pods) == n and all(p["v"] == "v2" and p["ready"] for p in pods): return dict(done=True, ticks=t, min_ready=min_ready, old=0, stuck_new=0)
    return dict(done=False, ticks=t, min_ready=min_ready, old=sum(p["v"] == "v1" for p in pods), stuck_new=sum(p["v"] == "v2" and not p["ready"] for p in pods))
for name, kw in (("maxSurge 1, maxUnavailable 0", dict(surge=1, unavailable=0)), ("maxSurge 2, maxUnavailable 1", dict(surge=2, unavailable=1)), ("maxSurge 0, maxUnavailable 1", dict(surge=0, unavailable=1)), ("maxSurge 0, maxUnavailable 4 (recreate-like)", dict(surge=0, unavailable=4))):
    r = rolling(**kw); print(f"rolling update of 4 pods, {name}: finished={r['done']} in {r['ticks']} ticks, never fewer than {r['min_ready']} ready pods")
for name, kw in (("maxSurge 1, maxUnavailable 0", dict(surge=1, unavailable=0)), ("maxSurge 1, maxUnavailable 1", dict(surge=1, unavailable=1))):
    r = rolling(new_works=False, **kw); print(f"new version never becomes ready, {name}: finished={r['done']}, after {r['ticks']} ticks {r['old']} old pods still serving, {r['stuck_new']} new pods not ready, never fewer than {r['min_ready']} ready pods")
# ---- 3. health checks: liveness vs readiness
print("readiness removes a pod from the load balancer without restarting it; liveness restarts the container; a slow-starting app needs a startup probe")

CPU requests and limits on real cgroups: shares, quota and throttled latency

Python 3, run as root on Linux with cgroups v1 (like the shell script above). It needs the small helper below, saved as cg.py; it creates and removes its own groups.

"""CPU requests and limits on real cgroups (v1, run as root). Requests become cpu.shares: a relative weight that matters only when the CPU is contended.
Limits become a quota (cfs_quota_us per cfs_period_us): a hard cap that applies even when the CPU is idle, and that shows up as latency."""
import multiprocessing as mp, random, subprocess, sys, time
from cg import Group

def spinner(group):
    p = subprocess.Popen([sys.executable, "-c", "while True: pass"]); group.add(p.pid); return p

def measure(groups_procs, seconds=4):
    t0 = {g.name: g.cpu_seconds() for g, _ in groups_procs}; time.sleep(seconds)
    out = {g.name: (g.cpu_seconds() - t0[g.name]) / seconds for g, _ in groups_procs}
    for _, p in groups_procs: p.kill(); p.wait()
    return out

def part1():
    print("1. Requests (shares) and limits (quota) with spinning processes")
    a = Group("expA", cpu_shares=1024, cpuset="0"); b = Group("expB", cpu_shares=256, cpuset="0")
    res = measure([(a, spinner(a)), (b, spinner(b))])
    print(f"   two processes on one CPU, shares 1024 and 256 (a request ratio of 4:1): {res['expA']*100:3.0f}% and {res['expB']*100:3.0f}% of the CPU")
    res = measure([(b, spinner(b))]); print(f"   the process with the small share, alone on the CPU:                    {res['expB']*100:3.0f}%   (a request is not a cap)")
    a.remove(); b.remove()
    c = Group("expC", quota_us=20_000, cpuset="0"); res = measure([(c, spinner(c))])
    print(f"   one process alone, limit 0.2 CPU (20 ms per 100 ms), the CPU otherwise idle: {res['expC']*100:3.0f}%   (a limit is a cap, even when idle)")
    n, th, t = c.throttle(); print(f"      throttled in {th} of {n} periods ({100*th/n:.0f}%), {t:.1f} s paused in total"); c.remove()

def worker(conn, ms):
    while True:
        if conn.recv() is None: return
        t0 = time.thread_time()
        while time.thread_time() - t0 < ms / 1000: pass
        conn.send(1)

def bursty(group, seconds=20, bursts_per_s=3, per_burst=3, ms=25, seed=1):
    """A service with two worker processes; a request is 25 ms of CPU in each worker (50 ms CPU, 25 ms of time on an idle two-core machine).
    Page loads arrive as bursts of 3 requests; average load: 3 bursts/s x 3 requests x 50 ms = 0.45 CPU."""
    rnd = random.Random(seed); pipes = []; procs = []
    for _ in range(2):
        parent, child = mp.Pipe(); p = mp.Process(target=worker, args=(child, ms)); p.start(); pipes.append(parent); procs.append(p)
    if group:
        for p in procs: group.add(p.pid)
    arrivals = []; t = 0.0
    while t < seconds: t += rnd.expovariate(bursts_per_s); arrivals += [t] * per_burst
    cpu0 = group.cpu_seconds() if group else 0; n0 = group.throttle() if group else (0, 0, 0)
    start = time.perf_counter(); lat = []
    for at in arrivals:
        now = time.perf_counter() - start
        if at > now: time.sleep(at - now)
        for c in pipes: c.send(1)
        for c in pipes: c.recv()
        lat.append((time.perf_counter() - start - at) * 1000)
    wall = time.perf_counter() - start
    for c in pipes: c.send(None)
    for p in procs: p.join()
    used = ((group.cpu_seconds() - cpu0) / wall) if group else float("nan"); n1 = group.throttle() if group else (0, 0, 0)
    lat.sort(); q = lambda x: lat[min(len(lat) - 1, int(len(lat) * x))]
    return dict(used=used, p50=q(.5), p99=q(.99), worst=lat[-1], periods=n1[0] - n0[0], throttled=n1[1] - n0[1], paused=n1[2] - n0[2])

def part2():
    print("\n2. The same bursty service under different CPU limits (average use about 0.45 of a CPU; two cores in this machine)")
    print(f"   {'limit':<22}{'CPU used':>9}{'p50':>9}{'p99':>9}{'worst':>9}   periods throttled")
    for label, quota in (("none", None), ("2.0 CPUs", 200_000), ("1.0 CPU", 100_000), ("0.6 CPU", 60_000), ("0.5 CPU", 50_000)):
        g = Group("expL", quota_us=quota); r = bursty(g); g.remove()
        thr = f"{r['throttled']}/{r['periods']} ({100*r['throttled']/max(1,r['periods']):.0f}%), {r['paused']:.1f} s paused" if quota and r["periods"] else "-"
        print(f"   {label:<22}{r['used']:>8.2f} {r['p50']:>7.0f} ms{r['p99']:>7.0f} ms{r['worst']:>7.0f} ms   {thr}")

if __name__ == "__main__":
    part1(); part2()

The cgroup helper used by the CPU and memory scripts (cg.py)

"""Tiny helpers for cgroups v1 (run as root). Each experiment creates its own groups under /sys/fs/cgroup/<controller>/<name> and removes them afterwards."""
import os, time
R = "/sys/fs/cgroup"
def w(path, value): open(path, "w").write(str(value))
def r(path): return open(path).read().strip()

class Group:
    def __init__(self, name, cpu_shares=None, quota_us=None, period_us=100_000, mem_limit=None, cpuset=None):
        self.name = name
        for c in ("cpu", "cpuacct", "memory", "cpuset"):
            os.makedirs(f"{R}/{c}/{name}", exist_ok=True)
        w(f"{R}/cpuset/{name}/cpuset.mems", r(f"{R}/cpuset/cpuset.mems"))
        w(f"{R}/cpuset/{name}/cpuset.cpus", cpuset if cpuset is not None else r(f"{R}/cpuset/cpuset.cpus"))
        if cpu_shares: w(f"{R}/cpu/{name}/cpu.shares", cpu_shares)
        w(f"{R}/cpu/{name}/cpu.cfs_period_us", period_us)
        w(f"{R}/cpu/{name}/cpu.cfs_quota_us", quota_us if quota_us else -1)
        if mem_limit: w(f"{R}/memory/{name}/memory.limit_in_bytes", mem_limit)
    def add(self, pid):
        for c in ("cpu", "cpuacct", "memory", "cpuset"): w(f"{R}/{c}/{self.name}/cgroup.procs", pid)
    def cpu_seconds(self): return int(r(f"{R}/cpuacct/{self.name}/cpuacct.usage")) / 1e9
    def throttle(self):
        d = dict(l.split() for l in open(f"{R}/cpu/{self.name}/cpu.stat")); return int(d["nr_periods"]), int(d["nr_throttled"]), int(d["throttled_time"]) / 1e9
    def mem(self, f): return int(r(f"{R}/memory/{self.name}/{f}"))
    def stat(self): return {k: int(v) for k, v in (l.split() for l in open(f"{R}/memory/{self.name}/memory.stat"))}
    def remove(self):
        for c in ("cpu", "cpuacct", "memory", "cpuset"):
            try: os.rmdir(f"{R}/{c}/{self.name}")
            except OSError: pass

Memory limits: a limit set from the average, and a page-cache test that did not show an effect here

Same requirements. The second test is included for completeness: in my sandbox it showed no difference between the two limits, so its result is not used in the text.

"""Memory limits on real cgroups (v1, run as root): a limit set from the average use, and page-cache pressure below the working set."""
import os, subprocess, sys, time
from cg import Group, w, r

MB = 1 << 20
SPIKY = r'''
import os, sys, time
open("/sys/fs/cgroup/memory/%s/cgroup.procs" % sys.argv[1], "w").write(str(os.getpid()))
base = bytearray(40 * 1024 * 1024)
for i in range(0, len(base), 4096): base[i] = 1                # a steady 40 MB working set
for n in range(6):
    time.sleep(0.4)
    spike = bytearray(50 * 1024 * 1024)                        # a 50 MB peak (a big request, a report, a batch) held for 0.2 s
    for i in range(0, len(spike), 4096): spike[i] = 1
    print("peak", n + 1, "survived", flush=True); time.sleep(0.2); del spike
'''

def part1():
    print("1. A memory limit set from the average, and one set above the peak (40 MB steady, a 50 MB peak for 0.2 s every 0.6 s; average use about 57 MB)")
    for label, limit in (("limit 64 MB (close to the average)", 64 * MB), ("limit 128 MB (above the peak)", 128 * MB)):
        g = Group("expM", mem_limit=limit)
        p = subprocess.run([sys.executable, "-c", SPIKY, "expM"], capture_output=True, text=True)
        peaks = p.stdout.count("survived")
        print(f"   {label:<36} exit status {p.returncode:>4}{' (128 + 9: killed with SIGKILL, OOMKilled in Kubernetes)' if p.returncode == -9 else '':<60} peaks survived {peaks}/6, highest use {g.mem('memory.max_usage_in_bytes')/MB:.0f} MB, limit hits {g.mem('memory.failcnt')}")
        g.remove()

def part2(file_mb=300, passes=3):
    print(f"\n2. Page cache counts against the limit: reading a {file_mb} MB file {passes} times in a container")
    path = "/var/tmp/cachefile.bin"
    if not os.path.exists(path) or os.path.getsize(path) != file_mb * MB:
        with open(path, "wb") as f:
            for _ in range(file_mb): f.write(os.urandom(MB))
    reader = r'''
import os, sys, time
open("/sys/fs/cgroup/memory/%s/cgroup.procs" % sys.argv[1], "w").write(str(os.getpid()))
for n in range(int(sys.argv[3])):
    t = time.perf_counter(); size = 0
    with open(sys.argv[2], "rb", buffering=0) as f:
        while chunk := f.read(1 << 20): size += len(chunk)
    print(f"{size / (1 << 20) / (time.perf_counter() - t):.0f}", flush=True)
'''
    for label, limit in (("limit 100 MB (less than the file)", 100 * MB), ("limit 600 MB (more than the file)", 600 * MB)):
        w("/proc/sys/vm/drop_caches", 3)
        g = Group("expP", mem_limit=limit)
        p = subprocess.run([sys.executable, "-c", reader, "expP", path, str(passes)], capture_output=True, text=True)
        st = g.stat()
        print(f"   {label:<36} MB/s per pass: {' -> '.join(p.stdout.split())}; cache kept at the end {st['cache']/MB:.0f} MB, major page faults {st.get('total_pgmajfault', 0)}, limit hits {g.mem('memory.failcnt')}")
        g.remove()

if __name__ == "__main__":
    part1(); part2()

Rollouts that stall: pending surge pods and readiness that never passes

Python 3, standard library only.

"""Why a rollout stalls: a Deployment update with and without spare capacity, and with a new version that never becomes ready.
The same rules as the rolling-update model above, plus a cluster that can only schedule as many pods as their requests allow."""
def rollout(replicas, capacity, surge, unavailable, new_works=True, ready_after=2, max_ticks=80):
    """capacity = how many of these pods the cluster can schedule at once (free CPU by request / the pod's request)."""
    pods = [dict(v="v1", state="ready", age=0) for _ in range(replicas)]
    for t in range(1, max_ticks + 1):
        for p in pods:                                                              # new pods become ready after their start-up (if the new version works)
            if p["v"] == "v2" and p["state"] == "starting":
                p["age"] += 1
                if new_works and p["age"] >= ready_after: p["state"] = "ready"
        scheduled = sum(p["state"] != "pending" for p in pods)
        for p in pods:                                                              # the scheduler places a Pending pod only if requests fit
            if p["state"] == "pending" and scheduled < capacity: p["state"] = "starting"; scheduled += 1
        while len(pods) < replicas + surge and sum(p["v"] == "v2" for p in pods) < replicas:
            pods.append(dict(v="v2", state="starting" if scheduled < capacity else "pending", age=0)); scheduled += pods[-1]["state"] != "pending"
        for p in [p for p in pods if p["v"] == "v1"]:                               # the controller removes an old pod only if enough pods stay ready
            ready = sum(q["state"] == "ready" for q in pods)
            if ready - 1 >= replicas - unavailable: pods.remove(p)
        if len(pods) == replicas and all(p["v"] == "v2" and p["state"] == "ready" for p in pods):
            return f"finished in {t} ticks"
    c = lambda v, s: sum(p["v"] == v and p["state"] == s for p in pods)
    return f"STALLED after {max_ticks} ticks: {c('v1','ready')} old pods ready, {c('v2','ready')} new ready, {c('v2','starting')} new starting but never ready, {c('v2','pending')} new Pending"

if __name__ == "__main__":
    print("16 replicas of a pod requesting 1 CPU; maxSurge 4 (25%)")
    cases = [
        ("cluster of 16 CPUs (100% requested), maxUnavailable 0, healthy new version", dict(capacity=16, surge=4, unavailable=0)),
        ("same cluster, maxUnavailable 4", dict(capacity=16, surge=4, unavailable=4)),
        ("same cluster plus one more node (24 CPUs), maxUnavailable 0", dict(capacity=24, surge=4, unavailable=0)),
        ("same cluster with requests lowered so that 20 pods fit, maxUnavailable 0", dict(capacity=20, surge=4, unavailable=0)),
        ("24 CPUs, maxUnavailable 0, but the new version never passes its readiness probe", dict(capacity=24, surge=4, unavailable=0, new_works=False)),
        ("24 CPUs, maxUnavailable 4, the new version never passes its readiness probe", dict(capacity=24, surge=4, unavailable=4, new_works=False)),
    ]
    for label, kw in cases: print(f"  {label}:\n     {rollout(16, **kw)}")

Sources

Practise this topic: DevOps & Cloud

More guides