Large Docker Images in Kubernetes: 7 Proven Fixes, Tested

Large Docker images slow Kubernetes down in two places: the first pull on every node, and the queue when several big images are pulled at once. The first version of this article listed the usual advice (smaller base images, multi-stage builds, parallel pulls, pre-pull DaemonSets, image garbage collection) but several of its examples do not work as written: the Go multi-stage build fails without a go.mod, the "preload" DaemonSet runs the whole application on every node, and the StorageClass example has nothing to do with where images are stored. This rewrite tests each idea on a real cluster and keeps only what the output supports.

Lab note: one Debian 12 KVM VM (4 vCPU, 4 GB), k3s v1.37.2+k3s1 with containerd v2.3.7-k3s1, images built with podman 4.3.1 and pushed to a local registry:2 over HTTP. Tested on 7 November 2026. Pulls from a local registry are much faster than from the internet, so look at the differences between runs, not at the absolute seconds.

Contents

• Step 1: make large Docker images smaller first
• Step 2: clean package caches in the same layer
• Step 3: how Kubernetes pulls large Docker images
• Step 4: serial vs parallel pulls (serializeImagePulls)
• Step 5: limit parallel pulls the right way on k3s
• Step 6: pre-pull large Docker images with a DaemonSet that does not run the app
• Step 7: disk space and image garbage collection
• Pull errors you will meet
• Conclusion

Step 1: make large Docker images smaller first

The cheapest pull is the one you do not need, and many large Docker images are large only by accident. The old post's multi-stage Dockerfile looks fine, but it copies a Go program without a module file, and current Go refuses to build it:

FROM golang:alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o main .
FROM alpine
WORKDIR /app
COPY --from=builder /app/main .
CMD ["./main"]

go: go.mod file not found in current directory or any parent directory; see 'go help modules'
Error: building at STEP "RUN go build -o main .": while running runtime: exit status 1
[exit 1]

(podman also needs the full name docker.io/library/golang:alpine; the short name failed with "short-name did not resolve to an alias" on Debian's default registries.conf.) With a two-line go.mod added, I built the same small HTTP server three ways:

$ podman images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | grep -E "app:|golang|alpine" | sort
docker.io/library/alpine:latest 8.73 MB
docker.io/library/golang:alpine 266 MB
registry.lab:5000/app:alpine 17.3 MB
registry.lab:5000/app:scratch 5.85 MB
registry.lab:5000/app:single 362 MB

The single-stage image is a typical example of large Docker images: it ships the whole Go toolchain plus the build cache: 362 MB for a 5 MB program. podman history shows where it comes from:

95.8 MB /bin/sh -c go build -o main .
257 MB COPY /target/ / # buildkit
8.71 MB ADD alpine-minirootfs-3.24.4-x86_64.tar.gz...

Multi-stage onto Alpine gives 17.3 MB, and a static binary on scratch gives 5.85 MB. The scratch version used CGO_ENABLED=0 go build -ldflags="-s -w" and USER 65534. The trade-off is real: there is no shell to debug with.

$ podman run --rm --entrypoint /bin/sh registry.lab:5000/app:scratch -c true
Error: crun: executable file `/bin/sh` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Use kubectl debug with an ephemeral container for such images, or stay on Alpine/distroless if your team needs a shell in production.

Step 2: clean package caches in the same layer

The old cleanup example was RUN apt-get update && apt-get install -y \ && rm -rf /var/lib/apt/lists/*. As written it installs nothing at all:

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

With a real package (curl), the order of the RUN lines decides the size, because a later layer cannot shrink an earlier one:

localhost/apt-one 90.1 MB     # update + install + rm in ONE RUN
localhost/apt-split 110 MB    # same three commands in three RUN lines
docker.io/library/debian 78 MB

20 MB of apt lists stay in the split image even though the last layer deletes them. Package caches are a common reason for large Docker images; the same rule applies to pip (--no-cache-dir), npm and dnf clean all.

Step 3: how Kubernetes pulls large Docker images

To make pulls of large Docker images visible I built three 600 MB images with random data (incompressible) and started pods on k3s. The kubelet writes the pull time and the time "including waiting" into the pod events:

-- small
Pulled      Successfully pulled image "registry.lab:5000/app:scratch" in 69ms (69ms including waiting). Image size: 2591497 bytes.
-- large
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 2.352s (2.368s including waiting). Image size: 633100730 bytes.

Read kubectl get events or kubectl describe pod when a rollout is slow: if "including waiting" is much larger than the pull time, the pod sat in the node's pull queue, not on the network.

imagePullPolicy matters on every restart. With IfNotPresent the second pod did not contact the registry at all. With Always it asked the registry again (13 ms here, because the layers were already cached). And a :latest tag without an explicit policy gets Always:

-- large2 (IfNotPresent)
Pulled      Container image "registry.lab:5000/big1:v1" already present on machine and can be accessed by the pod
-- large3 (Always)
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 13ms (13ms including waiting). Image size: 633100730 bytes.

$ kubectl get pod latest1 -o jsonpath='{.spec.containers[0].imagePullPolicy}'
Always

Per the Kubernetes images docs, Always resolves the tag to a digest and downloads only missing layers, so it is cheap when the registry is close and up, and a hard dependency when it is not. For large Docker images, pin a version tag and use IfNotPresent.

Step 4: serial vs parallel pulls (serializeImagePulls)

The kubelet configuration reference says serializeImagePulls "tells the Kubelet to pull images one at a time" with "Default: true". k3s does not use that default. Fresh install, kubelet configz:

{
  "serializeImagePulls": false,
  "maxParallelImagePulls": null,
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "imageMinimumGCAge": "2m0s",
  "imageMaximumGCAge": "0s"
}

So on k3s parallel pulls are already on. Check configz on your own distribution before changing anything: kubectl get --raw /api/v1/nodes/<node>/proxy/configz. Three 600 MB images requested at once with the k3s default:

-- a1
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 4.738s (4.739s including waiting). Image size: 633100728 bytes.
-- a2
Pulled      Successfully pulled image "registry.lab:5000/big2:v1" in 4.511s (4.511s including waiting). Image size: 633100728 bytes.
-- a3
Pulled      Successfully pulled image "registry.lab:5000/big3:v1" in 4.528s (4.528s including waiting). Image size: 633100733 bytes.

Then the upstream default, set with a k3s kubelet-arg in /etc/rancher/k3s/config.yaml and a restart:

kubelet-arg:
  - "serialize-image-pulls=true"

-- s1
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 2.123s (17.837s including waiting). Image size: 633100728 bytes.
-- s2
Pulled      Successfully pulled image "registry.lab:5000/big2:v1" in 131ms (15.724s including waiting). Image size: 633100728 bytes.
-- s3
Pulled      Successfully pulled image "registry.lab:5000/big3:v1" in 2.109s (15.597s including waiting). Image size: 633100733 bytes.

Real kubelet events from the lab: serial vs parallel pulls of three 600 MB images.

Each pull alone took about 2 seconds, but every pod waited 15 to 18 seconds in the queue; in parallel all three were ready after about 4.5 seconds. On a cluster that pulls large Docker images from the internet, the gap grows with image size. The old post claimed parallel pulling needs "version 1.19 and beyond"; I could not verify that version in the docs and dropped it.

Step 5: limit parallel pulls the right way on k3s

Unlimited parallel pulls of large Docker images can saturate a node's network or disk. maxParallelImagePulls caps them; the reference says it "cannot be set if SerializeImagePulls is true". The obvious k3s way, another kubelet-arg, broke the node:

kubelet-arg:
  - "serialize-image-pulls=false"
  - "max-parallel-image-pulls=2"

$ systemctl is-active k3s
activating
Error: failed to parse kubelet flag: unknown flag: --max-parallel-image-pulls

There is no such command line flag, and k3s kept restarting the kubelet in a loop. Test kubelet changes on one node first. The setting works as a KubeletConfiguration file loaded through the kubelet's config-dir:

# /etc/rancher/k3s/config.yaml
kubelet-arg:
  - "config-dir=/etc/rancher/k3s/kubelet.conf.d"

# /etc/rancher/k3s/kubelet.conf.d/50-pulls.conf
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serializeImagePulls: false
maxParallelImagePulls: 2

active
{"serializeImagePulls":false,"maxParallelImagePulls":2}
-- m1
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 2.623s (6.747s including waiting). Image size: 633100728 bytes.
-- m2
Pulled      Successfully pulled image "registry.lab:5000/big2:v1" in 4.154s (4.154s including waiting). Image size: 633100728 bytes.
-- m3
Pulled      Successfully pulled image "registry.lab:5000/big3:v1" in 4.132s (4.132s including waiting). Image size: 633100733 bytes.

Two pulls ran together, the third waited for a free slot, exactly as the images docs describe for maxParallelImagePulls.

Step 6: pre-pull large Docker images with a DaemonSet that does not run the app

The old "preload" DaemonSet puts the large image in the main container. Applied as written, it does pull the image, but it also runs the image's default command on every node, forever:

NAME                   READY   STATUS    RESTARTS   AGE
preload-images-c5jsc   1/1     Running   0          40s

$ kubectl exec ds/preload-images -- ps -o pid,comm
PID   COMMAND
    1 sleep

For a real application image that means a second copy of your service on every node. If the image's command ends quickly (a CLI or batch image), the DaemonSet restarts it again and again:

NAME                   READY   STATUS      RESTARTS      AGE
preload-images-fxlt2   0/1     Completed   3 (38s ago)   50s

The working pattern: pull in an initContainer that exits immediately, and keep the pod alive with the tiny pause image:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prepull-big
spec:
  selector:
    matchLabels:
      app: prepull-big
  template:
    metadata:
      labels:
        app: prepull-big
    spec:
      initContainers:
      - name: pull-big1
        image: registry.lab:5000/big1:v1
        imagePullPolicy: IfNotPresent
        command: ["true"]
        resources:
          requests: {cpu: 1m, memory: 8Mi}
      containers:
      - name: pause
        image: registry.k8s.io/pause:3.10
        resources:
          requests: {cpu: 1m, memory: 8Mi}

prepull-big-wf86d   1/1     Running   0          14s
Pulled      Successfully pulled image "registry.lab:5000/big1:v1" in 1.872s (1.872s including waiting). Image size: 633100728 bytes.
-- after (app pod scheduled after the pre-pull)
Pulled      Container image "registry.lab:5000/big1:v1" already present on machine and can be accessed by the pod

The init container needs a true binary in the image (Alpine and Debian based images have it; scratch images do not, so pick a command that exits at once). Update the DaemonSet's image tag in the same pipeline that deploys the app, or the nodes cache the old version.

Step 7: disk space and image garbage collection

The old post suggested a StorageClass for "high throughput" image storage. Images are not stored in PersistentVolumes; containerd keeps them on the node's own disk. After the tests:

$ du -sh /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
2.0G	/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
2.4G	/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs

A 633 MB image costs about twice that: compressed layers in the content store plus unpacked snapshots. Size node disks for that before you schedule many large Docker images on one node. The garbage collection values from the old post (85/80) are simply the defaults, as the configz output above shows. Per the garbage collection docs, disk usage above imageGCHighThresholdPercent triggers deletion of unused images, oldest first, until usage reaches imageGCLowThresholdPercent; imageMaximumGCAge (default 0s, disabled) can also remove images unused for longer than a set time. To clean a node by hand:

$ k3s crictl images | grep -c registry.lab; k3s crictl rmi --prune 2>&1 | grep -c Deleted
3
5

--prune removed every image not used by a running container, including the pre-pulled ones once their DaemonSet was gone. docker system prune -a from the old post only cleans a Docker host; Kubernetes nodes with containerd do not use the Docker daemon.

Pull errors you will meet

-- badtag
Failed      Failed to pull image "registry.lab:5000/big1:v9": rpc error: code = NotFound desc = failed to pull and unpack image "registry.lab:5000/big1:v9": failed to res
Failed      Error: ErrImagePull
BackOff     Back-off pulling image "registry.lab:5000/big1:v9"
Failed      Error: ImagePullBackOff
-- badreg
Failed      Failed to pull image "registry.nope:5000/big1:v1": failed to pull and unpack image "registry.nope:5000/big1:v1": failed to resolve reference "registry.nope:50

NotFound means the registry answered but the tag does not exist; "failed to resolve reference" without a code points at DNS or the network. Both end in ImagePullBackOff, so read the first Failed event, not the status column. For moving images between hosts without a registry, see how to copy Docker images from one host to another; for a cluster to try this on, see Kubernetes installation on Ubuntu.

Conclusion

For large Docker images in Kubernetes, the order of work is: shrink the image (multi-stage, one-layer cleanup), pin tags and use IfNotPresent, check whether your nodes pull serially (configz), cap parallel pulls through a KubeletConfiguration file rather than an unknown flag, pre-pull with an init container DaemonSet that does not run the app, and give nodes enough disk for compressed plus unpacked layers. Each step is visible in the kubelet's own events, so measure before and after.