Advice about Docker build caching tends to arrive as folklore: put the COPY late, enable some cache action, hope. The trouble is that "Docker caching" is not one cache: it's at least three, with different lifetimes, different invalidation rules, and different failure modes. Which of them matters, and whether the folklore helps, does nothing, or actively hurts, depends on the shape of the build in front of you.
There are too many build shapes to cover one by one: every stack, monorepo layout, and orchestration has its own wrinkle. So this post aims for a different kind of completeness: the concepts and rules of thumb that decide the outcome in any scenario, each illustrated with a measurement from a benchmark lab we built. The lab spans 18 scenarios across Go, Python, Rust, Node, Java, and Bazel, run on real CI runners, comparing every commonly used caching backend. Where our own product wins, I'll show it; where something is genuinely missing, ours included, I'll say so.
The concepts here stand on their own, so no Dockerfile expertise is required to follow along. When the examples reference specific instructions (FROM, COPY, RUN, multi-stage builds), Docker's Dockerfile overview and building best practices are good prose introductions to lean on.
A Docker build is a chain of instructions, each producing a layer keyed by the instruction and the content that feeds it. A change invalidates its layer and every layer after it, a wave that propagates to the end of the chain. That global reach is what makes the wave worth optimizing for: how far it travels decides how much of the build re-runs, on every single change.
(Strictly, the build graph is a DAG: with multi-stage builds each stage is its own chain, joined by COPY --from edges, and the wave only reaches the stages that depend on the change. But within a stage it's always a chain, most Dockerfiles are effectively a single chain, and that's what the examples here show; everything said about the wave applies per-chain.)
The chain is only the first cache. Two more sit beside it:
RUN --mount=type=cache): directories that persist across builds. They don't stop the wave, they cushion it: when the wave re-runs an instruction, the instruction finds its mount still warm, so a re-run isn't a from-scratch run. This is where package-manager and compiler state lives (GOCACHE, cargo's registry and target/, pip's wheels, pnpm's store)./var/lib/docker): pulled base images, built images, everything the daemon itself caches. It touches the chain at exactly one point, the FROM layer: when a base image changes, the wave starts at the very top, and whether the new base is a fast local read or a slow registry pull depends on this store. The wave never invalidates anything in it. And it lives well beyond builds: docker run, service containers, and image distribution are all reads against this store.A question that will come back for each of these caches, in every setup below: where does it live, and does it survive to the next CI job? Everything that follows is a zoom-in on some part of this picture.
One more piece of shared vocabulary: the change classes from the diagram above are also how every measurement in this post is structured. Each scenario builds four times, and each build probes one class:
Rule of thumb #1: think in change classes, not in builds. Your median CI build is a source-change, not a cold build. A cache setup that looks great on the "second build of the same commit" demo and terrible on a one-line code edit is optimizing the phase you hit least.
The general guideline: figure out which of your steps are expensive and stable, and order the Dockerfile so that frequently changing content sits below them. The most common instance is manifest-first ordering: copy your dependency manifest first, install dependencies, and only then copy the source. The expensive install layer then invalidates only when the manifest changes, not on every commit.
We measured the same service both ways (COPY . . before the install vs manifest-first) across five stacks. Source-change build times:
Two things worth noticing. First, cold time and the final image are identical in both variants; this optimization is free. Second, the gap is exactly the cost of your dependency step: cheap npm ci gives 3x, Python native wheels give 24x, and a 1GiB model file copied in the wrong layer turns every source edit into ~23s of pure I/O (we measured that too).
Rule of thumb #2: the payoff of layer ordering equals the cost of the step it protects. Know your most expensive stable step, and make sure nothing that changes often sits above it.
Layer caching is all-or-nothing: when the wave hits the install step, the whole step re-runs. Mount caches are what make the re-run cheap: the package manager finds its downloads, the compiler finds its incremental state. For compiled languages this is routinely the difference between "re-run means recompile the world" and "re-run means recompile one file".
Here's the catch that explains a large fraction of "we enabled caching and it's still slow" complaints. The very thing that lets a cache mount survive the wave (being a directory on the builder's local disk, outside the layer filesystem) also puts it outside what cache exporters handle. A cache-to export (type=gha, type=registry, type=inline) is a bundle of layer blobs: it serializes each instruction's resulting layer so another builder can skip re-running it. Mount contents are not part of any layer, so they are never in the bundle.
The two mechanisms attack different parts of the problem. An exported layer cache works along the wave: it lets a fresh machine skip every step the wave didn't reach, and it's the only mechanism that travels between machines. What it cannot do is help the step the wave did reach; making that re-run cheap is the mount cache's job. So on a fresh runner with an exported cache, the surviving layers are warm but every --mount=type=cache directory starts empty: the one step you needed to be fast runs from scratch.
The Rust scenario makes it concrete: a dependency change costs 3.6s when the mounts survive (persistent builder state), 164s with the GHA cache backend, and 19s with no cache at all. Read that again: the export backend was 8x slower than doing nothing, because it paid to import and export layers that the change invalidated anyway, and the state that would have helped wasn't in the export.
Rule of thumb #3: know where your build's win actually lives. If it lives in mount caches (compiled languages, Bazel, anything incremental), a layer-export backend cannot deliver it, no matter how it's tuned.
With the model in place, the backends sort themselves into two families:
gha, registry, inline, plus workarounds like buildkit-cache-dance, which patches the mount-cache gap by copying --mount=type=cache contents out of the builder after the build, storing them in the Actions cache, and injecting them back before the next one. Portable, works on any runner; pays serialization both ways, drops mount state (except cache-dance, which pays extra to keep it).The matrix below is the four-phase sequence across backends, for four of the lab's stacks:
The pattern is consistent. On the phases that dominate real CI traffic (warm, source-change, dep-change), persistent state wins by 10–30x, mostly because of the mount caches. Export backends cluster in an uncomfortable middle: often slower than no cache on change phases, because import/export overhead scales with cache size while the hit rate scales with what the change didn't touch.
Being honest about the other direction:
Rule of thumb #4: export backends charge a transfer tax on every build; only use one when the hits are worth more than the tax. The import and export happen whether or not the build hits, so the math is simple: an export backend makes sense when the win lives in layers and the cache is small. When the win lives in mount caches, or builds run often enough that 20–90 seconds of import/export per job adds up, keep the cache in place between jobs instead.
Persistent state introduces a question exports never face: who shares the disk? The default temptation is one cache per repo. The right answer is one cache per image lineage: a set of builds that actually reuse each other's layers.
We measured both failure directions:
Rule of thumb #5: share the cache exactly as far as the layers are shared. Related images (a monorepo tree with a common base): one key. Unrelated images that happen to live together: one key each. Repo topology is a proxy, and often a wrong one.
The lifecycle question: does a long-lived cache rot? We ran 25-cycle churn tests (a realistic 70/30 mix of source edits and dependency bumps on the same key) and tracked what actually grows:
Layer records grow slowly and linearly (BuildKit's size-capped GC handles them); mount caches converge on the union of dependency versions the cache has ever seen, then go flat. Bounded inputs, bounded cache.
The honest part is what doesn't get cleaned up. To BuildKit, a mount cache is a single record: an active repo touches it every build, so it never goes idle, and nothing trims the dead content inside it. Whether garbage accumulates depends entirely on the toolchain using the mount: Go's build cache self-trims (entries unused ~5 days), while Cargo's target/, the pip/npm/pnpm caches, and Bazel's disk cache all grow until something external prunes them. No platform (ours included) currently gives you visibility inside a mount cache; today the honest answer is to know your toolchain's behavior and occasionally reset keys for Cargo-like ecosystems with heavy version churn.
Rule of thumb #6: caches grow with distinct dependency versions seen, not with build count.
Everything so far was builder state. The second half of Docker's disk life is the image store, /var/lib/docker, and in CI it gets rebuilt from nothing constantly: every job re-pulls the same base images, the same services: sidecars (that postgres container, pulled on every single run), the same toolchain images.
Blacksmith's answer here is a different product with deliberately different scoping: container caching gives the whole org one shared image-store disk. Every job's VM boots with /var/lib/docker mounted from a copy-on-write clone of it; commits happen only when a job's image set actually changes. Org-wide sharing is correct here for exactly the reason it was wrong for build caches: public base images are identical across repos, build state isn't. It's also zero configuration: no action, no workflow change.
The most interesting consequence is fan-out: the build-once-run-everywhere pattern where one job builds an image and N jobs run it. The traditional path is builder → registry push → N pulls, and it pays image-size-scaled serialization at every arrow. We measured it with a deliberately fat 3.34GB image and three parallel consumers:
Once the image is in the org cache, a consumer's "pull" is a manifest check (~0.9s vs ~129s, 140x), and running the image reads 23MB off the clone: 0.7% of its bytes, streamed lazily as the container touches them. The per-consumer cost stopped scaling with image size.
The honest limit: this kicks in from the second wave onward. The first consumers after a genuinely new image still pull from the registry; the cache commits after somebody has the image, and there's no primitive yet for "hand this just-built image directly to the jobs behind me" without the initial registry round-trip. Related trap: an image rebuilt from identical sources gets new layer digests (COPY records mtimes), which quietly defeats any store-level dedup; reproducible tags matter.
There's also a producer trap worth calling out, because it inverts the win. A shared image-store disk is built for consumers: jobs that pull a stable set of images. A job that builds a multi-GB, uniquely-tagged image with load: true (or --load) is the opposite: the image is exported into /var/lib/docker and re-read for the push, and the unique tag forces a fresh multi-GB commit of the shared disk on every run. The fix is to skip the daemon entirely: push: true pushes straight from the builder to the registry. (Inspect the pushed image with docker buildx imagetools inspect instead of a local docker inspect.)
Rule of thumb #7: distribution is a caching problem too, and the win isn't a closer registry, it's not serializing at all.
And the rules of thumb, in one screen:
None of this makes a badly ordered Dockerfile fast, and nothing makes a true first build cheap. But between those two poles, nearly all of the 10–100x that CI teams leave on the table comes down to these seven, and the three-cache picture that generates them.
.jpg)
