Second in a series on the homelab message broker and its substrate. The first post describes KMQ, a broker built from FIFOs and awk.
A private OCI registry and a pull‑through cache
Pulling a container image is an operation that almost never gets inspected. The tooling is good. CI is fast. The cluster is happy. Most days, that is enough. Most days is not all days.
A registry is a system that answers questions about artifacts: do you have this digest, give me this manifest, give me these blobs. The protocol is small and well specified. The implementations are mostly fine. What changes between deployments is the topology, not the protocol. Where does the artifact live, who is allowed to see it, who is allowed to write it, at which moment in time can it be said that this digest is the one to trust.
Two answering systems got set up. One private, behind a segment with no internet egress. One public, sitting in front of three external registries, caching on the way. The two answers belong to different questions and the architecture has to keep them separate.
Two registries
A private registry holds artifacts produced internally or kept deliberately. The trust model is internal: a known author signed the manifest, a known operator controls the storage, a known chain of custody exists.
A pull-through cache does not produce, it intermediates. Trust still belongs to the upstream. What the cache adds is locality, latency reduction, bandwidth amortization and a local copy of what was seen the last time the question was asked.
The two systems share the OCI distribution API on the wire. The two systems do not share semantics.
Both, on different machines, with traffic crossing a router that segments two VLANs. The private registry on a small bare-metal server inside the inner segment. The pull-through cache on the router itself, which is the only host with internet egress.
The constraint that decided the shape
The router runs OpenWrt on aarch64. A previous attempt to run a Go-based registry directly on the router failed. The binary started, opened a socket, never accepted a connection. Nothing in the logs. The behavior reproduced with two different Go-based registry servers. After enough time, silence becomes a signal.
That decided the split. The Go program goes inside, on a host with a normal kernel. The router runs C, which is fine on this hardware. nginx in front of three OCI upstreams is a job that fits a pull-through cache cleanly.
A constraint upstream of the design is a gift. It removes one branch of the search tree.
The private registry
The private side runs zot, a single-binary registry in Go that ships an OCI-conformant minimal build with no extensions. Distroless is not relevant on this host. The registry runs as a system service, supervised by the init system, with logs routed through the same syslog as everything else. The minimal build has no UI, no search, no Trivy database, no metrics server. It speaks the distribution protocol.
Configuration is a small JSON file at the canonical path. The HTTP listener binds to the inner-segment IP and a chosen port. No TLS, no auth. The trust boundary is the network. The segment is reachable only from machines under direct control, and inbound from the working laptop subnet is opened explicitly in the firewall for a single port.
Adding TLS to a private registry that lives inside a segment with integrity properties at the network layer is a different decision from adding TLS to a service crossing an arbitrary path. A step-ca instance is available for issuing certificates internally. The certificate step is queued. As of this writing the registry exists, responds 200 on /v2/, accepts pushes from the laptop and answers pulls from the same.
Two notes from the build:
The Linux init script needed command_user, command_background, pidfile and a start_pre hook that asserts ownership on the storage and log directories. The first start crashed because a stray root-owned log file from an earlier verify run blocked the registry user from opening the log for append. checkpath in the init script handles directories but not files inside them. A one-line chown fixed it. The next iteration of the init script will assert file modes too, with checkpath -f.
The default firewall on the server side runs nftables with a default-drop input chain. It accepted SSH and HTTPS but not the registry port. One line in the nft ruleset closed the loop.
The cache, and the awkward problem in the middle
The cache is where the architecture had to do real work.
The router has a custom build of nginx, compiled statically against musl, with the proxy_cache module included but without OpenSSL. That build exists because there is no straight path to compile mainline nginx against both OpenSSL and the proxy_cache stack on this OpenWrt configuration. This was verified earlier. The factory-shipped nginx has OpenSSL but not proxy_cache. So one binary can do TLS but not caching, the other can cache but not do TLS.
Three upstream registries to cache. Each upstream serves only HTTPS. There is no HTTP fallback for any of them, which is correct.
Why these three. Chainguard (cgr.dev) for hardened minimal images consumed by KMQ and other in-cluster workloads. Docker Hub for the long tail of base images: alpine, busybox, the language runtimes that nothing else publishes consistently. GHCR for upstream tooling, including zot itself. The list is not chosen for symmetry. Each upstream is in the path because something specific consumes from it.
The pragmatic resolution to the TLS-or-cache split is composition. Two nginx processes, two configurations, chained on loopback. The cache layer terminates HTTP from clients on the inner-segment IP. It proxies to a TLS-terminator nginx listening on a private high port on loopback. The terminator opens the HTTPS connection upstream, with SNI, with certificate verification against the OS CA bundle, with the correct Host header for each upstream. The body comes back through the terminator, into the cache layer and either gets stored or not, depending on the response code and the request path.
The router has 47 GB of free disk on a USB drive already in use as an Alpine package mirror. The cache reuses it.
Three things in this design needed to be reasoned about explicitly.
Cache key and authentication. Every request to a public OCI registry begins with a 401 and a WWW-Authenticate: Bearer header pointing to a token endpoint. The client fetches the token directly from the upstream and re-issues the original request with the bearer credential. That second request is what produces the 200 with the manifest or blob. If the cache key includes the Authorization header, every client with a different token sees a miss. If the cache key is just the request URI, every authenticated client shares the cached object.
For an anonymous pull-through, sharing is the desired behavior. The cache key is $request_uri, period. The Authorization header is forwarded to the terminator and then upstream, so the upstream still authorizes per-request. The cache layer does not authenticate, does not store credentials, does not know who the clients are. It stores artifacts by their URI, and the URI for a blob is its digest, which is a content address. Two clients pulling the same blob with different tokens get the same byte sequence. The cache is correct.
The 401 itself must never be cached. There is a hard-coded proxy_cache_valid 401 0 in every server block. The first time this was forgotten, the cache held a 401 with an upstream WWW-Authenticate frozen in it, and every subsequent client tried to authenticate against a stale realm.
Blobs versus manifests. OCI distinguishes two kinds of object on the path. A blob lives at /v2/<name>/blobs/sha256:<digest> and is immutable: the digest is its identity. A manifest lives at /v2/<name>/manifests/<reference>, and the reference may be a digest (immutable) or a tag (mutable). The cache treats them differently. Blobs cache for a year. Manifests cache for an hour. Tagged manifests will go stale and the upstream will report a new digest after the TTL. Digest-addressed objects are content. Content does not change.
Redirects. This is the surprise. cgr.dev and Docker Hub do not serve blobs from their own infrastructure. They redirect blob requests, with a 307, to a signed URL on a CDN. The client follows the redirect and downloads the blob from a third party. If the cache layer does not store the 307 itself, every blob request hits the upstream API again, gets another 307, the client re-downloads the blob from the CDN. If the cache layer follows the redirect internally and caches the body, the result is true blob caching. nginx vanilla does not follow upstream redirects internally for proxy_pass, and trying to make it do so cleanly is a fight worth avoiding.
The middle path is to cache the 307 with a short TTL aligned to the signed URL’s expiration. Clients hit the cache, receive the redirect from local storage, follow it directly to the CDN. The cache holds a sequence of small redirect responses, not the multi-megabyte blobs. The CDN does its own caching downstream of that. The savings are real but smaller than they would be with body caching.
GHCR does not redirect. It serves blobs from its own infrastructure. For GHCR, the cache layer stores the actual blob bytes. A 50 MB image pulled twice in a row drops from 16 seconds to 5 seconds on the second pull. cgr.dev and Docker Hub stay around 50% reduction because the redirect URL changes per request and the body trip is uncached. Three upstreams, three behaviors. The cache topology is uniform. The effective speedup is not.
Docker Hub issues large bearer tokens. The default client_header_buffer_size and large_client_header_buffers on the factory nginx on this router were sized for normal HTTP traffic, not for JWTs with broad scopes. The first crane pull against the cache returned 400 Request Header Or Cookie Too Large from the TLS terminator. Increasing the buffers in the terminator’s server blocks fixed it.
A small lesson sits there. Every careful decision about cache keys and TTLs and certificate verification can hold up, and a buffer sized for an older internet rejects the connection before any of it runs. Always measure with the actual upstream, not with a synthetic substitute.
Validating the chain
Once the plumbing was up, the same image pull ran twice from the laptop with crane against each of the three upstreams. crane handles the OCI auth dance internally. What gets exercised is a real-world pull, not a synthetic curl with a hand-crafted token.
The outputs:
- chainguard/static:latest. First pull 2.3s, second 1.3s. Manifests cache, blobs go via redirect.
- library/alpine:3.21 from Docker Hub. First pull 3.5s, second 1.7s. Same redirect pattern.
- project-zot/zot-minimal:v2.1.15 from GHCR. First pull 16s, second 5s. Real blob caching on disk.
A small POSIX shell script runs the full smoke test from any host: DNS resolution, /v2/ probes, manifest retrieval, two pulls and a speedup measurement. Output is one line per event in logfmt. Anything can pipe it through awk. CI can read the same lines a human reads. There is no ASCII art, no color codes, no separator characters that would survive a bad terminal. A test result is a fact. A fact is a key-value pair.
ts=2026-05-04T08:42:11Z host=client01 result=pass event=v2_probe target=cache_ghcr endpoint=cache-ghcr.home.arpa:8080 code=401
ts=2026-05-04T08:42:13Z host=client01 result=pass event=cache_eff target=cache_ghcr image=project-zot/zot-minimal:v2.1.15 ms1=16194 ms2=5160 speedup_pct=68
That is the entire surface area of the test. One line, one event, parseable forever.
What this is
The separation of questions materializes as separation of machines. Internal artifacts answer to one process, on one host, on one segment, with one trust model. External artifacts answer through a different process, on a different host, with a different trust model. The architecture does not let them be confused in the code because they are not in the same code. That property is harder to add later than to keep from the start.
This is homelab hardware running at homelab budget. The criteria applied to it are not. Default-drop firewalls, segmented networks, content-addressed storage, structured logs, configuration in a git repository even when the deployment is not yet automated.
What is left
Nothing here is finished. The private registry has no TLS yet. The cache has no metrics export. The smoke test runs by hand. Configuration sits in a git repository, but the deployment is not yet driven from it. There is no Ansible role, no CI pipeline that applies the config to the router, no rollback path that does not involve someone typing on a console. All of that is queued.
What exists, exists in a way that the next iteration can build on. Configuration is text. Decisions are documented. The constraints that decided the architecture are written down. The events the system produces are structured and archivable.
A separate post picks up from here and looks at what becomes possible once a controlled artifact substrate exists locally. Drift detection across release histories, replay of past states, signature workflows that do not depend on a third-party registry being reachable. That belongs in its own piece.
References used while building this:
OCI Distribution Specification, endpoint definitions for /v2/, manifests and blobs.
zot installation guide, bare-metal Linux deployment.
nginx http_proxy_module, proxy_cache_path, proxy_cache_valid, proxy_cache_use_stale, proxy_ssl_server_name, proxy_ssl_verify.
OpenRC service script guide, command_user, command_background, pidfile, checkpath, dependency declarations.
OpenWrt firewall configuration, zone forwarding rules.
OpenWrt DHCP/DNS configuration, static A records via dnsmasq.
crane CLI documentation, client-side validation of OCI registries.
logfmt, structured logging in plain text.
home.arpa, RFC 8375, the special-use domain for residential networks.