The short version has four parts. The five heads carry the copy together but not alone. Patching all five recovers about 0.81 of clean performance, while no single head recovers more than 0.128. Their keys are fed by one upstream head, L4H11 in layer 4, and not by a layer 0 head as the textbook two-layer story and our own outline assumed. Their weights implement the two halves of induction, attend back and copy, but the head with the highest induction score, L5H5, is the weakest direct copier of the five. And removing all five costs only 33% of the model's in-context learning score, because GPT-2 small keeps many other induction heads in reserve.

That last result is the one the first post warned about. It does not undo the circuit. It changes what "the induction circuit" can mean in this model, from a pair of heads to a redundant population with a few members that matter more than the rest.

Everything runs on an M4 Pro laptop through MPS, with the same inputs as experiment 01. Every number below is read from a file in projects/tiny-circuits/results/ unless the text says otherwise.

Reading note. The argument runs in the order of the experiments, from what the heads attend to, through what causes the output, to what the weights say and what removal costs. The surprises are in the upstream head, the L5H5 result in the weights, and necessity. Skip to conclusions for the claims and their limits.

table of contents

• what the first post left open
• inputs and metrics
• attention patterns
• activation patching
• the upstream head
• the weights
• necessity
• against the literature
• what this does not establish
• instrument notes
• reproducibility
• conclusions

what the first post left open

The claimed mechanism has two heads and one edge. A previous-token head writes "the token before me was A" into each position. An induction head in a later layer uses that signature as its key, so a query for token A finds the position right after the last A, and its OV circuit copies the token there into the output. Experiment 01 tested only whether heads exist whose attention lands on the induction target. Experiment 03 tests causation by patching, experiments 03 and 04 test the upstream edge, experiment 05 reads the mechanism from the weights, and experiment 06 tests necessity by ablation. The first post's plan put path patching under experiment 04; in the code it landed in 03, next to the other patching runs.

inputs and metrics

Every experiment uses the input from experiment 01, a BOS token followed by 50 random tokens and then the same 50 tokens again, with seed 1337. Experiments 02 to 05 use a batch of 8. Experiment 06 uses a batch of 32 whose first 8 rows are the same sequences, because loss differences between ablations are small and a larger batch steadies them.

Patching needs a corrupted input. We corrupt the first half with fresh random tokens, giving [BOS][A'][A]. The second half, and so every prediction target, is identical in both runs, and the only thing removed is the earlier occurrence induction needs to look back at. Corrupting the second half instead would change the labels at the scored positions, and a residual patch there would mostly re-insert the clean input token.

The patching metric is the mean log probability of the correct next token over second-half positions, normalized so the clean run scores 1 and the corrupted run scores 0. A patch that scores 0.4 recovered 40% of the gap between the two runs. In the clean run the second-half mean log probability is about -0.22 nats, and in the corrupted run it is about -12.36, so the gap is large and the normalization is well conditioned.

The ablation metric in experiment 06 is an in-context learning score, the mean loss on the second half minus the mean loss on the first half, in the spirit of the ICL score in Olsson et al. (2022). The first half is random, so its loss sits near chance. The second half is predictable only by copying. For the intact model the first-half loss is 13.17 nats, the second-half loss is 0.24, and the ICL score is -12.94. A score of 0 would mean the model gains nothing from having seen the sequence before.

attention patterns

Experiment 02 plots the full attention pattern of each of the five heads on one sequence, next to L4H11, the head the literature names as GPT-2 small's main previous-token head. For every head we also split the attention of second-half queries into four buckets, the induction target, the previous token, BOS, and everything else, averaged over the batch.

Source, results/02_attention_viz.csv.

The induction column repeats the experiment 01 scores exactly, because it is the same quantity. What 02 adds is where the rest of the attention goes. None of the five heads puts more than 0.001 on the previous token, so they are not previous-token heads that happen to score well. Their leftover mass is split between BOS, where GPT-2 heads park attention when there is nothing to attend to, and a diffuse remainder that is largest for L7H2 at 0.143.

The plotted patterns (figures/02_attention_viz.png) show the geometry the score implies. In the first half, where no earlier copy exists, the five heads attend almost entirely to BOS. In the second half they draw one clean stripe offset 49 positions below the diagonal. L4H11 draws the off-by-one diagonal across the whole sequence and no stripe. This is still attention, and so still correlation. It rules out only that the scores come from some broader pattern overlapping the diagonal.

activation patching

Experiment 03 intervenes on activations. Each run takes one activation from the clean run and puts it into the corrupted run (denoising, which tests whether the component is sufficient to restore the answer), or takes one activation from the corrupted run and puts it into the clean run (noising, which tests whether the component is necessary).

the residual stream handoff

The first intervention patches the residual stream entering a layer, resid_pre, at one half of the sequence at a time. If induction works as described, the information the second half needs starts out at first-half positions, since that is where the earlier copy lives, and is moved to second-half positions by the induction heads.

That is what happens. Patching the clean first half into the corrupted run recovers 0.92 of performance at layers 4 and 5, then 0.35 at layer 6, 0.18 at layer 7 and 0.06 at layer 8. Patching the clean second half recovers nothing up to layer 5 (between -0.017 and 0.000), then 0.15 at layer 6, 0.46 at layer 7 and 0.87 at layer 8. The curves cross between layers 6 and 7. The handoff happens across exactly the layers where the five heads sit, 5 through 7, and it is complete by layer 8.

At layer 0 the two patches recover exactly 1.0 and 0.0, as they must, since resid_pre there is the token embedding. After layer 8 the second-half curve keeps rising to 0.998 at layer 11, so later layers add to the answer without moving it again. Patching a single position is much weaker, at most about 0.022, because the task is spread over 50 positions.

heads one at a time and together

The second intervention patches one head's output, hook_z, at all positions, in both directions.

Single heads matter little. In the denoising direction the best head, L7H2, recovers 0.128 and L6H9 recovers 0.120. L5H5, the top head by induction score, recovers only 0.031. In the noising direction the picture is flatter still. Corrupting one induction head costs almost nothing, with one exception, L5H1, whose corruption loses 0.160. The other four each lose less than 0.003.

Patched together, the five heads are a different object. Denoising all five at once recovers about 0.81 of clean performance, and noising all five loses about 0.77. The sum of the five single-head denoising effects is about 0.46, well short of 0.81, so the heads are more than additive when restored together. The reading we favor is redundancy. With four clean induction heads still running, corrupting a fifth changes little, because the others still deliver the copy. Only when all of them are corrupted at once does the redundancy run out.

The five are also not the only heads that recover performance. L9H6 recovers 0.104 and L9H9 recovers 0.103, level with the canonical heads. These rank seventh and eighth by induction score in experiment 01 (0.509 and 0.500). L10H0 recovers 0.082 and L10H1 recovers 0.067. The patching result is already hinting at the necessity result below. Induction in GPT-2 small is done by more heads than the five with the sharpest attention.

heads that push against the answer

Two heads have clearly negative denoising effects. Restoring the clean output of L10H7 lowers the metric by 0.092, and restoring L11H10 lowers it by 0.055. Giving these heads their clean input makes the model worse at predicting the repeated token.

These are the heads Wang et al. (2022) called negative name movers in the indirect-object circuit, and that McDougall et al. (2023) reinterpreted as copy suppression heads. A copy suppression head attends to tokens that earlier heads are predicting from context and pushes their logits down, which calibrates overconfident copying. On a task where copying is always right, that calibration shows up as a cost. Experiment 05 confirms the mechanism from the weights.

the upstream head

The outline for this project put the previous-token head in layer 0, following the two-layer attention-only model of Elhage et al. (2021). In GPT-2 small it is not there.

path patching into the keys

Path patching isolates one edge. For each of the 60 heads in layers 0 to 4, we take the difference between its corrupted and clean outputs and add it only to the key input of the five induction heads, leaving every other path, including the head's effects on MLPs and later heads, clean. We repeat this for the query and value inputs as a control. If a previous-token head feeds induction through K-composition, its effect should appear through keys and nowhere else.

One sender dominates. Corrupting L4H11's direct path into the induction keys loses 0.242 of clean performance. The next sender, L3H7, loses 0.013, and L2H2 loses 0.007. L4H11's query and value paths lose about 0.0001 each, and no sender's query or value path reaches 0.002. The edge is real, it runs through keys as K-composition predicts, and one head carries almost all of it.

why the total effect is smaller

Here is a surprise. L4H11's total effect is smaller than its direct path. Noising its output everywhere, rather than only on the path into the induction keys, loses 0.093 (from the head patching table), less than half of the 0.242 its key path alone loses.

The difference is MLP4, the MLP in the same layer. When L4H11's output is corrupted everywhere, MLP4 sees the corrupted output too and its response partly offsets the damage. Holding MLP4 at its clean output while noising L4H11 raises the loss to about 0.26, back to the size of the direct path. Zero-ablating L4H11 costs only about 0.02.

We do not know what MLP4 computes that lets it compensate. The simplest reading is that previous-token information is available to MLP4 from other sources, so it can partly rebuild the signal L4H11 would have written. Downstream components compensating for a damaged one is documented in larger models by McGrath et al. (2023), who call it the hydra effect, and backup heads in GPT-2 small by Wang et al. We have not seen it reported for this MLP and this head. The practical point is about method. Had we measured L4H11 only by its total effect or by zero-ablation, we would have ranked it as a minor head. Path patching was needed to see that it is the main source of the keys.

previous-token scores

Experiment 04 scores every head on attention from each position to the one before it, on the repeated random sequences and, as a check that the score does not depend on random tokens, on a short passage of English.

Source, results/04_prev_token_scores.csv, top six of 144 heads by the random-token score.

L4H11 is a near-perfect previous-token head on both kinds of input. Everything else is partial. The strongest layer 0 head is L0H7, at 0.185 on random tokens and 0.264 on English, so no layer 0 head plays the role the two-layer story assigns. L2H2 is the head Wang et al. name alongside L4H11 as a previous-token head, and it shows up here as the second strongest on English.

composition scores

Composition scores ask the same question from the weights alone, with no input at all. Following Elhage et al., the K-composition score between an upstream head A and an induction head B is the normalized Frobenius norm of $W_{OV}^{A} (W_{QK}^{B})^{\top}$, which measures how much of what A writes lands in the subspace B's keys read. Q- and V-composition are the analogous products with B's query side and B's OV circuit. The baseline is the same score for random matrices of the same shapes, 0.036.

Averaged over the five induction heads, L4H11 has the highest K-composition of any head in layers 0 to 4, at 0.096, against 0.052 for its Q-composition and 0.038 for its V-composition. Its K-composition is between 0.087 and 0.104 with each individual induction head. Across the 60 heads in layers 0 to 4, previous-token score and mean K-composition have a Spearman correlation of 0.78.

The scores are noisy in the way ARENA's tutorial and Elhage et al. warn about. L4H7 has a K-composition of 0.091, almost level with L4H11, yet its previous-token score is 0.177 and its path-patching effect through the induction keys is -0.002. The weights say L4H7 could compose with the induction heads, and the activations say it does not in any way that matters here. That is why we treat path patching as the primary evidence for the edge and composition scores as support.

the weights

Experiment 05 asks whether the weights implement induction independent of any input, which is the kind of evidence the first post said a circuit claim needs alongside patching.

OV circuits copy

A head's full OV circuit, embedding through value, output and unembedding, is a map from "the token this head attends to" to "which output logits go up." A copying head raises the logit of the token it attends to. We form this map on a fixed random subset of 2,000 vocabulary tokens and count how often a token's own logit is the largest in its row (top-1) or among the five largest (top-5). Chance is 1 in 2,000 for top-1. We also report the positive-eigenvalue share of Elhage et al., the sum of the eigenvalues divided by the sum of their magnitudes, which is 1 for a pure copying map and -1 for a pure anti-copying one.

The embedding matters. GPT-2 small is known to use its first MLP as an extension of the token embedding, so we compute the circuit twice, once with the raw embedding $W_E$ and once with the effective embedding $W_E + \text{MLP}_0(W_E)$, as McDougall et al. do.

Source, results/05_summary.json, 2,000 tokens.

With the effective embedding, L7H2, L7H10 and L6H9 map a token to itself for 87% to 92% of tokens, and to its top five for 96% to 99%. Their eigenvalue shares are 0.995 to 0.997. A Gaussian random head matched in norm scores a top-1 rate of 0.0006 and an eigenvalue share of -0.005. With the raw embedding, top-1 rates fall to between 0.004 and 0.058, which is the clearest single sign in this project that MLP0 acts as part of the embedding.

The eigenvalue test does not single out induction heads, and it should not be read as if it did. Among the other 139 heads the 90th percentile of the eigenvalue share is 0.988, and 33 of all 144 heads exceed 0.95. Copying OV circuits are common in the later layers of GPT-2 small. What distinguishes an induction head is copying combined with the right attention, not copying alone.

QK circuits attend back

The QK side is tested with L4H11 in the loop. For each query token t we score every key position whose previous token is one of the 2,000 subset tokens, with the key built from L4H11's output on that previous token, and ask whether the highest-scoring key is the one whose previous token is t. For all five heads it is, for 98.9% to 99.8% of tokens (L5H5 0.994, L6H9 0.998, L5H1 0.998, L7H10 0.989, L7H2 0.989). As a control, the same computation with plain token keys, no previous-token head, picks the matching key for exactly 0.0% of tokens in every head. The attend-back rule is in the weights, and it only exists once the previous-token head writes into the keys.

direct logit attribution

Direct logit attribution asks how much each head's output, passed straight through the final LayerNorm scale and the unembedding, raises the logit of the correct next token.

On second-half positions the five heads add L7H2 +1.51, L7H10 +1.16, L6H9 +1.08, L5H1 +0.69 and L5H5 +0.30. On first-half positions, where there is nothing to copy, all five contribute between -0.001 and +0.003. Together the five write 4.74 of the 13.37 that all 144 heads write directly, against a correct-token logit of 21.49 and a mean logit of about 0. L9H9 (+1.42), L9H6 (+1.41) and L10H0 (+1.17) each write as much as a canonical head.

The copy suppression heads show up from the other side. L10H7 writes -1.21 and L11H10 writes -0.77, the two most negative heads in the model, and their OV eigenvalue shares are -0.999 and -0.996, which is almost perfect anti-copying. Their weights do exactly what the patching result implied.

the L5H5 surprise

The next surprise is L5H5. It has the highest induction score in the model (0.925) and attend-back weights as sharp as the others (0.994), and yet it is the weakest copier of the five (top-1 0.155), the smallest direct writer (+0.30), and the smallest single-head patching effect (0.031 recovered). Its eigenvalue share of 0.953 says its OV circuit is broadly positive, just not sharply diagonal.

One reading is that L5H5 matters mostly indirectly, by writing something later heads read, rather than by writing the answer itself. An early induction head whose output is consumed by later induction or copying heads would look exactly like this. We have not tested that reading. Path patching from L5H5 into later heads would, and it is the obvious next experiment. The broader lesson is that rank by induction score, which measures attention placement, does not predict rank by causal effect, copying or direct write.

necessity

Experiment 06 removes heads and measures what is lost. Zero ablation sets a head's output to 0. Mean ablation replaces it with its mean over a separate batch of 64 repeated random sequences, position by position, which removes the token-specific information while keeping the average activation the rest of the model expects. Mean ablation is the cleaner test, and we report it first.

the five heads together

Mean-ablating all five induction heads moves the ICL score from -12.94 to -8.68, and the second-half loss from 0.24 to 4.46 nats. That removes 33% of the ICL score. Zero ablation gives a similar -8.94. Adding L4H11 to the five takes the score to -6.44 under mean ablation, removing half of it.

Twenty random sets of five heads, drawn from the other 139, are the control. Under mean ablation they average -12.86 with a standard deviation of 0.055, and the worst set reaches -12.76. The five induction heads matter far more than five random heads. They are not, on their own, the in-context learning of this model.

single heads

No induction head matters much alone under mean ablation. Each moves the ICL score by at most 0.047 nats (L6H9). L4H11 alone moves it by 0.294, the largest single-head effect in the model, which fits its role as the one head all five induction heads read their keys from.

Zero ablation tells a partly different story, and it is the one number here that disagrees with the patching picture in an informative way. Zero-ablating L5H1 costs 0.63 nats of ICL score, more than any other single head, while mean-ablating it costs 0.011. L5H1 also stood out in patching as the only induction head whose corruption alone loses real performance (0.160). One explanation is that L5H1's output carries a large constant component the rest of the model relies on, which the mean keeps and zero removes. We have not tested it.

backup induction heads

If the five are a subset of a larger population, ablating further heads in order of induction score should keep eroding the ICL score, and ablating random heads should not.

That is what the curve shows. With the top k heads mean-ablated, the ICL score is -12.03 at k = 4, -8.68 at k = 5, -5.34 at k = 10 and -1.13 at k = 20. By k = 20 the second-half loss is 11.80 nats against a first-half loss of 12.93, so in-context copying is nearly gone. Random sets of k heads, averaged over ten draws per k, stay near -12.7 all the way to k = 20.

The steps are informative. Most of the drop at k = 5 comes when L7H2 is added, after the first four heads together cost only 0.9 nats, so the four canonical heads above it are well covered by the rest. The curve jumps again at k = 8 when L9H9 joins L9H6, and at k = 13 when L8H1 is added. It bumps slightly the wrong way at k = 9 and k = 11, where the added heads are L10H7 and L11H10, the copy suppression heads. Removing a head that works against copying helps copying, which is the ablation result agreeing with patching and with the weights.

the knockout sweep

The last run mean-ablates every head alone. The largest effect by far is L4H11, at +0.294 nats. The next largest are L6H10 and L8H6, at +0.077 each, and 16 heads in total exceed +0.05. The heads whose removal most improves the ICL score are L6H6 (-0.086), L8H10 (-0.070) and L10H7 (-0.063), and all three have strongly negative OV eigenvalue shares in experiment 05 (-0.969, -0.992 and -0.999). The anti-copying heads are the heads whose loss helps.

No induction head appears near the top of the single-head sweep. That is the backup picture in its plainest form. The model has one bottleneck upstream, L4H11, and a redundant population downstream.

against the literature

The table sorts each result by how directly it can be compared with published work.

What we can match is head identities and signs. Magnitudes mostly cannot be compared, because the published work used other models (Olsson et al. and Elhage et al. study their own small models), other tasks (Wang et al. study indirect-object identification), or other metrics. We know of no published patching map, K-path size or top-k ablation curve for this setup. Every qualitative prediction we could check held, and the magnitudes here are new measurements rather than replications.

what this does not establish

Four limits apply to everything above.

First, the input is still repeated random tokens. That was the right choice for isolating induction, as the first post argued, and it means nothing here shows what these heads do on English. The one exception is the previous-token score of L4H11, which is 0.999 on a paragraph of English as well.

Second, the patching and ablation runs use one seed and small batches, 8 for patching and 32 for ablation. The random-head control gives a sense of the noise for ablation (a standard deviation of 0.055 nats under mean ablation), and there is no equivalent for patching. The ordering of the large effects is safe. Small differences, such as L9H6 against L9H9, are not.

Third, the mechanism is described for one edge and one layer of heads. We have not traced what reads from L5H5, what lets MLP4 compensate for L4H11, or why zero-ablating L5H1 costs so much more than mean-ablating it. Each is a named gap, not a detail.

Fourth, "the induction circuit" now needs a looser definition than the one the first post started with. A strict circuit, two heads and one edge that are together sufficient and necessary, does not describe GPT-2 small. A redundant population of induction and copying heads, fed by one previous-token head and opposed by a few copy suppression heads, does. That description fits every result here, and it is harder to test to the same standard.

instrument notes

reproducibility

Each experiment is a numbered script that writes its figures to figures/ and its tables to results/.

uv run python experiments/02_attention_viz.py
uv run python experiments/03_activation_patching.py
uv run python experiments/04_prev_token_heads.py
uv run python experiments/05_qk_ov_analysis.py
uv run python experiments/06_ablation.py

All five run on the same laptop as experiment 01. Experiment 03 is the slowest, since it runs more than 1,500 patched forward passes for the residual map, the head patching and the path patching. The charts in this post are drawn from JSON files copied from results/, and each names its source file.

conclusions

The first post found five heads that look in the right place. This one asked whether they do the work, and the answer has more parts than the two-head story allows.

• The five induction heads are causally responsible for in-context copying as a group. Patching all five recovers about 0.81 of clean performance and noising all five loses about 0.77, while the best single head recovers 0.128.
• The information moves from first-half to second-half positions between layers 5 and 8, across exactly the layers the five heads occupy.
• One upstream head, L4H11 in layer 4, feeds the induction keys. Its direct key path carries 0.242 of performance, the next sender carries 0.013, and its query and value paths carry nothing. No layer 0 head plays this role.
• MLP4 partly compensates when L4H11 is corrupted, so L4H11's total effect (0.093) understates its direct role. Path patching was needed to see it.
• The weights implement both halves of induction. The QK circuit with L4H11 keys picks the right key for about 99% of tokens and plain token keys for none. The OV circuits of three heads copy for 87% to 92% of tokens, once MLP0 is counted as part of the embedding.
• Rank by induction score does not predict rank by effect. L5H5 has the top score and is the weakest copier, direct writer and single-head patch of the five.
• Removing the five heads costs 33% of the ICL score, against about 1% for random heads. Removing the top 20 by induction score costs over 90%. GPT-2 small has many backup induction heads, and two copy suppression heads, L10H7 and L11H10, work against all of them.

The code is about 2,700 lines of C++ across the library, the broker, a CLI and a benchmark harness, plus about 830 lines of GoogleTest. There are 31 tests, and on 2026-09-26 an independent clean release rebuild passed all 31. The ASan plus UBSan and ThreadSanitizer builds also passed in my own runs, but those, and the benchmarks, were not rerun independently.

The headline measurement is how much batching matters. With 100 B messages, going from one record per produce request to 1,000 moved throughput from 32,886 to 12,744,103 messages per second, a factor of about 390. End to end, without an fsync, a message took 41.5 µs at the median from send to receive at 10,000 messages per second. Pushing data all the way through the drive's write cache with F_FULLFSYNC raised that to 4.0 ms. All benchmark numbers were measured on a shared Apple M4 Pro that was also running other people's builds and benchmarks, with the 1 minute load average recorded in every row (8 to 11 for the reported runs). They are indicative, not a clean benchmark.

Code is in projects/04-minikafka-log-broker. v0 is one broker, with no replication.

The argument of this post is that Kafka's speed comes from a few simple decisions about bytes, and that the hard part of reproducing it was not the protocol but keeping the index, the locks and the benchmarks honest. Skip to problems for the bugs.

table of contents

• what I wanted to build
• theory
• architecture
• implementation
• problems
• experiments
• results
• what I would change
• reproducibility

what I wanted to build

I wanted to understand why Kafka is fast and what it actually promises, by building the core of it myself. The v0 scope was chosen to be small enough to finish and verify but real enough to measure.

• Topics with partitions, each an append-only log of segment files on disk.
• A sparse offset index per segment, so that fetching from offset N does not scan the log.
• Crash recovery that finds and removes a torn write at the end of the log.
• A binary TCP protocol with batched produce and long-poll fetch.
• Consumer groups with broker-side partition assignment, and committed offsets stored in an internal log so they survive a restart.
• Size-based retention.
• Producer and consumer libraries, a CLI, tests for ordering, durability and resume, and throughput and latency benchmarks.

Replication, a controller, zero-copy sendfile, compaction and exactly-once delivery are out of scope for v0. They are on the roadmap, and a few of the measurements below show exactly where their absence costs something.

theory

a partition is a log

A Kafka partition is an ordered sequence of records where each record gets the next integer offset. Almost everything else follows from that one decision.

Appends only touch the end of a file, so writes are sequential, and the operating system's page cache absorbs them. A consumer that is caught up reads data that was written moments ago, which is still in the same cache, so a well-behaved consumer rarely touches the disk at all.

A consumer's entire state is one number per partition, the next offset to read. That makes committing progress cheap and moves the question of which messages have been seen out of the broker.

ordering and keys

Ordering is promised only within a partition. A producer that wants all events for one customer to stay in order gives them the same key, and the key is hashed to pick the partition. Partitions are also the unit of parallelism, since each partition in a group is read by exactly one consumer at a time.

finding offset N

A consumer asks for "offset 1,048,576 onwards". Records have variable length, so offset N is not at a computable byte position. The broker needs an index.

Kafka's index is sparse. Every few KiB of log, it writes an entry mapping a relative offset to a file position. A lookup is a binary search for the last entry at or below the target, then a short forward scan through record headers. Because offsets are dense, the floor entry is never more than one interval of bytes behind the target, so the scan is bounded.

A dense index would cost 8 bytes per record for no real gain, since the fetch reads a chunk of the file anyway. At a 4 KiB interval, a 64 MiB segment needs at most 16,384 entries, or 128 KiB, small enough to keep in memory.

batching

Every produce request pays fixed costs, a system call on each side, a network round trip, a lock acquisition and, if configured, an fsync. Batching amortises them over many records, and most of the experiments below ask how much that buys.

durability is a policy

Acknowledging a write after write(2) means the data is in the kernel. It survives the broker process crashing or being killed, but not a kernel panic or power loss. fsync(2) asks the kernel to push the data to the drive. On macOS, fsync hands data to the drive but does not flush the drive's own write cache, and fcntl(F_FULLFSYNC) does. Kafka itself acknowledges after the write by default and relies on replication for durability. A single broker has no replicas, so it has to choose, and v0 exposes the choice as a flag and measures it.

architecture

Clients hold one TCP connection each, with one request in flight. The broker accepts connections on one thread and serves each connection on its own thread. A request frame is decoded and dispatched to one of ten APIs, which are CreateTopic, Metadata, Produce, Fetch, ListOffsets, OffsetCommit, OffsetFetch, JoinGroup, Heartbeat and LeaveGroup.

one byte format for the wire and the disk

The most important decision is that a record looks the same on the wire and on disk.

record = header (16 B) + payload
  u64 offset        0 from producers, assigned by the broker
  u32 payload_len   bytes after the crc field
  u32 crc           CRC-32C of the payload
  payload:
  i64 timestamp_ms
  u32 key_len
  key bytes
  value bytes       payload_len - 12 - key_len

A record costs 28 bytes plus its key and value. Because the formats match, the broker never re-encodes anything. On produce it validates the batch, writes the assigned offsets into the headers in place and issues a single pwrite. On fetch it copies a byte range out of the file into the response. The consumer does the parsing and CRC checking. This is Kafka's main trick, and it is also what would make zero-copy sendfile a small change later.

Integers are little-endian, unlike Kafka's big-endian. Every machine this runs on is little-endian, so encoding is a memcpy, and the code asserts the host byte order at compile time.

the produce path

1. The producer appends encoded records to a per-partition buffer and sends it when it reaches a record count or byte limit, or on flush().
2. The broker reads the whole frame, and validate_batch walks it, bounds-checking every length and verifying every CRC. Nothing touches the log until the whole batch is known good.
3. PartitionLog::append takes the partition mutex, rolls to a new segment if needed, patches offsets into the headers, and writes the batch. With a flush policy other than none, the segment is flushed before the lock is released.
4. If any fetch is waiting, the broker bumps a long-poll epoch and wakes it. It replies with the base offset.

the fetch path

1. PartitionLog::read takes the mutex only long enough to check the offset range, pick the segment by binary search over base offsets, look up the floor index entry, and snapshot the segment's current size.
2. Outside the lock, it preads from the index position, skips records below the target, and cuts at the last whole record that fits in max_bytes. The first record is always returned whole, so an oversized record cannot wedge a consumer.
3. If no partition had data and the request allows waiting, the fetch sleeps on a condition variable until a produce bumps the epoch or the deadline passes, and then reads again.

invariants the code relies on

DESIGN.md lists eight. The two that carry the most weight are that offsets in a partition are dense and assigned under the partition mutex, and that bytes of a segment below its current size never change, which is what lets fetches read without the lock. A third, that every record on disk passed validate_batch, is what lets recovery treat any bad record as the end of the log.

implementation

the segment

A segment is one .log file and one .index file, named by the base offset padded to 20 digits so that a directory listing sorts in offset order. The index lives in memory as a vector of {u32 relative offset, u32 file position} pairs, and new entries are appended to the index file after the data they point at has been written.

The index loop is the part I got wrong first (see problems). The current version walks the batch record by record while appending and drops an entry at every record boundary where 4 KiB or more has been written since the previous entry.

void Segment::append(const uint8_t* data, size_t n, uint64_t first_offset, uint32_t count) {
  const size_t old_entries = index_.size();
  size_t pos = 0;
  for (uint32_t i = 0; i < count; ++i) {
    if (bytes_since_index_ >= index_interval_) {
      index_.push_back({static_cast<uint32_t>(first_offset + i - base_),
                        static_cast<uint32_t>(size_ + pos)});
      bytes_since_index_ = 0;
    }
    const size_t rec = kRecordHeaderSize + load_u32(data + pos + 8);
    bytes_since_index_ += rec;
    pos += rec;
  }
  pwrite_all(log_fd_, data, n, size_);
  // new index entries go out in one write, after the data they point at
  ...
}

Lookups use std::upper_bound to find the first entry past the target and step back one. Reads go through a small buffered reader that fetches a window of the file with one pread and serves header lookups from it, so a fetch is usually one or two system calls rather than one per record.

recovery

Only the last segment of a partition can have a torn tail, because before rolling to a new segment the outgoing one is flushed with the configured policy. So on startup only the active segment is scanned. Each record must have the expected next offset, a sane length and a matching CRC, and the file is truncated at the first record that fails.

while (limit - pos >= kRecordHeaderSize) {
  const uint8_t* h = r.at(pos, kRecordHeaderSize);
  const uint64_t off = load_u64(h);
  const uint32_t len = load_u32(h + 8);
  const uint32_t crc = load_u32(h + 12);
  if (off != expect || len < kPayloadFixedSize || len > kMaxPayloadSize ||
      len > limit - pos - kRecordHeaderSize)
    break;
  const uint8_t* body = r.at(pos + kRecordHeaderSize, len);
  if (crc32c(body, len) != crc) break;
  ...  // rebuild index entries as we go
  pos += kRecordHeaderSize + len;
  ++expect;
}
if (pos != size_) ftruncate(log_fd_, pos);

The expected-offset check matters as much as the CRC. A torn write can leave a header whose length field happens to be plausible, and the offset is a second, independent check that the bytes belong where they are. Sealed segments trust their index, but if an index file is missing, has a partial entry, or is not strictly increasing in both offset and position, the segment rebuilds it by the same scan. The cost of scanning only the active segment is that corruption inside a sealed segment is not detected at startup. A consumer would see a CRC error when it reached it.

CRC-32C uses the ARMv8 CRC instructions, eight bytes per __crc32cd, with a table-driven software fallback. A test checks the two against each other and against known vectors.

reads without the lock

The partition mutex serialises appends. I did not want a slow consumer's disk read to block a producer, so a read holds the lock only to decide what to read.

{
  std::lock_guard lk(mu_);
  ...  // range checks, pick segment by binary search
  seg = *std::prev(it);                   // shared_ptr<Segment>
  start_pos = seg->floor_position(offset);
  limit = seg->size();                    // snapshot, bytes below this are immutable
}
res.data = seg->read_from(start_pos, offset, max_bytes, limit);

Two things make this safe. Bytes below the snapshotted size never change, so a concurrent append cannot tear what the read sees. And the read holds a shared_ptr to the segment, so if retention deletes the segment in the middle of the read, the file is unlinked but the descriptor stays open until the last reference drops.

retention

Size retention deletes whole segments from the front, but only if what remains is still at least the retention target, and it never deletes the active segment. After retention runs, the log's size is between the target and the target plus one segment. The internal offsets log has retention forced off, because without compaction retention would silently forget committed offsets.

long polling without a lost wakeup

A consumer that is caught up sends a fetch with a maximum wait. The broker should park that request and wake it when data arrives, but it must not miss a produce that lands between the fetch's read and its wait. The usual bug is a wakeup sent before the waiter is waiting.

The fetch registers itself as a waiter before its first read, snapshots an epoch counter, reads, and then waits for the epoch to change. A produce increments the epoch after its append, and only if the waiter count is above zero, so producers pay nothing when no one is polling.

if (guard.c) guard.c->fetch_add(1, std::memory_order_acq_rel);   // register first
for (;;) {
  uint64_t epoch;
  { std::lock_guard lk(data_mu_); epoch = data_epoch_; }         // snapshot before reading
  ...  // read every requested partition
  if (any || max_wait_ms == 0 || stopping_ || now >= deadline) break;
  std::unique_lock lk(data_mu_);
  data_cv_.wait_until(lk, deadline, [&] { return data_epoch_ != epoch || stopping_.load(); });
}

The partition mutex orders the fetch's read against the producer's append. Either the read sees the new data, or the append happened after it, in which case the producer sees the registered waiter and bumps the epoch, which the wait predicate then observes.

consumer groups and fencing

A consumer joins a group with the topics it wants. The coordinator keeps, per group, a generation number, an ordered map of members, and the current assignment. Any membership change, whether a join, a leave or a session expiry, bumps the generation and recomputes a range assignment on the broker. Each subscribed member gets a contiguous block of partitions, and the first few get one extra when the count does not divide evenly.

Kafka splits this into JoinGroup and SyncGroup, with a barrier in between so that every member has given up its old partitions before anyone gets new ones. I left SyncGroup out. The cost is that during a rebalance two consumers can briefly fetch the same partition, until the old owner's next heartbeat tells it the generation changed. What keeps that from corrupting progress is fencing on commit.

ErrorCode GroupCoordinator::check_commit(const std::string& group, const std::string& member_id,
                                         uint32_t generation) {
  if (member_id.empty()) return ErrorCode::None;  // standalone consumer, no fencing
  ...
  if (!g.members.count(member_id)) return ErrorCode::UnknownMember;
  if (generation != g.generation) return ErrorCode::IllegalGeneration;
  return ErrorCode::None;
}

A consumer that was rebalanced away cannot commit, so it cannot move the new owner's offset backwards or forwards. The overlap turns into duplicate delivery, never into lost or rewound progress, which is at-least-once.

Committed offsets are records in an internal partition, __consumer_offsets-0, keyed by group, topic and partition, with the 8-byte offset as the value. The offset store holds one mutex across the append and the map update, so the log order and the in-memory order agree, and startup replays the log into the map. That gives commits exactly the same durability path as data with no second storage engine.

the clients

The producer keeps one buffer per partition and hashes keys with FNV-1a to choose one, or goes round robin for records without a key. The consumer fetches all of its assigned partitions in one long-poll request. Both sit on a Client with one blocking connection and one request in flight, which keeps ordering trivial and caps unbatched throughput.

testing

31 GoogleTest cases cover the codec, the log (including torn and corrupt tails, a broken sealed index and retention across a restart), the group coordinator, and the broker over real TCP, where they check ordering, key stickiness, durability across a restart, long-poll wakeup, commit and resume, and fenced stale commits.

The concurrent ordering test is the one I trust most. Six producers, each with a different batch size, write 3,000 messages each to one partition. The test then reads the partition back and asserts that offsets are exactly 0, 1, 2 and so on with no gaps, and that each producer's messages appear in the order it sent them.

In my own runs the suite passed in a release build in 0.85 s, under ASan plus UBSan in 4.08 s, and under TSan (results/test_summary.txt). A smoke test runs the real broker binary and CLI, consumes with a group, restarts the broker, produces one more message, and checks that the group sees only the new one while a new group sees everything (results/smoke_test.txt). On 2026-09-26 an independent clean release rebuild passed all 31 tests. The sanitizer and TSan builds were not rerun independently.

problems

1. the sparse index was only sparse between batches

My first Segment::append added at most one index entry per call, at the start of the batch. Tests passed, because every lookup was still correct. It was only slow.

It showed up in the first consumer benchmark. Reading 10,000 B messages with 4 KiB fetches ran at 14.5 MB/s, and no fetch size got the median above 82.5 MB/s (results/run1_batch_start_index/consume_throughput.csv). A consumer reading 10 KB records should be the easy case, so the shape was the clue rather than the level. Larger messages were doing worse than small ones.

The benchmark fills the partition with batches of about 1 MiB, which for 10 KB messages is 104 records per produce. With one index entry per batch, a fetch for a record in the middle of a batch had to start from the batch's first record and skip forward through up to 1 MiB of headers. With 4 KiB fetches, each fetch returns one 10 KB record, so reading one batch meant 104 fetches each skipping on average half a megabyte, about 50 times more bytes read than delivered by my arithmetic. The index was sparse between batches but not within them, and with big batches that is not sparse at all. (The devlog describes this as one entry per 10 MB, but the benchmark code fills 10 KB messages in batches of 104, so about 1 MiB is the right figure.)

The fix is the loop shown above, which indexes every record boundary past the 4 KiB interval, including inside a batch. Segment.IndexHasEntriesInsideLargeBatches appends a 1 MiB batch in one call, checks there are at least size / 4096 - 1 entries, and checks that for offsets 1, 500 and 999 the floor position is within one interval plus one record of the target and that reading from it returns exactly that record.

After the fix the same point measured 225.3 MB/s, and 4 MiB fetches reached 2,336.5 MB/s (results/consume_throughput.csv). I only claim the direction from that comparison, not the ratio. The first run was at a load average of 275 to 330 on 14 cores and the second at about 9, so machine load and the fix are confounded.

2. Apple clang's sanitizer runtime hangs

On macOS 26.5 with Apple clang 17, the ASan build deadlocked during runtime initialisation, before main, even for an empty program. That rules out my code. I moved the sanitizer and TSan builds to Homebrew LLVM, and CMakeLists.txt now prints a warning if you ask for a sanitizer with the Apple toolchain. Upstream clang emits DWARF 5, which Apple's linker warns about, so non-Apple builds pass -gdwarf-4.

3. the machine was not mine

The first two full benchmark runs happened at load averages between 138 and 330 on 14 cores, with other agents' builds and benchmarks running. The numbers were low and unstable. For example, 1000 B messages at batch 1000 measured 294.5 MB/s in the loaded run (results/run2_loaded/produce_throughput.csv) and 2,528.0 MB/s once load dropped to about 8 (results/produce_throughput.csv), more than eight times faster for the same code.

I could not get an idle machine, so I changed the harness instead. Every CSV row now records the 1 minute load average at the time it was measured, every throughput point is the median of 3 repetitions with the min and max kept, and I reran everything once load was around 8 to 11. The loaded runs are kept in results/run2_loaded/ and results/run1_batch_start_index/ rather than deleted. One detail to watch in the files is that results/bench_env.txt still has a "load average before run" of 204.79 in its header, which was recorded before the loaded run. The per-row load1 column is what describes the reported numbers.

Even at low load, nominally identical configurations disagree. 1000 B messages at batch 100 to one partition with no flush measured 1,001.6 MB/s in the producer sweep, 1,187.7 MB/s in the flush sweep, and 1,340.3 MB/s as the single-producer point of the scaling sweep. That spread of about a third is a fair estimate of how much to trust any single number in this post.

4. one noisy point even at low load

In the first low-load producer run, 100 B messages at batch 100 measured 14.6 MB/s, below batch 10, with a min to max spread of 11.7 to 38.6 (results/produce_throughput_noisy.csv). A curve that goes down when batching goes up usually means Nagle's algorithm holding small writes, but TCP_NODELAY was already set. Running the same mode again gave 233.4 MB/s with a spread of 229.6 to 240.2 (results/produce_throughput.csv). A tight spread on the rerun against a threefold spread in the original points to interference from other processes, not to my code. I kept both files and report the second.

5. slower send rates had worse tails

Without a flush, p99 latency at 1,000 messages per second was 3.02 ms, while at 10,000 per second it was 142 µs (results/latency.csv). Sending less made the tail about twenty times worse.

My explanation is that at the low rate the producer sleeps between sends, and the broker's connection thread and the consumer's long-poll thread park, so every message pays for waking threads up, and probably for cores leaving low-power states. At 10,000 per second the threads stay warm. The same pattern appears in the loaded and unloaded runs. I have not confirmed it by profiling, so it is a hypothesis.

experiments

All experiments use mk-bench, which starts a broker in the same process with a fresh data directory on the internal SSD and talks to it over loopback TCP through the real client code. Throughput is counted as payload bytes (the message value, not the 28 bytes of record overhead) in decimal megabytes. Throughput points are the median of 3 repetitions.

The hardware is an Apple M4 Pro with 14 cores and 48 GiB, running macOS 26.5, built with Apple clang 17 at -O2 (results/bench_env.txt). The machine was running other jobs throughout. The 1 minute load average was 8 to 11 during the reported runs and is recorded per row. None of the benchmarks were rerun independently, so these are single-session, loaded-machine numbers.

1. Producer throughput against batch size (1, 10, 100 and 1,000 records) and message size (100 B, 1 KB, 10 KB), one partition, no fsync. The batch is pre-encoded, so this measures transport plus broker (results/produce_throughput.csv).
2. Consumer throughput against fetch max_bytes (4 KiB to 4 MiB) and message size, reading a pre-filled partition from offset 0 and verifying every record's CRC (results/consume_throughput.csv).
3. End-to-end latency. A producer paced to a fixed schedule stamps each message with a monotonic clock, and a consumer long-polls and records receive time minus send time. Five configurations cover send rate, batch size and flush policy (results/latency.csv, with every sample in results/latency_raw.csv).
4. Cost of durability, with 1 KB messages at each batch size under the three flush policies (results/flush_policy.csv).
5. Producer scaling with 1, 2, 4 and 8 concurrent producers, each to its own partition, 1 KB messages, batch 100 (results/producer_scaling.csv).

results

batching dominates producer throughput

With 100 B messages, going from batch 1 to batch 1000 raised throughput from 32,886 to 12,744,103 messages per second (3.3 to 1,274.4 MB/s), a factor of about 390. At batch 1, every message size lands between 25,457 and 32,886 messages per second. That is the rate of one blocking round trip per connection, so at batch 1 the protocol is the limit, not the disk or the broker.

Large messages saturate earlier. 10 KB messages reach 1,311.4 MB/s at batch 10 but only 2,020.6 MB/s at batch 1000, because a 10-record batch of 10 KB is already 100 KB per request and the fixed costs are amortised. The best point was 1 KB messages at batch 1000, 2,528.0 MB/s, or about 2.5 million messages per second into one partition.

fetch size does the same for consumers

100 B messages went from 92.8 MB/s with 4 KiB fetches to 2,150.1 MB/s (21.5 million messages per second) with 1 MiB fetches, and 4 MiB did not help further (2,134.9 MB/s). These numbers include a CRC check of every record on the consumer.

10 KB messages are flat at 225.3 and 219.7 MB/s for 4 KiB and 16 KiB fetches, because each fetch returns exactly one record. The first record is always returned whole even when it exceeds max_bytes, so the fetch size stops mattering below the record size. They reach 2,336.5 MB/s at 4 MiB. 1 KB messages are the odd ones out, topping out at 1,645.8 MB/s, with a wide min to max spread at 1 MiB (725.7 to 1,785.0), which I read as noise rather than a real knee.

latency

Without fsync, end-to-end latency is tens of microseconds at the median. At 10,000 messages per second, p50 was 41.5 µs, p99 142 µs and p99.9 1.21 ms. The produce acknowledgement alone had a p50 of 32.9 µs in the same run, which suggests that most of the median is the cost of one request crossing loopback TCP and the broker, not the long poll. At 50,000 per second in batches of 10, p50 was 45.5 µs and p99 185 µs.

With fsync at 1,000 per second, p50 was 115 µs and p99 586 µs. With F_FULLFSYNC at 200 per second, p50 was 4.03 ms and p99 7.79 ms. That median is about 35 times the plain fsync median, and it is the cost of actually getting data past the drive's write cache, and it is the honest price of single-node durability on this machine.

durability costs depend on batching

At batch 100 with 1 KB messages, throughput was 1,187.7 MB/s with no flush, 594.9 MB/s with fsync and 23.2 MB/s with F_FULLFSYNC. At batch 1000 the gap narrows to 1,519.5, 1,324.1 and 197.6 MB/s, because each flush is amortised over about 1 MB. At batch 1, F_FULLFSYNC managed 231 messages per second, about 4.3 ms per message.

Each flush happens per append while the partition lock is held, so concurrent producers to one partition queue behind each other's flushes. Group commit would fix that but needs asynchronous acknowledgements, which v0 does not have.

producer scaling is sublinear

Aggregate throughput was 1,340.3 MB/s with one producer, 1,490.7 with two, 1,812.9 with four and 2,061.2 with eight (results/producer_scaling.csv), 1.54 times the single-producer rate with eight times the producers. Each producer has its own partition and its own lock, so it is not contention on the log. Plausible limits are memory bandwidth for copying through loopback TCP and the page cache, and the other load on the machine. I have not profiled it, and given the one-third spread between nominally identical runs noted in problems, I would not read much into the shape beyond "sublinear".

what load does to all of this

For contrast, the second loaded run, at per-row load averages of 138 to 231, gave 294.5 MB/s for 1 KB messages at batch 1000 and 382.0 MB/s at best for consumers (results/run2_loaded/). The first run at 275 to 330 had a p99 of 172 ms at 1,000 messages per second without fsync (results/run1_batch_start_index/latency.csv), against 3.02 ms at low load. Throughput from a shared machine is a lower bound, and tail latency from a shared machine mostly measures the other jobs.

what I would change

pipeline producer requests

One request in flight caps unbatched throughput at about 25,000 to 33,000 messages per second per connection. Allowing several in flight is easy for throughput and hard for ordering, since a retry of an earlier batch could land after a later one. Kafka solves that with per-partition sequence numbers checked by the broker, which is also the first step toward idempotent producers and exactly-once. I would build them together.

add a SyncGroup barrier

Today two consumers can briefly read the same partition during a rebalance. Generation fencing turns that into duplicate delivery rather than lost progress, but it is still a weaker guarantee than Kafka's, where the old owner revokes before the new one starts.

an event loop, sendfile and group commit

Thread per connection kept every handler straight-line code and was fine for a handful of benchmark connections, but it will not survive thousands of clients. A kqueue event loop is the right design there, and since the on-disk format already equals the wire format, fetches could then use sendfile and skip the copy through user space. The same asynchronous acknowledgement path would allow group-commit fsyncs across concurrent producers.

compact the offsets log

__consumer_offsets only grows, and startup replays all of it. Compaction, keeping only the latest record per key, is what Kafka uses to bound it, and it is on the roadmap along with time-based retention.

profile before explaining

Producer scaling and the low-rate latency tail both have plausible explanations in this post that I did not verify. I would rerun them on an idle machine with sampling (Instruments on macOS) before trusting either, and I would rerun every benchmark on an idle machine before quoting any number here as more than indicative.

reproducibility

Requirements are CMake 3.24 or newer, Ninja and a C++20 compiler. The project was developed with Apple clang 17 on macOS 26.5 on an Apple M4 Pro, and GoogleTest is fetched by CMake. From projects/04-minikafka-log-broker, the following commands build everything.

# Release (-O2)
cmake -S . -B build/release -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build/release

# ASan + UBSan, with Homebrew LLVM because Apple clang 17's sanitizer runtime hangs on macOS 26.5
cmake -S . -B build/debug -G Ninja -DCMAKE_BUILD_TYPE=Debug -DMK_SANITIZE=ON \
      -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++"
cmake --build build/debug

# ThreadSanitizer
cmake -S . -B build/tsan -G Ninja -DCMAKE_BUILD_TYPE=Debug -DMK_TSAN=ON -DMK_BUILD_BENCH=OFF \
      -DCMAKE_CXX_COMPILER="$(brew --prefix llvm)/bin/clang++"
cmake --build build/tsan

Tests and the smoke test.

ctest --test-dir build/release --output-on-failure
ctest --test-dir build/debug --output-on-failure     # ASan + UBSan
./build/tsan/mk-tests                                 # TSan
./scripts/smoke_test.sh                               # real broker + CLI, including a restart

Benchmarks and plots. The script runs one mode at a time and writes every CSV in results/, with the load average in each row. results/latency_raw.csv is gitignored and regenerated.

./scripts/run_benchmarks.sh 3
python3 -m venv .venv && .venv/bin/pip install matplotlib
.venv/bin/python scripts/plot.py

Code is in projects/04-minikafka-log-broker, with the library in src/ and include/minikafka/, the broker and CLI in tools/, the benchmark harness in bench/, the tests in tests/, the formats, invariants and trade-offs in DESIGN.md, and the build log in DEVLOG.md.

The number I trust most is not an accuracy. Trained side by side with a PyTorch twin from identical weights on identical batches, in float64, the two models finish three MNIST epochs with a maximum weight difference of 5.3e-14, and an independent rerun landed at 3.9e-14. That is float64 rounding territory, and it leaves no room for a bug in any gradient or optimizer update on that path. In float32 the same experiment ends ten epochs 0.313 apart, and most of this post's middle is about why that is not a bug.

With that established, the models are almost a formality. A 784-512-256-10 MLP reaches 97.89% MNIST test accuracy after ten float32 epochs (the PyTorch twin reaches 97.81%), which is one draw from a spread of about 0.2 points that depends on the BLAS thread count, and a character-level model reaches 1.794 nats per character on held-out Tiny Shakespeare against 2.482 for a bigram baseline. A full training step is 1.2x to 2.5x slower than PyTorch on one CPU thread, and a hand-written numpy baseline splits that gap into autograd overhead and kernel speed. Every timing was taken on a machine loaded by other jobs, so the absolute times are upper bounds and the single-threaded ratios are the useful part.

Code is in projects/11-smolgrad.

Reading note. The argument is that an autograd engine is short, and that almost all of the difficulty is in proving it right and in the handful of places where numpy's semantics and the chain rule disagree about who owns an array. Skip to problems for the bugs.

table of contents

• what I wanted to build
• theory
• architecture
• implementation
• problems
• experiments
• results
• what I would change
• reproducibility

what I wanted to build

I wanted to know what actually happens inside loss.backward() by writing it, and to hold the result to a standard that a toy would not survive. A scalar autograd in the style of micrograd is an afternoon. A tensor autograd that trains the same networks PyTorch does, to the same numbers, is a different object, because broadcasting, reductions over several axes, repeated indices and dtype promotion all have gradient rules that are easy to get almost right.

The goals for v0 were these.

• A Tensor with reverse-mode autodiff over a graph rebuilt every forward pass, and enough ops to train real models, from broadcasting arithmetic and batched matmul through reductions over arbitrary axes, indexing, softmax and cross-entropy.
• A module system (Module, Linear, Embedding, LayerNorm, Sequential) with SGD with momentum and AdamW.
• Proof that it is right, meaning a finite-difference check for every op and parity tests against PyTorch down to whole optimizer trajectories.
• Proof that it is useful, meaning a trained MNIST MLP and a trained character model.
• A benchmark against PyTorch that explains where the gap comes from, not just how big it is.

The only runtime dependency is numpy. PyTorch appears only as a test oracle and as the benchmark baseline.

theory

reverse mode in one paragraph

A training loss is one scalar $L$ computed from a few million inputs. Forward-mode differentiation would need one pass per input to get $\partial L / \partial \theta$. Reverse mode gets all of them in a single backward sweep whose cost is a small constant multiple of the forward pass. The trick is that each op $y = f(x_1, \dots, x_k)$ never needs its full Jacobian. It needs only its vector-Jacobian product (VJP), a function that takes $\bar{y} = \partial L / \partial y$ and returns $\bar{x}_i = \partial L / \partial x_i$ for each input.

The engine does three things.

1. During the forward pass, record each op's inputs and a closure that computes its VJP.
2. Topologically sort the graph from the loss.
3. Walk that order in reverse, seed $\bar{L} = 1$, call each closure, and sum every gradient that arrives at a node from its consumers.

The sum in step 3 is the multivariate chain rule. A node used twice (a residual connection, or x * x) receives two contributions, and forgetting to add them is the first bug everyone writes.

the VJPs that matter

For a gradient $G$ flowing into the output, the rules I needed are short.

The softmax rule matters because it never forms the $C \times C$ Jacobian $\mathrm{diag}(s) - s s^{\top}$. It contracts the Jacobian with $G$ analytically, which is the whole point of VJPs.

The gather rule is where the real difficulty hides. If index 7 appears three times in a batch, row 7 of the embedding table must receive the sum of three gradient rows. numpy's W[idx] += G keeps only one of the three writes, silently. More on that in problems.

broadcasting has an adjoint

Broadcasting copies a value along an axis. The adjoint of copying is summing. So every binary op has to take the gradient it received, which has the broadcast output's shape, and sum it back down to each input's shape. There are two cases. Broadcasting can prepend axes (a bias of shape (512,) added to activations of shape (128, 512)), and it can stretch a size-1 axis (shape (128, 1) against (128, 512)). Here is the function that undoes both, in full.

def _unbroadcast(grad, shape):
    if grad.shape == shape:
        return grad
    extra = grad.ndim - len(shape)
    if extra > 0:
        grad = grad.sum(axis=tuple(range(extra)))
    stretched = tuple(i for i, s in enumerate(shape) if s == 1 and grad.shape[i] != 1)
    if stretched:
        grad = grad.sum(axis=stretched, keepdims=True)
    return grad.reshape(shape)

Every binary op routes both parent gradients through it, and the engine asserts after every VJP that the gradient's shape equals its tensor's shape. That assertion catches an entire class of bug at the op that caused it rather than three layers later.

checking gradients numerically

The oracle for every op is a central difference.

$$\frac{\partial L}{\partial x_i} \approx \frac{L(x + h e_i) - L(x - h e_i)}{2h},$$

whose truncation error is $O(h^2)$. I run it in float64 with $h = 10^{-6}$. For an op with a tensor output, the checker contracts the output with a fixed random tensor $w$ and checks $L = \sum w \odot f(x)$ instead of $\sum f(x)$. With plain sum(), every output element gets weight 1, so a backward that permutes or transposes output gradients could pass. A random projection gives each element its own weight and closes that hole.

architecture

The value and the graph node are the same object. A Tensor wraps one np.ndarray (float32 by default, float64 when given float64) and uses __slots__, and there is no separate Node or Function class.

One training step flows like this.

closures, not Function objects

Each op defines its backward as a closure that captures what it needs from the forward pass. exp captures its output, softmax captures the probabilities. The closure returns parent gradients rather than writing them anywhere, which keeps all accumulation in one place (the engine) and makes each op's backward a pure function that finite differences can test in isolation.

The alternatives were a Function class per op, as in PyTorch's autograd.Function and tinygrad, or a tape of (op, inputs, output) records. Closures are the least code per op. Their cost is that they are opaque, so you cannot inspect, rewrite or fuse the graph. That trade-off comes back in what I would change.

gradients live in a local dict

During backward() the gradients of intermediate nodes live in a dict keyed by id(node), local to that call, and each entry is popped as soon as its node is processed. Only leaves get a .grad, and it accumulates across calls until zeroed, like PyTorch.

The micrograd approach of storing .grad on every node is simpler and has two bugs waiting. A second backward call double counts intermediate gradients unless something resets them, and every intermediate keeps a gradient array alive as long as the graph lives. The local dict avoids both.

PyTorch conventions, on purpose

Linear.weight is stored as (out_features, in_features), init is $U(-1/\sqrt{\text{fan\_in}}, 1/\sqrt{\text{fan\_in}})$, the SGD momentum buffer starts at $v = g$ on the first step, the AdamW update order matches, relu's gradient at exactly 0 is 0, and LayerNorm uses the biased variance. None of these are the only reasonable choice. They are PyTorch's choices, and matching them is what makes step-for-step parity tests possible, which is what makes the 5.3e-14 result possible.

implementation

The library is 891 lines across seven files. tensor.py (439 lines) holds the Tensor, the engine and every primitive op. functional.py holds the composed and fused ops, nn.py the module system, optim.py the optimizers and gradcheck.py the checker.

recording an edge

Every op builds its output through one function, and that function is the only place the graph grows.

@staticmethod
def _make(data, parents, backward, op):
    out = Tensor(data, dtype=data.dtype)
    if _GRAD_ENABLED and any(p.requires_grad for p in parents):
        out.requires_grad = True
        out._parents = parents
        out._backward = backward
        out._op = op
    return out

Under no_grad(), or when no parent needs a gradient, nothing is stored, so evaluation builds no graph at all. A typical op then reads like its math.

def __mul__(self, other):
    other = self._const(other)
    a, b = self, other

    def backward(g):
        ga = _unbroadcast(g * b.data, a.shape) if a.requires_grad else None
        gb = _unbroadcast(g * a.data, b.shape) if b.requires_grad else None
        return ga, gb

    return Tensor._make(a.data * b.data, (a, b), backward, "mul")

the engine

The topological sort is an iterative depth-first search with an explicit stack of (node, expanded) pairs, emitting a node only after all its parents. The reverse sweep is the part worth reading.

grads = {id(self): grad}
handed_out = set()
for node in reversed(self._toposort()):
    g = grads.pop(id(node), None)
    if g is None:
        continue
    if node._backward is None:  # leaf
        if node.grad is not None:
            node.grad = node.grad + g
            continue
        if id(g) in handed_out or not g.flags.owndata or not g.flags.writeable:
            g = g.copy()
        handed_out.add(id(g))
        node.grad = g
        continue
    for p, pg in zip(node._parents, node._backward(g)):
        if pg is None or not p.requires_grad:
            continue
        assert pg.shape == p.shape, f"{node._op}: grad {pg.shape} vs param {p.shape}"
        k = id(p)
        grads[k] = pg if k not in grads else grads[k] + pg

The last line is the chain rule's sum. Note that it allocates a new array (grads[k] + pg) rather than adding in place, because pg may be the same object another parent also received. The leaf branch with its three-way copy condition looks paranoid. It is the fix for the first bug in problems.

fused where it pays, composed where it does not

linear and layer_norm are compositions of primitives. LayerNorm is about ten graph nodes, slower than a fused kernel, but its gradient comes free from primitives that are already checked, and it matches torch.nn.LayerNorm gradients to a relative 1e-8.

cross_entropy is fused. Composing log(softmax(x)) produces three graph nodes and computes a log of something that can underflow to zero. The fused version computes a max-shifted log-sum-exp once and has the well-known backward.

def backward(g):
    grad = np.exp(logp)
    grad[np.arange(N), t] -= 1.0
    grad *= g * scale  # g is the upstream 0-d gradient, scale is 1/N for mean
    return (grad.reshape(logits.shape),)

A test checks it against the composed version, and another feeds it logits large enough that the naive form overflows.

embedding is fused because its fast backward cannot be expressed with the other ops. That backward turned out to be the most expensive line in the character model, which is the fourth entry in problems.

modules without registration

A Module finds its parameters by walking vars(self) in insertion order, recursing into sub-modules and lists of them, with no register_parameter. The deterministic order matters because optimizer state is a list aligned with the parameters, and the parity tests copy weights to PyTorch by zipping the two parameter lists.

AdamW with no allocations

AdamW is the decoupled-weight-decay version, where decay multiplies the weights directly instead of being added to the gradient and then divided by the adaptive denominator. Written as ordinary numpy expressions, one step created about six parameter-sized temporaries per tensor. The version that shipped uses one preallocated scratch buffer per parameter and out= arguments throughout.

np.multiply(g, 1.0 - self.b1, out=tmp)   # m = b1*m + (1-b1)*g
m *= self.b1
m += tmp
np.multiply(g, g, out=tmp)               # v = b2*v + (1-b2)*g^2
tmp *= 1.0 - self.b2
v *= self.b2
v += tmp
np.sqrt(v, out=tmp)                      # denom = sqrt(v)/sqrt(bc2) + eps
tmp *= inv_sqrt_bc2
tmp += self.eps
np.divide(m, tmp, out=tmp)               # p -= lr/bc1 * m/denom
tmp *= step_size
p.data -= tmp

It still makes about ten passes over each parameter, which is where most of the remaining optimizer gap to PyTorch comes from.

the tests

pytest collects 101 tests in three files.

• test_gradcheck.py has 58 finite-difference cases covering every op, including broadcasting variants, batched and broadcast-batch matmul, reductions over several axes and indexing with repeated indices, plus a self-test that the checker rejects a deliberately wrong backward. A checker that cannot fail proves nothing.
• test_engine.py has 19 tests of engine semantics. Diamonds, accumulation across backward calls, aliasing, no_grad, dtype preservation, a 20,000-iteration chain of about 40,000 nodes, and both paths of the embedding scatter against np.add.at.
• test_parity_torch.py has 23 tests comparing op values and gradients, modules, and 20-step SGD and AdamW trajectories against PyTorch. The trajectory tests require losses equal to a relative 1e-9 at every step and final weights equal to 1e-8 relative in float64.

problems

This section lists only the issues that left evidence in the code, the tests or the results files.

1. leaf gradients aliasing each other

add returns the same array object as the gradient for both parents, since $\partial(a + b)/\partial a$ and $\partial(a + b)/\partial b$ are both the identity. And sum's backward returns np.broadcast_to(g, shape), which is a read-only view with zero strides, not a real array. The first version of the engine stored whatever arrived at a leaf directly as its .grad.

That caused two failures. Two parameters could end up sharing one gradient array, so an in-place change to one silently changed the other. And the first time a leaf's gradient came from sum and was later accumulated into, the in-place add failed outright because the array was read-only.

The regression test is four lines and describes the bug exactly.

a = Tensor(np.zeros(3), requires_grad=True)
b = Tensor(np.zeros(3), requires_grad=True)
(a + b).sum().backward()
a.grad += 100.0
np.testing.assert_allclose(b.grad, [1.0, 1.0, 1.0])

The obvious fix is to copy every incoming gradient at every leaf. That works and costs a full copy of every parameter's gradient every step. The shipped fix copies only when it must, which is when the array is a view (not g.flags.owndata), is read-only, or has already been handed to another leaf in this same call (tracked in handed_out). Otherwise the leaf adopts the fresh array the VJP produced. The caller's seed gradient is always copied, since it might be the caller's own array and could otherwise become a leaf's .grad, and a second test covers that.

The lesson is that the engine's contract with every VJP needs an ownership rule. Mine is that closures never mutate their input gradient, and the engine makes sure a leaf's gradient is its own.

2. silent float64 promotion

Tensor(float32) * 0.5 must stay float32. Wrapping a Python scalar in a plain numpy array makes it float64, and numpy's promotion rules then make the whole product float64. Nothing crashes. Activations would quietly double in size and every downstream op would run at float64 speed.

The fix is Tensor._const, which wraps a constant in the calling tensor's dtype. The optimizer had the same bug in a different place. AdamW's bias correction 1 / sqrt(1 - b2**t) computed with numpy is a float64 scalar, and multiplying a float32 buffer by it promotes. It is now converted to a Python float first. test_float32_stays_float32_with_python_scalars guards the op side, and the float32 parity test asserts the loss dtype.

3. the recursion limit

A recursive topological sort is five lines and breaks at Python's default recursion limit of 1,000 frames. An MLP never gets near that. A long unrolled recurrence, or any test that chains ops in a loop, does. I replaced it with the iterative DFS above, and test_deep_graph_does_not_recurse runs y = y * 1.0 + 0.0 twenty thousand times and checks the gradient is exactly 1.

4. np.add.at was the most expensive call in a training step

The embedding backward has to scatter-add gradient rows into the table with repeated indices accumulating. The textbook numpy answer is np.add.at(full, idx, g), and that is what the generic indexing op still uses. It is correct. It is also an unbuffered per-element loop.

Profiling 200 steps of the character model with cProfile showed it. The top of the profile, from results/profile_char_step.txt, was taken at a load average of 252.

The scatter alone cost more than the whole optimizer. The char model scatters 4,096 rows (a batch of 256 times a 16-character context) into a 65-row table. I benchmarked two replacements in scripts/bench_scatter.py.

• The first is a one-hot matmul. Build an $N \times V$ one-hot matrix and compute $\mathrm{onehot}(\mathrm{idx})^{\top} G$. It is a single BLAS call, and wasteful in FLOPs, but at $V = 65$ the waste is irrelevant.
• The second is sort and reduceat. Sort the indices, find the start of each run of equal indices, and sum each run with np.add.reduceat. It does no wasted arithmetic, but pays for a sort and a gather.

At the char model's shape the one-hot product took 72 µs against 1,272 µs for np.add.at, a 17.6x speedup, with sort and reduceat at 484 µs. The one-hot matrix grows as $N \times V$, though, and at a GPT-2-sized vocabulary of 50,257 with 8,192 rows it would be over 400 million entries, so it is not even attempted there. At that shape sort and reduceat wins, 9,399 µs against 12,530 µs.

The shipped function picks by size.

def _scatter_add_rows(idx, g, num_rows):
    n = idx.shape[0]
    if n * num_rows <= (1 << 20):
        onehot = np.zeros((n, num_rows), dtype=g.dtype)
        onehot[np.arange(n), idx] = 1
        return onehot.T @ g
    order = np.argsort(idx, kind="stable")
    s = idx[order]
    starts = np.flatnonzero(np.concatenate(([True], s[1:] != s[:-1])))
    out = np.zeros((num_rows, g.shape[1]), dtype=g.dtype)
    out[s[starts]] = np.add.reduceat(g[order], starts, axis=0)
    return out

Two honest caveats. At the middle shape ($V = 1{,}000$, $N = 8{,}192$), the threshold sends the call to sort and reduceat even though one-hot had the lower minimum (1,794 µs against 1,981 µs). The medians rank them the other way (4,461 µs against 3,226 µs), so under this much load I cannot call it either way, and the threshold is a memory rule as much as a speed rule. Second, sort and reduceat is not bit-identical to np.add.at. Its maximum absolute error in float32 was 7.6e-6 at the small shape, because it sums each run in a different order. That is rounding, not a bug, and it foreshadows problem 6.

Rerunning the same profile after the change, the step went from 23.78 ms to 11.51 ms under the profiler and the scatter dropped out of the top ten. Not all of that halving is the scatter, though. The scatter accounted for 5.6 ms per step, and the rest of the 12.3 ms drop came from lines whose code did not change, such as the AdamW step falling from 1.030 s to 0.395 s over 200 steps. That part is load, and the honest claim is a 5.6 ms per step saving.

5. AdamW's temporaries

In the second profile the optimizer is the top line. For a 535,818-parameter MLP at small batch it is a large, batch-independent share of every step, and the expression form of AdamW allocated about six parameter-sized arrays per tensor per step. The out= rewrite shown in implementation removed them. It did not close the gap to PyTorch's multi-tensor AdamW, which I return to in results.

6. float32 drifts away from PyTorch, and it is not a bug

This is the one that looked like a real bug and took the most care to rule out.

The MNIST script trains a PyTorch twin from the same initial weights on the same batches and logs the maximum absolute weight difference after each epoch. In float32 it was 0.098 after one epoch and 0.313 after ten. That is not rounding error by any normal standard, yet every parity test passed. Either there was a bug that only shows up over thousands of steps, or training amplifies rounding differences into large weight differences.

Those two explanations make different predictions about precision. A real bug in a gradient or an update rule does not care about dtype. It should make the float64 twins diverge just as fast. Amplified rounding should shrink by roughly the ratio of the two machine epsilons, about nine orders of magnitude. So I reran the twin experiment in float64 for three epochs.

After 1,407 AdamW steps on 535,818 parameters, the two implementations agree to 5.3e-14. The epoch train losses agree to 15 significant figures. Test accuracy is identical at every epoch. There is no room for a bug in the forward pass, any gradient, or the optimizer on this path.

To see how the float32 gap grows, scripts/divergence.py runs the twins for 300 steps in both dtypes with both optimizers and logs the weight difference every step.

In float64 the difference stays at or below 1.5e-14 for 300 steps with either optimizer. In float32 it starts at the rounding level and does not grow smoothly. It jumps. The SGD run sits at 7.5e-8 through step 30 and is at 1.8e-6 one step later, 24 times larger. The AdamW run is flat at about 8.3e-6 for 77 steps, then climbs, and jumps again between steps 91 and 93. Discrete jumps like that are consistent with a ReLU unit whose pre-activation is near zero landing on different sides of zero in the two runs, after which the two networks are computing slightly different functions. I have not traced a specific unit, so that is an interpretation, not a finding.

The implementations round differently because they sum in different orders. numpy's reductions and Accelerate's GEMM do not accumulate in the same order as PyTorch's kernels, float32 addition is not associative, and training is a chaotic map that amplifies the difference.

AdamW starts noisier than SGD, 8.3e-6 after one step against 3.7e-9, and that has a clean explanation. Adam's update is $m / (\sqrt{v} + \epsilon)$, which normalizes each coordinate's step toward the full learning rate regardless of the gradient's magnitude. For a weight whose true gradient is near zero, a rounding-level difference in that gradient becomes a difference of order the learning rate in the update. SGD scales the update by the gradient itself, so tiny differences stay tiny.

The practical takeaway is that float32 parity with a reference implementation is only meaningful over a handful of steps, and long-run parity has to be tested in float64. The ten-epoch float32 twins still end at 97.89% and 97.81%, which is the level at which they should agree. The same sensitivity shows up between thread configurations, not just between implementations, as the MNIST results below describe.

7. the machine was shared

Every timing in this project was taken while other jobs were running on the same 14-core M4 Pro. The results files record the one-minute load average at each measurement, and it was 338 to 358 for the single-threaded benchmark, 358 to 360 for the default-threads benchmark, 194 to 202 for the scatter benchmark and 252 for the profile. A load of 300 on 14 cores means each runnable process waits most of the time.

The benchmark interleaves the implementations round-robin, so any burst of load hits all three alike, reports the minimum over all runs as the best estimate of intrinsic cost, since load can only add time, and records the load average in every results file. The medians in the CSVs measure the other jobs more than smolgrad, and I do not quote them. The absolute times below are upper bounds.

How much load distorted things shows in the training runs. The MNIST script's first epoch took 19.2 s and its tenth took 4.5 s with no code change, and the character model averaged 71 ms per step during training against a benchmark minimum of 3.4 ms.

experiments

1. Correctness is checked by pytest -q, 101 tests.
2. MNIST uses a 784-512-256-10 ReLU MLP, 535,818 parameters, AdamW at learning rate 1e-3 and weight decay 0.01, batch 128, 10 epochs in float32, with a PyTorch twin from identical weights on identical batches. Then the same twin setup in float64 for 3 epochs.
3. The drift experiment runs 300 steps for each of float32 and float64 crossed with SGD with momentum and AdamW, logging the weight difference to the PyTorch twin every step.
4. The character model is a Bengio-style character MLP on Tiny Shakespeare. An Embedding(65, 32) over a 16-character context is flattened to 512, then Linear(512, 512), LayerNorm, tanh, Linear(512, 65). That is 299,105 parameters, trained with AdamW at 3e-3 with a linear decay over the last quarter, batch 256, 6,000 steps, and compared against uniform, unigram and bigram baselines on the same validation split.
5. The step-time benchmark times a full training step (forward, backward, AdamW) of the MNIST MLP at batch 1 to 2048 and the character model at batch 256, single-threaded and with default threading, in three implementations. They are smolgrad, PyTorch eager on CPU, and the same MLP hand-written in numpy with manual gradients, no graph, and smolgrad's own AdamW. A microbenchmark also times 2,000 chained multiplies on one-element tensors, forward plus backward.

The hand-written numpy MLP is the design choice that makes the benchmark worth reading. Without it, "1.8x slower than PyTorch" could mean a slow graph engine or slow kernels. With it, smolgrad minus numpy is the price of autograd, and numpy minus PyTorch is the price of numpy.

results

correctness

All 101 tests pass. The float64 twin result above is the strongest end-to-end evidence, since it exercises every op in the MLP, the fused cross-entropy, the engine's accumulation and AdamW over 1,407 steps at once.

MNIST

I report only the final epoch. results/mnist_summary.json also records a best test accuracy of 98.11% at epoch 5, but there is no validation split, so picking the best epoch means picking it on the test set, and that number is optimistic by construction. A separate validation split would fix this.

The float32 run is deterministic for a fixed thread count but not across thread counts. An independent rerun with BLAS capped at 2 threads matched itself exactly, and reached 98.07% final accuracy with a final test loss of 0.0778, against 97.89% and 0.0863 here. The runs already differ after the first epoch (96.96% against 96.77%), which points to summation order in the matrix multiplies rather than a bug, and the float64 twin rules out a gradient error. So read 97.89% as one draw from a spread of about 0.2 points.

Test loss bottoms out at epoch 5 at 0.0623 and rises afterwards while train loss keeps falling, which is ordinary overfitting with no regularization beyond weight decay. The 0.08-point accuracy difference between the twins is float32 drift, not a property of either implementation. At epoch 8 the ranking was the other way around, 97.51% against 97.88%.

Tiny Shakespeare

The character model reaches 1.794 nats per character on the validation split after 6,000 steps, 0.688 below the bigram baseline. Train loss at the end was 1.576, so the model is starting to overfit its 16-character windows, and validation loss was still falling at the last evaluation, helped by the learning-rate decay. Samples at temperature 0.8 get the play format right, with speaker names and line breaks, mostly real short words and invented longer ones.

KING RICHARD III:
Ay, I woonesh thee, I say though it east hear good.

That is about what a 300,000-parameter model with a 16-character window should manage.

step time against PyTorch

Single-threaded, taking the minimum over 320 steps for batches up to 128 and 120 steps above, at a load average of 338 to 358.

The character model step at batch 256 was 3.40 ms for smolgrad and 2.77 ms for PyTorch, a ratio of 1.2x.

The batch 2048 row, where hand-written numpy is 17 ms slower than smolgrad, is load noise. The two run the same matmuls. I keep the row because deleting inconvenient measurements is worse than labelling them, and its other numbers deserve the same suspicion.

where the gap comes from

At batch 128 the phases split like this.

My reading of it, with the usual hedge that these are minima under heavy load and that phase minima do not add exactly to step minima.

The autograd machinery costs about 0.85 ms per step (2.94 minus 2.09), and almost all of it is in backward, 1.15 ms against 0.38 ms. The forward passes are within 0.06 ms of each other, so recording the graph is cheap. The backward cost is extra memory traffic. The hand-written version computes the relu mask and applies it in one expression, sums the bias gradient directly, and never builds a dict or checks an ownership flag. smolgrad runs _unbroadcast on every bias add, allocates a new array for every relu mask multiply, allocates again when summing gradients that arrive at the same node, and copies wherever a leaf cannot adopt its gradient.

The kernel gap is smaller, about 0.43 ms (2.09 minus 1.66), and most of it is the optimizer. PyTorch's foreach AdamW updates all parameters with multi-tensor kernels, while mine makes about ten numpy passes over each parameter. At batch 128, AdamW is 1.33 of smolgrad's 2.94 ms, or 45% of the step, and because its cost depends only on the parameter count it takes a larger share the smaller the batch.

Per-op Python overhead is not the problem. On one-element tensors, a forward plus backward multiply costs 6.2 µs in smolgrad and 12.6 µs in PyTorch single-threaded, and 6.2 µs against 10.0 µs with default threading. PyTorch's dispatcher does more work per op than a Python closure does. This surprised me, and it means the gap on real workloads is about how many bytes move, not how many Python calls happen.

The default-threads benchmark (40 or 15 steps per point, at a load average near 360) is less reliable. In it PyTorch at batch 128 is slower than hand-written numpy, 5.24 ms against 2.25 ms, which is what ten PyTorch threads on an oversubscribed machine look like. I use it only for the per-op overhead numbers.

what I would change

record ops, not closures

Closures made v0 short, and they make the graph a black box. There is no way to look at a chain of add, relu, mul and fuse it. The planned lazy-graph milestone needs an explicit op record (type, inputs, attributes), and I would switch to that first.

fuse the obvious hot spots before writing a backend

The phase table says where the 1.8x lives. A fused bias-plus-relu with an in-place mask in backward attacks the 0.77 ms of backward overhead directly. A multi-tensor AdamW over one flat parameter buffer would replace about ten numpy passes over each of the MLP's six parameter tensors with about ten passes over one buffer, and should attack most of the 0.33 ms optimizer gap. A fused LayerNorm forward and backward would replace ten graph nodes with one in the character model. All three are possible in numpy without writing a kernel.

add a version counter

Closures read their parents' .data lazily at backward time, so updating a parameter in place between forward and backward silently corrupts the gradients. PyTorch catches this with a version integer on every tensor. v0 relies on the convention that only the optimizer mutates in place, and it runs after backward.

write backward rules in Tensor ops for higher-order gradients

Right now each VJP is raw numpy, so the backward pass builds no graph and grad-of-grad is impossible. Supporting it means a second set of backward rules written in Tensor ops, behind a create_graph=True flag so the first-order path does not pay for it.

rerun every timing on an idle machine

scripts/run_bench_when_quiet.sh already waits for the load average to drop below a threshold. The numbers here were taken at a one-minute load average of 194 to 360 and are upper bounds, with the single-threaded ratios roughly right. The float64 parity result does not depend on the machine at all, which is why it leads this post.

reproducibility

Needs uv. Python is pinned to 3.12. From projects/11-smolgrad, the following commands set up the environment, run the tests, download the data and reproduce every experiment.

uv sync
uv run pytest -q

uv run python scripts/download_data.py              # MNIST (SHA-256 checked) and Tiny Shakespeare, about 13 MB
uv run python scripts/train_mnist.py --torch-twin   # 10 epochs, float32, with the PyTorch twin
uv run python scripts/train_char.py                 # 6,000 steps
uv run python scripts/divergence.py                 # float32 vs float64 drift, 300 steps each

uv run python scripts/bench.py --threads 1          # single-threaded step-time benchmark
uv run python scripts/bench.py                      # default threading
uv run python scripts/plot_bench.py
uv run python scripts/bench_scatter.py              # embedding backward strategies
uv run python scripts/profile_step.py               # cProfile of a char-model step

# on a shared machine, wait for load average < LOAD_MAX (default 20) first
scripts/run_bench_when_quiet.sh

The float64 MNIST twin run is the command below, and it writes results/mnist_train_f64_twin.csv and results/mnist_summary_f64_twin.json.

uv run python scripts/train_mnist.py --torch-twin --dtype float64 --epochs 3 --tag _f64_twin

Every number in this post comes from a file in results/, which holds the raw CSV, JSON and profile output of each run, each timing file stamped with the load average at the time. The exceptions are the rerun figures from the independent verification, which are recorded in DEVLOG.md.

verification

An independent reviewer who did not build the project reran it on 2026-09-26, with BLAS threads capped at 2 on a shared machine and every output written to a scratch folder. All 101 tests passed. The float64 MNIST twin reproduced, with 97.75% test accuracy for both implementations, every loss matching the committed CSV to about 1e-16, and a final weight difference of 3.9e-14 against the committed 5.3e-14, both at rounding level. The full character run matched results/char_summary.json to 1e-7, with validation loss 1.794 and the same three baselines. The drift experiment matched in float64 and for float32 SGD, while float32 AdamW agreed at step 1 and then drifted to a different end value, as expected for the chaotic float32 behavior described in problem 6. The float32 MNIST run reproduced only up to the thread-count spread described in the MNIST results. The reviewer also checked that the hand-written numpy baseline computes the same loss and gradients as smolgrad, and corrected the README, which had claimed checksums for both datasets when only MNIST is checked. Timing was deferred. The benchmark was only checked to run, and at a load average near 9 an MNIST epoch took about 1.0 s and the character run 17 s, against 4.5 to 19.2 s and 478 s here, so every timing in this post still needs a quiet machine.

Code is in projects/11-smolgrad, with the library in src/smolgrad/, the tests in tests/, and the design notes and full build log in DESIGN.md and DEVLOG.md.

The server is 1,922 lines of C++20 in src/, headers and comments included. There are 49 GoogleTest cases plus an end-to-end test written in stdlib-only Python against the real binary, and on 2026-09-27 an independent clean release rebuild passed all 50. The AddressSanitizer plus UBSan build also passed in my own runs, but it was not rerun independently, and neither were the benchmarks.

The headline measurement is how little the server's own work matters. At 50 connections, SET throughput went from 107,160 ops/s with one request in flight per connection to 2,942,811 ops/s with 128 in flight, a factor of 27 with no change to the server. Server CPU per command fell from about 9 µs to about 0.34 µs over that range, which says nearly all the cost at depth 1 is system calls and wakeups, not the hash table. All benchmark numbers were measured on a shared Apple M4 Pro that was also running other jobs, with the 1 minute load average recorded in every row (7 to 13 for the reported runs). They are indicative, not a clean benchmark.

Two gaps should be stated up front. The epoll backend exists as source but has never been compiled, because there was no Linux machine in the session, so the Linux half of the poller abstraction is a claim rather than a result. And there was no comparison against real Redis, because neither redis-server nor redis-benchmark was installed. Every number below is internally consistent, and none of them says whether kvd is fast relative to the thing it imitates.

Code is in projects/02-kvd-resp-server. The argument of this post is that a single-threaded server's speed is decided by how many system calls each command shares, and that the hardest thing to keep honest was the measurement rather than the code. Skip to problems for the bugs.

table of contents

• what I wanted to build
• theory
• architecture
• implementation
• problems
• experiments
• results
• what I would change
• reproducibility

what I wanted to build

Redis runs every command on one thread and is still fast enough that most people never need anything else. I wanted to understand why by building the core of it and measuring where it breaks.

The v0 scope was fixed before writing code. One event loop on kqueue, with an epoll path behind the same interface. RESP2 with pipelining and the inline form that makes nc usable. GET, SET with every expiry and condition flag, DEL, EXISTS, the EXPIRE family, TTL, INCR and friends, and KEYS with glob patterns. Lazy and active expiry modelled on Redis. An AOF with three fsync policies, replay and torn-tail recovery. And a load generator, so the design choices could be measured rather than asserted.

Multithreaded I/O, RDB snapshots, AOF rewrite, lists and hashes and sorted sets, pub/sub and replication were left for later milestones. COMMAND and CONFIG GET are stubs that return empty arrays, so a redis-cli handshake should succeed, though I could not check that here.

theory

why one thread can be enough

A request/response server spends most of its life waiting on the network. With a thread per connection the waiting is done by blocked threads, and the cost is context switches and stack memory. An event loop inverts that. One thread asks the kernel which sockets are ready (kqueue on macOS, epoll on Linux) and touches only those. As long as every command is short, one core can serve thousands of connections, and because only one thread ever touches the data, every command is atomic without a lock.

The catch is the word short. One slow command, a KEYS over a million keys or an fsync on a slow disk, stalls every client at once.

where the time goes

A GET that finds its key in a hash table costs well under a microsecond. The read(), send() and kevent() calls around it cost several. So the throughput of a server like this is set mostly by how many system calls each command pays for.

Pipelining attacks exactly that. If a client sends 16 commands before reading any replies, the server picks them up with one read(), executes them back to back, and answers with one send(). The syscall cost is shared 16 ways.

Little's law as a sanity check

In a closed-loop benchmark, where each connection keeps a fixed number of requests outstanding, Little's law ties the three quantities together.

$$\text{requests in flight} = \text{throughput} \times \text{mean latency}$$

With 50 connections and one request in flight each, a server doing 100,000 requests per second must show a mean latency of $50 / 100{,}000$ s $= 500$ µs, however fast each individual command is. Latency in a saturated closed-loop benchmark is set by the queue, not by the work. I used this to check every result below.

expiry without a timer per key

A key with a TTL must disappear on time, but nobody wants a timer per key. Redis combines two cheap mechanisms. Lazy expiry checks the deadline whenever a key is touched, so an expired key is never visible to a client. Active expiry runs about ten times a second, samples 20 random keys that have a TTL, deletes the expired ones, and repeats while more than a quarter of the sample was expired. That bounds how much memory dead but untouched keys can hold, at a cost proportional to the garbage rather than to the keyspace.

durability is a policy

The AOF is a log of write commands in the same RESP format clients send, and replaying it rebuilds the dataset. The durability knob is when to call fsync. always means before acknowledging a write, everysec means once a second, and no leaves it to the operating system.

architecture

Everything runs on one thread.

One loop iteration does four things in order.

1. before_sleep() writes the AOF buffer to the file, fsyncs it under always, and only then tries to send() every client's pending replies.
2. poller.wait() blocks until a socket is ready or the next cron tick is due. If a client still has unsent output that the kernel did not refuse, the timeout is zero.
3. Events are dispatched. The listening socket accepts until EAGAIN. A readable client gets one read() of up to 16 KB, and every complete command in its buffer is parsed and executed.
4. If the cron is due, it runs one active expiry cycle and the everysec fsync check.

Each component is one file under src/. The poller is a four-method interface with kqueue and epoll implementations behind it, so the server never includes a platform readiness header.

implementation

the loop that makes pipelining work

Pipelining is not a feature the server implements so much as a property of this loop. Every complete frame in the input buffer is executed in one pass, and all the replies land in one output buffer that goes out in one send().

void Server::process_input(Client& c) {
    int64_t now = wall_ms();  // one clock read per batch, like Redis's cached mstime
    while (!c.close_after_write && c.in_pos < c.in.size()) {
        if (c.out.size() - c.out_pos > cfg_.out_soft_limit) {
            c.read_paused = true;               // backpressure, see below
            poller_->set_read(c.fd, false);
            ++stats_.read_pauses;
            break;
        }
        ParseResult r = parse_request(std::string_view(c.in).substr(c.in_pos), args_);
        if (r.status == ParseStatus::Incomplete) break;
        // ... protocol errors reply and close ...
        c.in_pos += r.consumed;
        if (args_.empty()) continue;
        if (proc_.execute(args_, now, c.out) == CommandProcessor::Action::Close)
            c.close_after_write = true;
    }
    // compact `in`, then queue the client for before_sleep() if it has output
}

Each client carries an input buffer with a consumed offset and an output buffer with a sent offset. Advancing an offset is O(1). Erasing the front of a std::string after every command would make a deep pipeline quadratic in its length, so the input buffer is cleared when fully consumed (the common case) and compacted only when the dead prefix is over 64 KB and more than half the buffer.

a parser that copies once

The RESP parser is stateless. It is handed the unconsumed tail of the input buffer and returns complete, incomplete or error. For a multibulk frame it makes a first pass that records only offsets, and copies arguments into std::strings only once the whole frame is present.

struct Span { size_t off, len; };
std::vector<Span> spans;
for (int64_t k = 0; k < n; ++k) {
    // ... find "$<len>\r\n", validate len ...
    size_t need = data + static_cast<size_t>(len) + 2;
    if (buf.size() < need) return {};          // incomplete, nothing copied
    spans.push_back({data, static_cast<size_t>(len)});
    pos = need;
}
args.resize(spans.size());
for (size_t k = 0; k < spans.size(); ++k) args[k].assign(buf.data() + spans[k].off, spans[k].len);

An 8 MB value that trickles in across hundreds of reads is scanned for headers many times but copied once. There is a test that sends exactly that.

a dense vector of TTL keys

Active expiry needs a uniformly random key among those with a TTL, in O(1). Redis gets that from a second hash table of expiring keys and a random bucket walk. I kept a std::vector of raw pointers to the unordered_map nodes that have a TTL, with each entry storing its own index into that vector.

void Store::remove_ttl(Node& node) {
    Entry& e = node.second;
    if (e.expire_at_ms == kNoExpiry) return;
    size_t slot = e.ttl_slot;
    Node* last = ttl_nodes_.back();
    ttl_nodes_[slot] = last;          // swap the last pointer into our slot
    last->second.ttl_slot = slot;
    ttl_nodes_.pop_back();
    e.expire_at_ms = kNoExpiry;
}

void Store::erase(Map::iterator it) {
    remove_ttl(*it);  // must happen before erase, ttl_nodes_ points into the node
    map_.erase(it);
}

This is legal only because std::unordered_map never moves its nodes. Rehashing invalidates iterators but not pointers or references to elements. The price of that guarantee is that the map is node-based, one allocation per key, which comes back in the replay results. Store::check_invariants walks the map and checks that every TTL node is in the vector at the slot it records and that the counts match, and a randomized test runs it after each of 20,000 mixed operations.

The map also uses a transparent hash, so a lookup by string_view does not allocate a temporary std::string.

active expiry with a wall-clock budget

while (!ttl_nodes_.empty()) {
    size_t n = std::min(kSampleSize, ttl_nodes_.size());   // 20
    size_t expired_this_round = 0;
    for (size_t k = 0; k < n && !ttl_nodes_.empty(); ++k) {
        Node* node = ttl_nodes_[pick(rng_)];
        if (node->second.expire_at_ms <= now_ms) {
            erase(map_.find(node->first));
            ++expired_this_round;
        }
    }
    if (expired_this_round * 4 <= n) break;                 // at most 25% expired
    if (elapsed_us() >= budget_us) break;                   // 25% of a cron tick
}

At the default 10 Hz the budget is 25 ms per cycle, so expiry cannot stall the loop for longer than that. The budget is measured in wall-clock time, which matters later.

logs that do not depend on when they are replayed

A relative TTL replayed a day later would extend every key's life by the downtime. So each write is rewritten before it is logged.

Keys whose deadline passed while the server was down are purged in one sweep right after replay. Replay runs through a CommandProcessor with no propagation hook, so replayed commands are not appended to the file again.

reply after log, and group commit

The ordering in before_sleep is the durability guarantee.

void Server::before_sleep() {
    if (aof_ && !aof_->flush() && !aof_error_logged_) { /* log once, retry next time */ }
    std::vector<int> batch;
    batch.swap(pending_);
    for (int fd : batch) { /* ... */ try_write(c); }
}

Aof::flush does one write() of everything this iteration produced, then one fsync() if the policy is always, and only then does any reply leave the process. So under always an acknowledged write is on disk (in the sense macOS fsync means, discussed below). The end-to-end test checks it directly by writing 200 keys with appendfsync always, killing the server with SIGKILL, restarting it and finding all 200.

Because the fsync is per loop iteration and not per command, it is also a group commit. With 50 busy connections, one fsync covers every write any of them made in that iteration. That turns out to be the most important performance property of the AOF.

A torn final record, from a crash in the middle of a write(), is truncated on startup, which is what Redis does with aof-load-truncated yes. Garbage anywhere else in the file stops the server from starting, on the view that corruption in the middle means something else went wrong and silently dropping everything after it is worse than refusing to run.

backpressure by pausing reads

A client that pipelines faster than it reads replies makes its output buffer grow without bound. Redis disconnects such a client at a hard limit. I stop reading from it once its unsent output passes a soft limit, 1 MB by default, and resume when it drains. Because the check runs before every command, the output buffer is bounded by the soft limit plus one reply, and a fast pipelining client still makes progress at the speed it drains. The resume path has a subtlety that is in problems.

Two smaller choices keep the common path cheap. Write interest is registered with the poller only when send() hits EAGAIN, so a normal reply costs no extra kevent call. And readiness is level-triggered with one 16 KB read per wakeup, which is simpler than draining every socket under edge triggering and fairer, since one busy client cannot monopolize an iteration.

testing

Unit tests cover the parser, the glob matcher, the store and command semantics. In-process TCP tests run the real server on an ephemeral port and cover a 20,000 command pipeline, an 8 MB value, 16 clients doing 2,000 INCRs each, backpressure and AOF restart. scripts/client_test.py, an independent stdlib RESP client, drives the real binary through every command, real-time expiry, SIGTERM, SIGKILL recovery and a torn tail, standing in for redis-cli.

problems

1. Apple clang's sanitizer runtime hangs

The Debug build with -fsanitize=address,undefined hung at startup before reaching main. It is not my code. An empty int main(){return 0;} built with Apple clang 17 and -fsanitize=address hangs the same way on this macOS 26.5 machine, and it was still hanging when I rechecked at the end, killed by a 15 second timeout.

The fix was to point the asan CMake preset at Homebrew LLVM's clang++ and keep Apple clang for the release build. Both builds passed the same 50 tests in my runs. The independent rebuild covered only the release build.

2. a backpressure test that passed in one build and failed in the other

The first version of Server.BackpressurePausesAndResumesReads relied on the default 1 MB soft limit being crossed. Whether it was crossed depended on how fast the kernel drained socket buffers relative to how fast the server produced replies, which differed between the fast -O2 build and the slow ASan build. The assertion that reads had been paused was not deterministic.

The test now sets a 64 KB soft limit and pipelines 30,000 GETs of a 1 KB value, about 30 MB of replies, before reading anything. With the limit that far below the reply volume, the pause always happens regardless of build speed. It then reads all 30,000 replies, checks each one, and checks the pause counter is nonzero.

3. a paused client can stall forever

This one is a hazard the design had to close rather than a bug I watched happen, but it is the kind that would only show up as a hung client in production. When a client is paused, its input buffer may already hold complete commands that were read but not executed. When its output drains, re-enabling read interest is not enough. The socket may have nothing new to read, so no readable event ever arrives, and the buffered commands sit there forever.

So the resume path runs them itself.

if (!c.read_paused) return true;
// Output drained. Its input buffer may already hold complete commands that
// no new read event will ever announce, so process them now.
c.read_paused = false;
poller_->set_read(c.fd, true);
process_input(c);
if (c.out.empty()) return true;
// otherwise loop and try to send what that produced

The backpressure test exercises it, since the client in that test sends its whole pipeline before reading and the server necessarily pauses with commands still buffered.

4. benchmarking on a machine twenty times oversubscribed

The first full benchmark run happened while other heavy jobs were running. results/run1_loaded/bench_pipeline.jsonl records a 1 minute load average of 364.7 at the start, on 14 cores. The numbers looked like results and were not.

SET at 50 connections without pipelining gave a median of 17,325 ops/s with a p99 of 51.9 ms, against 107,160 ops/s and 935.5 µs once the machine was quieter. The instability was as telling as the level. The three workload-mix repetitions in the loaded run gave 27,832, 40,969 and 480,780 ops/s (results/run1_loaded/bench_workload.jsonl), a factor of 17 between runs of the same thing.

I noticed because the p99 numbers were absurd. Two changes made the data usable. First, every row now records the load average before and after the run and the CPU time the server process used, read from ps. Throughput per server CPU second is a number that load degrades much less, because it measures what the server did while it had a core rather than how often it got one. At depth 1 it was 92,593 in the loaded run against 112,360 in the quiet one. When the server is saturated, wall-clock throughput and throughput per CPU second should agree, which is a quick check that the server, not the load generator, was the bottleneck. Second, when the load dropped I reran everything one experiment at a time, with kvbench capped at 4 client threads so the benchmark itself added little load. The loaded run is kept in results/run1_loaded/ rather than deleted.

The CPU time comes from ps, which has 10 ms resolution. The shortest runs used 0.33 s of server CPU, so rounding alone can move a per CPU second figure by about 3 percent. It is a coarse instrument.

5. the expiry experiment raced itself

The expiry experiment loads 200,000 keys that all expire at one absolute instant, via PXAT, and then watches DBSIZE. If the deadline is chosen before encoding and sending the keys, a slow machine can reach the deadline while keys are still loading, and the drop smears across the load.

The script now encodes every command first, measures how long loading 200,000 persistent keys took, and sets the deadline to twice that plus 2 seconds. It records the margin it actually achieved, 1,779 ms before the deadline in the reported run (results/bench_expiry.jsonl, the expiry-meta row).

6. a run too short to measure

My first single-connection fsync experiment used 20,000 requests per run. It showed the effect, but the AOF-off baseline swung so much between repetitions that the noise was larger than the thing being measured. I raised it to 100,000 requests per run and replaced the file, so the short run is not in results/. Even at 100,000 requests the AOF-off runs ranged from 27,562 to 44,434 ops/s (results/bench_fsync1.jsonl), which is worth keeping in mind below.

7. the session was interrupted

Work stopped partway through to reduce CPU load on the shared machine. The code, tests and loaded run survived, and on resuming I rebuilt both configurations with -j2, ran the two test suites one at a time, and redid the benchmarks as described above.

experiments

All experiments ran on an Apple M4 Pro (14 cores, 48 GB) over loopback TCP, with the release build (-O2, Apple clang 17), kvbench using 4 client threads, and 3 repetitions per configuration. I report medians of each metric over the three runs. scripts/run_benchmarks.py writes the raw results/bench_*.jsonl files, one JSON line per run, and scripts/plot_results.py builds results/summary.csv and the charts. The 1 minute load average during the reported runs was between 7.2 and 12.9, recorded per row.

kvbench is a closed-loop generator. Each connection keeps up to P requests in flight, topping the window back up as replies arrive. Latency is measured from the moment a request's batch was written to the moment its reply was parsed, so at P above 1 it includes time queued behind earlier requests in the same pipeline. That is the latency a pipelining client actually sees.

1. Pipeline depth. 50 connections, SET with 16 B values, depth 1 to 128, AOF off.
2. Connection count. Depth 1, SET, 1 to 256 connections, AOF off.
3. Workload mix. 50 connections, depth 16, 100,000 keys, pure SET against pure GET against a 50/50 mix.
4. Persistence with many clients. 50 connections, SET at depth 1 and 16, with AOF off and with appendfsync set to no, everysec and always.
5. Persistence with one client. 1 connection, depth 1, 100,000 SETs, the same four modes.
6. Active expiry. 200,000 persistent keys plus 200,000 keys sharing one deadline, no client touching them afterwards, DBSIZE sampled about every 50 ms.
7. AOF replay. Fill an AOF with 100,000, 400,000 and 1,600,000 SETs, then time replay at startup.

There is no side-by-side run against real Redis. The Python client in scripts/client_test.py plays the interoperability role, which checks correctness against the protocol as I understood it, not against Redis's behavior.

results

pipelining is the whole story for throughput

At 50 connections SET throughput went from 107,160 ops/s at depth 1 to 999,599 at depth 16 and 2,942,811 at depth 128, with no change to the server (results/bench_pipeline.jsonl). Throughput per server CPU second tracks wall-clock throughput within about 5 percent at every depth, 112,360 against 107,160 at depth 1 and 2,941,176 against 2,942,811 at depth 128. The single server thread was the bottleneck throughout, not the load generator.

That makes the per-command cost easy to read off. At depth 1 one SET costs the server about 1 / 112,360 s, or 8.9 µs of CPU. At depth 128 it costs about 0.34 µs. The hash table insert did not get 26 times cheaper. What changed is how many commands share each read(), send() and kevent().

The spread between runs grows with depth. At depth 64 the three runs gave 1,187,745, 2,276,344 and 2,631,208 ops/s, so the curve's exact shape above depth 16 should not be read closely. The p99 at depth 16 (1.74 ms) is also higher than at depth 32 (1.48 ms), which is the kind of wobble to expect from three runs on a shared machine.

Little's law checks out

With 50 connections at depth 1, the law predicts a mean latency of 50 / 107,160 s = 467 µs, and the measured p50 is 438.0 µs. At depth 128 there are 6,400 requests in flight, predicting 2.17 ms against a measured p50 of 2.05 ms. The law is about the mean and I am comparing against the median, so close agreement is what I would expect rather than a proof, but a load generator that miscounted in-flight requests or timestamps would fail this check.

The p99 stayed between 1.7 and 2.4 times the p50 at every depth, 4.39 ms against 2.05 ms at depth 128. No connection was being starved, which is what level-triggered, one-read-per-event dispatch is supposed to buy.

more connections do not help without pipelining

One connection did 40,193 ops/s at a p50 of 21.7 µs (results/bench_clients.jsonl). Its throughput per server CPU second was 111,111, almost three times the wall-clock rate, which says the server was idle most of the time, waiting for the client's round trip. Throughput rose to 115,697 at 8 connections and 120,252 at 16, and then stayed between 104,740 and 114,065 all the way to 256. Past that point each extra connection only adds queueing. The p50 at 256 connections was 2.18 ms, again what Little's law predicts (256 / 112,758 s = 2.27 ms).

So at depth 1 the server saturates at around 115,000 to 120,000 SETs per second on this machine, and the only way past that is to put more commands in each system call.

reads and writes cost about the same

At 50 connections and depth 16, GET did 1,198,850 ops/s, SET 1,072,646 and the 50/50 mix 1,046,355 (results/bench_workload.jsonl). The spread between repetitions is larger than the gap between workloads. SET alone ranged from 1,032,764 to 1,240,625 and GET from 940,801 to 1,269,237.

with many clients, appendfsync always was almost free

At 50 connections and depth 16, AOF off gave 1,217,265 ops/s and always gave 1,042,970, with no at 987,323 and everysec at 1,077,584 (results/bench_aof.jsonl). no coming out slowest, when it does strictly less work than always, shows that the differences between modes here are mostly noise. At depth 1 the four modes landed between 112,334 and 118,052.

The reason always is cheap is group commit. With 50 busy connections one fsync per loop iteration covers every write made in that iteration, so the fsync cost is divided among dozens of commands. Each AOF was 70.8 MB for 1.2 million SETs across the depth 1 and depth 16 runs, 59 bytes per logged command.

with one client, always nearly triples latency

On a single connection at depth 1 there is no one to share the fsync with. Median latency went from 22.2 µs with AOF off to 59.0 µs with always, and throughput from 40,120 to 14,165 ops/s (results/bench_fsync1.jsonl). Against the no mode, which also writes the file but never fsyncs on the reply path, the gap is 26.5 µs, so a plain fsync on this machine costs a few tens of microseconds.

That is cheap because macOS fsync pushes data to the drive but does not flush the drive's own write cache. F_FULLFSYNC would, at a much higher cost, and I did not measure it. So always here protects against a process crash, which the SIGKILL test demonstrates, and not against power loss.

Writing the AOF without an fsync on the reply path (no and everysec) put the p50 at 30.7 to 32.5 µs, 8 to 10 µs above AOF off, which I would attribute to the extra write() per iteration. I hold that loosely. In the first repetition AOF off measured 28.0 µs and no 29.5 µs, almost no gap, and the per-run medians of the AOF-off mode ranged from 21.5 to 28.0 µs.

active expiry is fast when the machine is quiet

All 200,000 TTL keys shared one deadline and no client touched them, so only active expiry could remove them. DBSIZE was still 400,000 at 63 ms after the deadline, 294,740 at 116 ms and 200,000 at 171 ms, and stayed there (results/bench_expiry.jsonl). The server's shutdown log confirms 200,000 keys were expired actively.

That is two 10 Hz cron cycles, the first removing about 105,000 keys and the second the remaining 95,000. Each cycle is allowed at most 25 ms. If each one ran to its budget, which is my inference and was not measured, deleting a sampled expired key costs about 0.24 µs.

Under heavy load the same experiment took until 791 ms after the deadline to finish, in eight uneven steps, one of which removed only 20 keys (results/run1_loaded/bench_expiry.jsonl, load average 222). The cycle budget is wall-clock time, so a server that gets descheduled in the middle of a cycle burns its budget without doing work. That bounds how long clients wait, but it means expiry slows down exactly when the machine is busiest.

replay grows faster than linearly

Replaying 100,000 SETs (5.9 MB) took 24 ms, 400,000 (23.6 MB) took 121 ms and 1,600,000 (94.4 MB) took 854 ms (results/bench_replay.jsonl). Per command that is 240 ns, 302 ns and 534 ns. Wall time and CPU time agree to within a few milliseconds (850 ms of CPU for the largest), so this is not waiting on disk.

My guess, which I have not tested, is some mix of two things. The hash table and its node allocations fall out of cache as the keyspace grows, and std::unordered_map's all-at-once rehashes get more expensive as the table grows. Calling reserve before replay would test the second half of that.

The loaded run makes a separate point about wall time. At load average 251 to 257, the three 1.6 million command replays took 1,685, 4,785 and 3,601 ms of wall time but 1,248, 1,241 and 1,233 ms of CPU (results/run1_loaded/bench_replay.jsonl). The server's own work barely changed. The scheduler added the rest. That is why the startup log prints both.

measurement conditions mattered more than any tuning I did

Load cost the pipeline experiment 5.5 to 10 times its throughput and 37 to 55 times its p99 (the loaded chart above). Nothing I changed in the code moved any number by that much.

what I would change

replace the standard hash map with an incremental-rehash table

Redis's dict keeps two tables during a resize and moves a few buckets per operation, so no single command pays for the whole rehash. std::unordered_map rehashes everything at once, which is a latency spike for every client each time the table doubles. The replay numbers suggest something that scales badly is already visible at a million keys, and an open-addressing table would also drop the per-key node allocation. The dense TTL vector relies on node stability, so it would need to store keys or indices instead of pointers.

move the everysec fsync off the loop

It runs inline in the cron now, so a slow disk shows up as client latency. Redis does it on a background thread.

add AOF rewrite

The file only grows, including records for keys that were later deleted or expired. A rewrite that snapshots the live keyspace into a fresh file, with a fork() for copy-on-write, is the v2 milestone.

measure against real Redis

Installing redis-server and running the same kvbench sweep against both would say whether these numbers are good, not just consistent with each other. Running redis-benchmark and redis-cli against kvd would be the real interoperability test that the Python client only stands in for.

build and run the epoll backend

It is written against the same interface but has never been compiled. Until it runs on Linux, the portable poller is a claim, not a result.

benchmark on a quiet machine from the start

I lost the first full benchmark run to other jobs. Recording load and server CPU per row from the first run, and checking them before trusting a number, would have saved a rerun.

reproducibility

Build. This needs CMake 3.24 or newer, Ninja and a C++20 compiler, and GoogleTest is fetched at configure time. The asan preset uses Homebrew LLVM at /opt/homebrew/opt/llvm/bin/clang++ because Apple clang 17's ASan runtime hangs on macOS 26.5.

cd projects/02-kvd-resp-server
cmake --preset release && cmake --build --preset release   # -O2, build/release/
cmake --preset asan    && cmake --build --preset asan      # Debug + ASan/UBSan, build/asan/

Tests. Each preset runs the 49 GoogleTest cases and the end-to-end Python client test.

ctest --preset release
ctest --preset asan
python3 scripts/client_test.py --server build/release/kvd   # e2e test alone

Run the server and poke it.

build/release/kvd --port 6379 --aof appendonly.aof --appendfsync everysec
printf 'SET a 1\r\nGET a\r\n' | nc 127.0.0.1 6379

Benchmarks and plots. The runner writes one JSON line per run to results/bench_*.jsonl, with the load average and server CPU seconds in each row. The reported numbers used --threads 4.

build/release/kvbench -p 6379 -c 50 -P 16 -n 1000000 -w set   # one run, human readable
python3 scripts/run_benchmarks.py --reps 3 --threads 4
python3 -m venv .venv && .venv/bin/pip install matplotlib
.venv/bin/python scripts/plot_results.py                      # results/summary.csv and PNGs

Code is in projects/02-kvd-resp-server, with the server in src/, the load generator in bench/loadgen.cpp, the tests in tests/, the benchmark runner, plotter and end-to-end client in scripts/, the raw results in results/ (and the loaded run in results/run1_loaded/), the invariants and trade-offs in DESIGN.md, and the build log in DEVLOG.md.

The code is about 1,700 lines of C++ in src/ and include/, headers and comments included, plus about 930 lines of GoogleTest. There are 36 tests, and on 2026-09-27 an independent clean release rebuild passed all 36, including the tests that fork a writer process and kill it. The AddressSanitizer plus UBSan build also passed all 36 in my own run, but it was not rerun independently, and neither were the benchmarks.

The headline result is the crash testing. A child process writes and acknowledges each commit over a pipe, and the parent kills it with SIGKILL. In the verified release run, 12 random kills covering 9,721 acknowledged operations lost none of them. A second test aims kills into the middle of a large checkpoint, and 6 of its 10 killed rounds landed in the window where the journal was complete but the pages were only partly written in place, and were repaired from the journal. A third test crashes the process at each of 5 named steps of the checkpoint protocol and then crashes it again at the same step inside recovery. Every one of these reopens the database, walks the whole tree checking its invariants, and compares it to a model. These are process crashes. The OS page cache survives them, so they say nothing about power loss, which I did not test.

The second story is what durability costs on a Mac. With no sync a commit took 1.2 µs at the median. With fsync it took 24 µs, and with F_FULLFSYNC, the only call on macOS that makes the drive flush its own cache, it took 4.01 ms, about 3,300 times the unsynced cost. Batching 256 puts into one F_FULLFSYNC commit brought throughput from 243 to 59,450 puts per second. All benchmark numbers come from a shared Apple M4 Pro that was running other jobs, with the 1 minute load average recorded in every row (3.9 to 6.3 for the reported runs). They are indicative, not a clean benchmark. The 1M key database also fits in memory, so the throughput numbers measure CPU and system call cost, not the disk.

Code is in projects/01-kvdb-btree-wal-engine. The argument of this post is that a small storage engine becomes trustworthy through its crash tests rather than its design document, and that on macOS "durable" is a choice between three very different prices. Skip to problems for what went wrong.

table of contents

• what I wanted to build
• theory
• architecture
• implementation
• problems
• experiments
• results
• what I would change
• reproducibility

what I wanted to build

I wanted a storage engine I understand all the way down, from bytes in a 4 KiB page, to a page in a buffer frame, to a frame pinned by a tree operation, to a commit in a log record, to a crash that loses nothing it promised to keep.

The v0 scope was fixed up front. A pager over one file. A buffer pool with LRU eviction and pin counts. A B+ tree with insert, point lookup, range scan and lazy delete. A WAL with fsync and redo recovery on startup. A REPL. A crash test that really kills a process. And benchmarks against std::map and SQLite at 1M keys.

The bar was correctness first. Every claim about durability had to be backed by a test that kills a process with SIGKILL and checks what survives against a model, and every performance number had to come from a run saved under results/. Transactions, concurrency, page checksums, real delete and an LSM variant are later milestones and are not in this post.

theory

B+ trees on disk

A disk-based B+ tree stores sorted keys in fixed-size pages. Internal pages hold separator keys and child pointers. Leaves hold the data and are linked left to right, so a range scan is one descent followed by a walk along the leaf chain. The point of the shape is fanout. In kvdb an internal cell for a 16-byte key is 24 bytes plus a 2-byte slot, so a 4 KiB page holds about 150 children, and 1M keys fit in a tree of height 4. A lookup touches four pages and does a binary search inside each.

Splits decide how full the leaves are. A textbook split cuts a full node in half, and under random inserts leaves settle at about 69% full (ln 2). Under ascending inserts the left half is never touched again and stays half empty, so the append pattern deserves a special case.

the write-ahead rule

Durability rests on one rule. A change is committed once a description of it is on stable storage, and that description must get there before the data pages change on disk. After a crash, the committed state is rebuilt from the last consistent page file plus the log.

The general design is ARIES, which is STEAL (a dirty page may be written back before its transaction commits) and NO-FORCE (pages need not be written at commit). It is flexible and it needs both undo and redo logging, page LSNs to know which changes a page already contains, and a three-pass recovery.

I took a simpler corner of the design space. kvdb's buffer pool is NO-STEAL, meaning dirty pages never reach the page file except at a checkpoint. The page file on disk is therefore always exactly the state as of the last checkpoint, and the log only needs logical redo records ("put k v"). There is no undo, no page LSN, and recovery is "load the snapshot, redo the operations after it".

making the checkpoint atomic

The remaining problem is the checkpoint itself. It overwrites many pages in place, and a crash halfway leaves a parent from the new generation pointing at a child from the old one, which is a corrupt tree. The fix is a double write. Copy every page the checkpoint will overwrite to a side file, sync it, and only then write in place. A crash before the journal is complete leaves the page file untouched. A crash after it leaves a complete journal that recovery replays. SQLite's rollback journal and InnoDB's doublewrite buffer do versions of the same thing.

The subtle part is deciding whether a journal is complete. Writing the header last is not enough, because without a barrier between them the header can reach the disk before the body. So kvdb accepts a journal only if the magic matches, the file size equals exactly the size the header's page count implies, and a CRC over the entire body matches. A journal cut short at any byte fails one of those checks.

three durability levels on macOS

On macOS, fsync() pushes data to the drive but does not make the drive flush its volatile write cache. Only fcntl(fd, F_FULLFSYNC) does that. So there are three levels, and kvdb exposes all of them as SyncMode.

kNone survives a process crash because a pwrite that has returned is in the kernel's page cache, and the kernel writes it out whether or not the process is still alive. That matters for reading the crash tests correctly, as I explain below.

architecture

A database at path P is three files. P is the page file, where page 0 is a meta page and every other page is a B+ tree node. P-wal holds one CRC-framed record per committed batch since the last checkpoint. P-journal is empty except while a checkpoint runs.

The meta page holds the magic number, format version, root page id, page count, tree height and checkpoint LSN. It lives in the buffer pool like any node, so a root split or a page allocation is checkpointed atomically with the nodes it refers to. There is no separate "superblock write" to get wrong.

Nodes are slotted pages. A 16-byte header is followed by an array of 2-byte slot offsets sorted by key, growing toward the end of the page, while cells (klen, vlen, key, value) grow backward from the end.

A leaf's link is its right sibling. An internal node's link is its leftmost child, and each cell's value is a 4-byte child id. Keys are capped at 128 bytes and values at 512, which makes a worst-case cell plus slot 646 bytes, so any overflowing node holds at least 6 cells and a byte-balanced split always produces two halves that fit. That cap is what lets v0 skip overflow pages.

A WAL record is u32 crc32 | u32 len | u64 lsn | payload, and the payload is the WriteBatch encoding itself, so logging a batch is one copy and one pwrite. The journal is u64 magic | u32 page count | u32 crc32(body) followed by (u32 page id, 4096 bytes) per page.

Three invariants carry the design. Between checkpoints the page file is byte-identical to the last completed checkpoint. A batch is committed if and only if its WAL record is intact on disk. And replaying a batch over a page file that already contains some or all of it gives the same result, because putting or deleting a whole key is idempotent.

implementation

the write path

A write is four lines, and the order is the whole durability argument.

void DB::write(const WriteBatch& batch) {
  if (batch.count() == 0) return;
  maybe_checkpoint(true);
  const uint64_t lsn = next_lsn_++;
  wal_->append(lsn, batch.rep(), opts_.sync);  // commit point
  apply(batch.rep());
  applied_lsn_ = lsn;
  ++stats_.commits;
}

put and del are one-op batches, and deleting a missing key is not logged at all. Wal::append frames the record, computes the CRC with the ARMv8 CRC32 instructions (8 bytes per instruction, with a table fallback that a test checks agrees), issues a single pwrite, and syncs according to the mode. The WAL file is grown in 4 MiB ftruncate steps, so most appends land inside the file and the file has a zero-filled tail that replay must treat like garbage.

replay stops at the first thing it does not trust

while (off + kHeaderSize <= data.size()) {
  const uint32_t crc = load<uint32_t>(data.data() + off);
  const uint32_t len = load<uint32_t>(data.data() + off + 4);
  if (len > data.size() - off - kHeaderSize) break;          // torn tail
  if (crc32(data.data() + off + 4, 12 + len) != crc) break;  // corrupt
  const uint64_t lsn = load<uint64_t>(data.data() + off + 8);
  if (lsn <= last_lsn) break;                                // stale bytes
  fn(lsn, std::string_view(data.data() + off + kHeaderSize, len));
  last_lsn = lsn;
  off += kHeaderSize + len;
}

A torn write, a flipped byte and the preallocated zero tail all end replay the same way, and the file is then truncated at the last good record so new records are never appended after garbage. The WAL tests cut the log at every possible byte offset of the last record and check that replay keeps exactly the records before it.

the buffer pool

The pool is one page-aligned allocation split into frames, an unordered_map page table, a free list and a std::list LRU. The key design choice is that the LRU list contains only frames that are both unpinned and clean. Eviction is then lru_.back(), and it structurally cannot pick a pinned or dirty page. If every frame is pinned or dirty, fetch throws rather than corrupting anything. PageGuard is an RAII pin that carries the dirty bit to unpin, so a tree operation cannot forget to release a page.

Because the pool is NO-STEAL, dirty pages accumulate until a checkpoint. The DB layer checkpoints when the WAL passes 64 MiB, at clean close, at the end of recovery, and whenever the dirty count comes within 64 frames of capacity. That last trigger is where NO-STEAL shows its cost, as the small-pool benchmark shows.

the append split

On overflow the tree materializes the node's entries plus the new one and splits by bytes rather than by count, since cells vary in size. One special case does a lot of work.

if (idx == n - 1 && node::link(leaf.data()) == kNoPage) {
  // Appending past the end of the rightmost leaf (ascending inserts): leave
  // the old leaf full and start a new one. Gives ~100% fill instead of 50%.
  m = n - 1;
} else {
  m = std::clamp(byte_midpoint(ents), 1, n - 1);
}

The first key of the right half is pushed into the parent, and internal nodes split the same way, except the middle key moves up rather than being copied. BTree::verify() walks the whole tree checking each node's format, every key against the range its parent's separators allow, that all leaves are at the same depth, and that the sibling chain visits them in order. Every crash test calls it after reopening.

the checkpoint

tree_->set_checkpoint_lsn(applied_lsn_);
const auto pages = pool_->dirty_pages();        // sorted by page id, meta included
// stream (id, page) entries to the journal body in 1 MiB chunks, CRC as we go
failpoint::hit("ckpt.before_journal_header");
pwrite_all(journal_fd_, hdr, kJournalHeader, 0);
sync_fd(journal_fd_, mode);
failpoint::hit("ckpt.journal_synced");
for (size_t i = 0; i < pages.size(); ++i) {
  pager_->write_page(pages[i].first, pages[i].second);
  if (i == pages.size() / 2) failpoint::hit("ckpt.mid_data");
}
pager_->sync(mode);
failpoint::hit("ckpt.data_synced");
// then truncate the WAL (failpoint ckpt.wal_reset) and the journal

The failpoint::hit calls are no-ops unless a test has armed that name, in which case the process calls _exit(77) on the spot, with no destructors and no flushing.

Recovery is the mirror image. If the journal is valid, every page in it is written to its home location and synced, then the journal is truncated. If it is not valid, the crash happened before any in-place write, so the page file is still the previous checkpoint and the journal is simply discarded.

batches bigger than the pool

One case needed care. A single batch can dirty more pages than the pool holds. apply then checkpoints in the middle of the batch, but without truncating the WAL and without moving the checkpoint LSN past the previous batch. After a crash, recovery redoes the whole batch on top of a page file that already contains part of it, which is safe because whole-key puts and deletes are idempotent.

crash tests with real processes

The random-kill test forks a child that opens the database with a 256-page pool and a 64 KiB WAL threshold, so checkpoints happen constantly, and runs a deterministic stream of operations. Most are single puts, about one in five is a delete, and every 25th is a 20-op batch, to check that batches are atomic. After each write() returns, the child writes the op index to a pipe.

while (seen < kill_after && ::read(fds[0], &ack, sizeof ack) == sizeof ack) {
  last = ack;
  ++seen;
}
::kill(pid, SIGKILL);
// Drain acks the child managed to send before dying: those were committed.
while (::read(fds[0], &ack, sizeof ack) == sizeof ack) last = ack;

The parent then reopens the database, runs verify(), and requires the contents to equal a std::map model after every acknowledged op, or after exactly one more. The extra one is the op that may have reached the WAL between write() returning and the ack being sent. Anything else fails, including a lost acknowledged op, a resurrected delete and a half-applied batch. The test runs 12 rounds on the same file, so each round also recovers on top of the previous recovery.

The flock that stops two processes opening the same database dies with the process, which is what lets the parent reopen immediately after a kill.

problems

1. the random kills never hit a checkpoint

The random-kill test was supposed to be the checkpoint test too. It was not. Checkpoints are short compared with the time between them, so a kill at a random moment almost always lands in the write path. In the final release run, 0 of 12 rounds found a journal to replay. The test was passing without ever exercising the journal.

I found this because the test prints how many rounds recovered from a journal, and the number was zero. I added two tests to cover the gap. KillNineDuringLargeCheckpoint builds a checkpoint big enough to aim at (6,000 mixed ops plus 50,000 new keys, with an 8,192-page pool), runs one uncounted calibration round that reports how long the journal and data phases take, and then kills 10 rounds at random delays between halfway through the journal write and the end of the checkpoint. In the release run the calibration checkpoint took 6.7 ms, 1.4 ms of it writing the journal and 5.4 ms writing pages in place and truncating. 6 of the 10 kills landed after the journal was complete and were repaired from it. The rest landed before it was complete and were correctly discarded.

That hit rate depends on timing. Under ASan the journal phase took 39.4 ms instead of 1.4 ms, and only 2 of 10 kills landed in the repair window. So the aimed test is probabilistic, and I did not want the protocol's correctness to rest on luck. The deterministic CheckpointCrash tests arm a failpoint at each of five steps (before the journal header, after the journal sync, halfway through the in-place writes, after the data sync, after the WAL reset), crash there, then arm the same failpoint and open the database, so recovery's own checkpoint crashes at the same step. A third open must produce exactly the model. All five pass in both builds.

2. SQLite's full sync was faster than a full sync

With synchronous=FULL and fullfsync=ON, SQLite committed 1,006 times per second against kvdb's 247 with F_FULLFSYNC. That looked like kvdb doing something wrong, until I compared latencies. A real F_FULLFSYNC on this machine takes about 4 ms, in both kvdb and a bare pwrite loop, while SQLite's median commit took 0.69 ms. SQLite cannot be issuing a full flush on every commit.

My hypothesis is that Apple's system SQLite, which is the copy the benchmark links, uses the lighter F_BARRIERFSYNC for WAL commits, which orders writes without waiting for the cache flush. I have not verified it, with fs_usage or by building SQLite from source. Until I do, the F_FULLFSYNC column of the durability comparison is not like for like, and I mark it that way in the figure below.

3. a comment claimed a 14x speedup nobody had measured

A comment in wal.h said that on APFS an append which extends the file costs about 14 times more than a write inside a preallocated file. I had not measured that, so I wrote the walgrow experiment. It did not hold up. Preallocation made a 128-byte append 1.70 times faster with no sync, 1.09 times faster with fsync, and made no difference with F_FULLFSYNC (247 versus 248 appends per second). I kept preallocation, since it costs nothing and replay already handles a garbage tail, and corrected the comment to cite the file.

4. the machine was busy

The machine was shared with a dozen other builds. An earlier benchmark run, later interrupted, happened at a 1 minute load average of about 256 on 14 cores, and its wall-clock numbers were roughly an order of magnitude below what I measured later at a load of 4 to 6. I threw it away. Every benchmark row now records ops per CPU second and the load average, the build script uses -j2, and I ran one benchmark or test binary at a time. Even so, results/machine.txt shows a 15 minute load average of 42 when the reported run started, so the machine had been very busy shortly before.

5. Apple clang's sanitizer runtime hung

An early UBSan-only run under Apple clang 17 stalled inside the large-checkpoint crash test and never finished, and the sanitizer runtime hung at startup on this macOS version more generally. I moved the sanitizer preset to Homebrew LLVM 23, which runs ASan and UBSan together without trouble. Switching compilers then surfaced two smaller problems. Three files used errno without including <cerrno> and had only compiled because an Apple header pulled it in transitively. And Homebrew LLVM's DWARF 5 debug info made Apple's linker print a harmless warning for every object file, burying real warnings, so the sanitizer build now passes -gdwarf-4 for upstream Clang.

experiments

All runs were on an Apple M4 Pro (14 cores, 48 GB) running macOS 26 (Darwin 25.5.0), release build at -O2, with the 1 minute load average recorded next to each row. Keys are 16 bytes (user%012d, so lexicographic order is numeric order) and values are 100 bytes.

1. Correctness. The 36 tests in 8 suites, in a release build and in an ASan plus UBSan build. The release suite was rerun independently on 2026-09-27 and passed. The sanitizer suite was not rerun.
2. Throughput and latency at 1M keys, 3 repetitions, medians reported. For each engine, insert 1M keys in sequential or shuffled order, then read all 1M in sequential and shuffled order, then scan everything. Every operation is timed individually with steady_clock, so p50, p99 and p99.9 are exact. The engines are std::map<std::string, std::string>, kvdb with a 65,536-frame (256 MiB) pool, kvdb with a 4,096-frame (16 MiB) pool, and SQLite 3.51.0 with journal_mode=WAL, synchronous=OFF, a 256 MiB cache, prepared statements and a WITHOUT ROWID table. kvdb ran with SyncMode::kNone, so both disk engines write their log on every commit and neither syncs.
3. Durability cost. 2,000 random puts, one per commit, under each sync mode for kvdb and SQLite, then kvdb with F_FULLFSYNC and 1, 4, 16, 64 or 256 puts per WriteBatch. One run each.
4. WAL preallocation. 3,000 raw 128-byte pwrite appends, each followed by no sync, fsync or F_FULLFSYNC, into a file that either grows with each append or was sized in advance.

What the benchmarks do not measure matters as much. With the 256 MiB pool, the whole database is resident. The pool hit rate was 1.000 for both fills, and the page file was 122.1 MiB after the sequential fill and 170.9 MiB after the random one. Even the 16 MiB pool misses only into the OS page cache, since the machine has 48 GB and the file was just written. No benchmark here reads from the SSD. The throughput numbers are the cost of CPU work and system calls, and the only experiment where the storage device is on the critical path is the durability one.

results

crash safety

All 36 tests passed in 2.5 s in the release build (results/tests_release.txt) and in 34.7 s under ASan plus UBSan (results/tests_asan.txt). The sanitizer run's random-kill test covered 9,693 acknowledged ops with 2,212 records redone, the different count coming from different kill timing, and it also lost none.

It is worth being exact about the scope. The random-kill test runs with SyncMode::kNone, and with kNone the checkpoint's journal "sync" is a no-op too. That is fine for a process crash, where every returned pwrite is already in the kernel's page cache and ordering between files is preserved by that cache. It would not be fine for power loss, where unsynced data can vanish and writes can reach the disk out of order. None of these tests pulls the plug, so the claim is that kvdb loses no acknowledged write to a process crash at any point I could aim at, including inside checkpoints and inside recovery. The protocol is designed for power loss when run with kFullFsync, but that is a design argument, not a test result.

what durability costs

Each step costs one to two orders of magnitude. Going from no sync to fsync cut kvdb's commit rate by a factor of 18, and going from fsync to F_FULLFSYNC cut it by another 154. At the median, a truly durable single-put commit on this Mac costs about 4 ms, which is about 3,300 times the unsynced commit.

The fsync row is the uncomfortable one. It is the default in kvdb, as it is in SQLite on macOS, and it costs 24 µs, which feels like durability. It is not durability against power loss, because the data may still be sitting in the drive's cache. The honest price of that guarantee is the 4 ms row.

batching is the only way to pay less

With F_FULLFSYNC on every commit, putting more operations in each WriteBatch scales almost perfectly. Throughput was 243, 980, 3,960, 15,485 and 59,450 puts per second at 1, 4, 16, 64 and 256 puts per batch, while the median commit stayed at 4.00 to 4.05 ms throughout. The flush is the cost and it is paid once per batch, so the 256-put batch is about 245 times faster than single puts. A WriteBatch is one WAL record, one pwrite and one sync, and it is atomic, which the random-kill test checks with its 20-op batches.

This is group commit done by hand, because v0 is single-threaded. Letting concurrent committers share one flush automatically is on the v2 list.

throughput when everything is in memory

These are medians of 3 runs from results/summary_main.csv. Given that none of it touches the SSD, I read them as follows.

kvdb beats a red-black tree on reads

Random gets ran at 1.25M per second in kvdb against 0.84M for std::map, with a p50 of 750 ns against 1,042 ns. A red-black tree filled in random order scatters 1M nodes across the heap and follows about 20 pointers per lookup, many of them likely cache misses. The B+ tree touches 4 pages and binary-searches inside each. I did not profile this, so the explanation is a plausible one rather than a measured one. The full scan shows the same effect more strongly, 61M keys per second for kvdb against 11M for std::map after a random fill, because a leaf chain is sequential memory and a red-black tree walk is not.

kvdb loses to a red-black tree on writes

Sequential puts were 5.3 times slower than std::map and random puts 2.5 times slower. Each put encodes a batch, computes a CRC, issues one pwrite system call to the WAL and may split nodes. The system call per put is the obvious target, and it is the first thing on the change list below.

the gap to SQLite is mostly generality

kvdb's random puts were 8.8 times faster than SQLite's and its random gets 7.3 times faster. I would not read that as kvdb being a better storage engine. SQLite here runs prepared statements through its bytecode VM, encodes records, and maintains a far more general B-tree, while kvdb has a specialized key-value path. It is a comparison of a narrow tool against a general one, both in memory.

the append split works

Sequential fill produced 30,304 leaves at 99% fill. Random fill produced 43,324 leaves at 69% fill, which is the textbook ln 2 value for random inserts. Both trees have height 4, and the random-fill file is 40% larger for the same data.

NO-STEAL turns memory pressure into checkpoints

With a 16 MiB pool, random fill needed 213 checkpoints against 3 for the 256 MiB pool, and ran 2.1 times slower (207K versus 439K puts per second, pool hit rate 0.832). Random inserts dirty leaves all over the tree, and a NO-STEAL pool cannot write any of them back without a full checkpoint. Sequential fill barely noticed (9 checkpoints, 719K versus 728K per second, hit rate 0.988), because it only ever dirties the rightmost path.

WAL preallocation is a small win

Whatever extending the file costs on APFS, it shows up when nothing else is expensive and disappears under a 4 ms flush.

what I would change

make the pool STEAL and the log physiological

NO-STEAL made recovery easy to reason about, and the crash tests are simple partly because of it. But the small-pool run shows that it converts memory pressure directly into checkpoints, 213 of them for 1M random puts, and it rules out transactions larger than memory. With concurrent transactions coming in v1 and v2, the right design is page LSNs and physiological log records, which let the pool evict dirty pages and let checkpoints be fuzzy.

checksum every page

Today a torn or bit-flipped page in the data file would be read silently. The WAL and the journal carry CRCs, and pages should too, with torn-page detection at read time.

buffer the log instead of a pwrite per put

The write path is where kvdb loses most to std::map, and one system call per put is the obvious cause. An in-memory log buffer flushed at commit boundaries would keep the same durability semantics for synced modes and remove most of the system calls for kNone.

real delete

Lazy delete was the right cut for v0, since lookups and scans stay correct and deleting everything then reinserting works. But space is never reclaimed, so a delete-heavy workload wastes space without bound. Merge, redistribution and a free-page list are in v3.

settle the SQLite question and test power loss

I would verify SQLite's sync behavior with fs_usage or a source build, so the durability comparison is like for like. And the crash tests should gain a mode that simulates power loss, for example by interposing on writes and dropping everything after the last completed sync, since the process-crash tests cannot say anything about that case.

benchmark larger than memory, on a quiet machine

Every throughput number here is an in-memory number. A database several times larger than RAM, with the page cache dropped, would make the buffer pool go to the SSD and make the random-read results reflect I/O. It should run on a machine with nothing else on it.

reproducibility

Build. This needs CMake 3.24 or newer, Ninja and a C++20 compiler, and GoogleTest is fetched at configure time. SQLite is optional, and the macOS SDK's copy is found automatically and enables the SQLite baseline. The asan preset uses Homebrew LLVM at /opt/homebrew/opt/llvm/bin/clang++ because Apple clang 17's sanitizer runtime hangs on this macOS version.

cd projects/01-kvdb-btree-wal-engine
cmake --preset release            # -O2, build/release
cmake --build --preset release -j2
cmake --preset asan               # ASan + UBSan, build/asan
cmake --build --preset asan -j2

Tests.

./build/release/kvdb_tests        # 36 tests, a few seconds
./build/asan/kvdb_tests           # the same 36 under ASan + UBSan
ctest --preset release            # or through CTest

Benchmarks and plots. The script writes every raw CSV and log to results/, with the load average in each row, and takes about 5 minutes. The plotting script needs matplotlib.

scripts/run_bench.sh 1000000 3
python3 -m venv .venv && .venv/bin/pip install matplotlib
.venv/bin/python scripts/plot_results.py      # results/summary_main.csv and PNGs
./build/release/kvdb_bench --exp=sync --n=2000 --out=/tmp/sync.csv   # one experiment

Code is in projects/01-kvdb-btree-wal-engine, with the public API in include/kvdb/db.h, the engine in src/, the crash tests in tests/recovery_test.cc, the benchmark harness in bench/bench.cc, the formats, invariants and trade-offs in DESIGN.md, and the build log in DEVLOG.md.

The result splits in two. Among the 50 most liquid USDT pairs on Binance, yesterday's losers do rank above yesterday's winners the next day. The holdout mean rank information coefficient of one day reversal is 0.019 with a Newey-West t of 2.24, which clears the pre-registered bar of 2 by a small margin. That hypothesis is supported. The same signal traded as a dollar-neutral portfolio has a holdout gross Sharpe of -0.78 and a net Sharpe of -4.06. It lost money before paying a single basis point of cost. The second hypothesis, that the portfolio picked in development makes money after 15 bps per trade, is not supported. That portfolio earned a net Sharpe of 0.08 with a probabilistic Sharpe ratio of 0.55, against a required 0.95.

So the honest answer to "can we predict markets" from this study is yes, a little, at the level of daily cross-sectional ranks, and no, not in a way that paid after costs in the pre-registered test. There is also a post-hoc observation that the ridge and gradient boosting models did much better in the holdout than the model my selection rule picked. I explain below why I cannot claim that as a result.

Code, the pre-registration, the lock file and every result file are in projects/09-crypto-reversal-preregistered. Every number in this post comes from a file in its results/ directory, named where it is used.

Reading note. If you only want the verdicts, read results. The most useful sections for anyone building a backtest are problems and the selection failure at the end of experiments.

table of contents

• what I wanted to build
• theory
• architecture
• implementation
• problems
• experiments
• results
• what I would change
• reproducibility

what I wanted to build

I wanted a research pipeline I could defend line by line, applied to one falsifiable question. Does yesterday's loser beat yesterday's winner tomorrow among liquid crypto pairs (short-term reversal), and if so, is any of that left after paying to trade it?

The question was chosen to be boring. Reversal is one of the oldest documented cross-sectional effects in equities, it needs nothing but prices, and it has an obvious enemy in trading costs because it turns the book over almost completely every day. A study where the answer could plausibly go either way, on free data, with a known failure mode, is a good test of the process rather than of my ability to find a clever signal.

The process was the point. Most backtests I have read fail in the same four ways. They peek at the future inside a feature or a universe definition. They quietly drop the coins or stocks that died. They re-use the test period while iterating, so it stops being a test. And they report the best of many variants without saying how many variants there were. I wanted each of those to be either impossible by construction or caught by a test.

the pre-registration

PREREGISTRATION.md was written before any data was downloaded and fixes everything a result could be tuned against. Nothing in it may change after the holdout except an amendments section, which is still empty.

The two hypotheses and their decision rules, in full.

The file also fixes the data source, the point-in-time universe rule, the 9 features, the target and execution timing, all 9 models and both portfolio constructions (18 trials), the cost of 15 bps per unit of one-way turnover, the sensitivity grid of 0, 5, 10 and 25 bps, and the selection rule. The rule is to take the trial with the highest development net Sharpe at 15 bps, breaking ties by mean IC.

The periods are fixed too.

2018-01-01        2020-01-01                       2024-07-01              2026-08-31
    |  warm-up         |  development, 9 half-year folds  |  holdout, run once       |
    |  features and    |  walk-forward, all selection     |  refit once on data      |
    |  training only   |  5 day embargo per fold          |  up to 2024-06-25        |

The holdout script writes a lock file containing the SHA-256 of the pre-registration before it computes anything, and refuses to run if the lock exists. While writing this post I recomputed the hash of the current PREREGISTRATION.md, and it matches the one in results/holdout/LOCK.json (0f38b396...79f6dc), so the file has not been edited since the holdout ran.

An independent reviewer who did not build the project later checked the same thing from the other side. The hash of PREREGISTRATION.md still equals the one recorded in the lock when the holdout started, at 03:07 UTC on 2026-09-27. The file was last modified at 22:30:43 US Eastern the evening before, 35 seconds before the download log was created and about 37 minutes before the holdout ran, so it was fixed before any price was seen. The project was not yet in git at that point, so this rests on file timestamps plus the lock hash rather than on commit history. The same reviewer reran the development study and recomputed the holdout from the frozen selection, and both matched the committed files exactly (see reproducibility).

A negative answer was an acceptable outcome, and half of one is what I got.

theory

cross-sectional prediction and the information coefficient

The study never predicts whether the market goes up. Each day it ranks the coins in the universe by a forecast and asks whether that ranking lines up with the ranking of next-day returns. The daily Spearman correlation between the two is the information coefficient.

$$\mathrm{IC}_t = \operatorname{corr}\big(\operatorname{rank}(f_{i,t}),\ \operatorname{rank}(r_{i,t+1})\big)$$

where $f_{i,t}$ is the forecast for coin $i$ at the close of day $t$ and $r_{i,t+1}$ is its return over the next day. A mean IC of 0.02 sounds like nothing. Grinold's fundamental law of active management says the information ratio of a strategy scales like $\mathrm{IC}\sqrt{B}$, where $B$ is the number of independent bets per year, and 50 names a day for 365 days a year is a lot of bets. The catch is that the law ignores costs, and it treats every bet as equally sized. Both assumptions turn out to matter here.

why reversal might exist

In equities, short-term reversal is usually explained as payment for liquidity. Someone who has to trade pushes the price, and whoever absorbs that flow is paid by the rebound. Part of it is also bid-ask bounce, where a close that prints at the bid looks like a loss and the next close at the ask looks like a gain. Bounce should be small here, because Binance daily closes on liquid pairs are last trades on books with spreads of a few basis points.

what costs do to a daily book

A daily-rebalanced long-short book with average turnover $T$ per day (the sum of absolute weight changes) and a one-way cost $c$ loses $T \cdot c \cdot 365$ per year. At 15 bps and a turnover of 1.3, which is typical for a 1 day reversal signal, that is about 71% per year. The holdout confirms the arithmetic to the decimal. The reversal portfolio's annualized gross return was -17.2% and its net return -89.0%, a gap of 71.8 points, and $1.311 \times 0.0015 \times 365 = 0.718$ (results/holdout/holdout_report.json). Any gross edge has to beat that number before it is worth anything.

statistics that account for how the numbers were made

Daily ICs are autocorrelated when a signal changes slowly. A 20 day momentum ranking barely moves day to day, so consecutive ICs are not independent draws, and the naive t-statistic overstates confidence. I use a Newey-West t-statistic, which replaces the variance of the mean with a Bartlett-weighted sum of autocovariances up to lag $L = 5$.

$$\hat\sigma^2_{NW} = \hat\gamma_0 + 2\sum_{l=1}^{L}\Big(1 - \frac{l}{L+1}\Big)\hat\gamma_l, \qquad t = \frac{\bar x}{\sqrt{\hat\sigma^2_{NW}/n}}$$

For portfolio returns I use the probabilistic Sharpe ratio (Bailey and López de Prado, 2012). It is the probability that the true Sharpe exceeds a threshold $SR^*$, given the observed per-period Sharpe $\widehat{SR}$, the sample length $n$, and the skew $\gamma_3$ and kurtosis $\gamma_4$ of the returns.

$$\mathrm{PSR}(SR^*) = \Phi\left(\frac{(\widehat{SR} - SR^*)\sqrt{n-1}}{\sqrt{1 - \gamma_3\widehat{SR} + \frac{\gamma_4 - 1}{4}\widehat{SR}^2}}\right)$$

Fat tails widen the denominator, so a crypto strategy needs a longer or cleaner record than a normal-returns calculation would suggest.

For the selection step I use the deflated Sharpe ratio (2014), which is the PSR with the threshold raised to the Sharpe you would expect from the luckiest of $N$ worthless strategies. With $\gamma$ the Euler-Mascheroni constant and $z(\cdot)$ the standard normal quantile,

$$SR^* = \sqrt{\operatorname{Var}[\widehat{SR}_{\text{trials}}]}\,\Big((1-\gamma)\,z\big(1 - \tfrac{1}{N}\big) + \gamma\, z\big(1 - \tfrac{1}{Ne}\big)\Big)$$

The more variants you try, and the more their Sharpes disagree, the higher the bar. The implementation is five lines.

def expected_max_sharpe(n_trials: int, var_sr: float) -> float:
    if n_trials < 2:
        return 0.0
    z1 = stats.norm.ppf(1 - 1 / n_trials)
    z2 = stats.norm.ppf(1 - 1 / (n_trials * math.e))
    return math.sqrt(var_sr) * ((1 - EULER_GAMMA) * z1 + EULER_GAMMA * z2)

A formula like this is easy to get subtly wrong and impossible to eyeball, so tests/test_metrics.py checks it against a Monte Carlo estimate of the mean maximum of 50 normals over 20,000 draws and requires agreement within 5%.

architecture

The pipeline is a straight line from a public archive to a report, with two scripts that carry the protocol.

Three design choices carry most of the weight.

everything is a rank

Features and the training label are converted to centered cross-sectional ranks in $[-0.5, 0.5]$ each day. Crypto daily returns have tails that would dominate any raw regression. The largest move inside the development universe was DOGE at +392% on 2021-01-28 (results/dev/data_report.json), and a squared-error fit on raw returns would spend most of its capacity on a handful of such days. The cost of ranking, which I did not appreciate until the holdout, is that the models optimize rank IC, and rank IC is not the same thing as PnL.

wide frames for features, a long frame for models

Every rolling feature is one vectorized call on a date by symbol frame with a complete daily index, so a row is a day and a missing bar is a NaN that propagates through min_periods. Only universe members are stacked into the long (date, symbol) frame the models see. In development the wide frames are 2,373 days by 522 listing episodes (results/benchmark.json).

the backtest is two lines of arithmetic

Weights formed at the close of day $t$ earn the return from close $t$ to close $t+1$. On a market that trades around the clock, the daily close at midnight UTC and the next open are the same instant, which removes a whole class of execution-timing questions that daily equity backtests have to answer.

W = weights.unstack("symbol").fillna(0.0).sort_index()
R = fwd_ret.unstack("symbol").reindex_like(W).fillna(0.0)
if lag:
    W = W.shift(lag).fillna(0.0)
gross = (W * R).sum(axis=1)
turnover = W.diff().abs().sum(axis=1)

Net returns at any cost are gross - turnover * bps / 1e4, so the whole cost grid comes from one backtest. A coin that has no bar the next day (a delisting) earns 0. The weight was chosen without knowing the bar would be missing, and the backtest must not know it either. Zero is still a choice, and an optimistic one for long positions, since a real delisting usually comes with a loss. It touches 58 of 79,190 development member rows, so the effect on the results is small, but it is not modeled.

implementation

The library is about 600 lines of numpy, pandas, scikit-learn and scipy, in eight modules under src/marketpred/, plus 25 pytest tests that run on synthetic panels with no network.

data

I list every USDT symbol in the archive (735) and download every monthly daily-kline zip from 2018-01 to 2026-08. That is 27,721 files, fetched with 32 threads in 400 seconds, parsed into 829,342 rows across 734 symbols and a 25 MB parquet (results/download_log.txt). Downloading every pair rather than a list of well-known coins is the main defense against survivorship bias. A hand-picked list of "top coins" encodes today's knowledge of which ones survived.

the universe

On each day a symbol is eligible only if, using data up to and including that day, it has at least 60 bars of history, its base asset is not a stablecoin, fiat currency, gold token, wrapped or staked duplicate, or leveraged token, its trailing 30 day median quote volume is at least 1,000,000 USDT, and that volume ranks in the top 50 of eligible names. Days with fewer than 20 names are skipped.

The rule sounds simple and is the easiest place in the whole pipeline to leak the future. "Top 50 by volume" computed over the full sample, or a coin's history counted from its first bar in a later file, silently uses information from after day $t$. eligibility() builds every quantity from cumulative or trailing operations on the wide frames, and a test perturbs every price from a date $T$ on and asserts that the universe and every feature up to and including $T-1$ are unchanged. That test had a hole in its first version, described in problems.

features

Nine features, computed at the close of day $t$ and ranked among that day's members.

A member with a missing value gets rank 0, the cross-sectional median, so a model never sees a NaN and never learns from missingness itself.

models

Four single factors with no fitting (rev_1d, rev_5d, mom_20d, mom_60d), ridge regression at alpha 1, 10 and 100, and scikit-learn's HistGradientBoostingRegressor at depth 3 and depth 6 (200 iterations, learning rate 0.05). The fitted models predict the cross-sectional rank of the next-day return from all nine ranked features.

walk-forward with an embargo

Development runs nine half-year test folds, 2020H1 through 2024H1, each with an expanding training window. The label on a training row dated $d$ is only known at the close of $d+1$, so the embargo has to be counted from the label, not the row.

@property
def train_end(self) -> pd.Timestamp:
    # A training row dated d has a label that is known at the close of d+1.
    # Requiring d + 1 <= test_start - EMBARGO_DAYS leaves a gap of
    # EMBARGO_DAYS between the last label and the first test day.
    return self.test_start - pd.Timedelta(days=EMBARGO_DAYS + 1)

Five days is more than a 1 day label strictly needs, and it costs almost nothing. A test fits a spy model that records the latest date it was shown and asserts it never passes train_end.

the holdout lock

The holdout script is short and its first few lines are the protocol.

if LOCK.exists():
    sys.exit(f"refusing: holdout already evaluated, see {LOCK}")
sel = json.loads((PROJECT_ROOT / "results" / "dev" / "selection.json").read_text())
prereg = (PROJECT_ROOT / "PREREGISTRATION.md").read_bytes()
lock = {"started_utc": datetime.now(timezone.utc).isoformat(),
        "preregistration_sha256": hashlib.sha256(prereg).hexdigest(),
        "selection": sel, "status": "started"}
LOCK.write_text(json.dumps(lock, indent=2))

The lock is written before any holdout number exists, so a crash halfway through still counts as the one touch. This is not tamper-proof against me, since I could delete the file, but deleting it would be a deliberate act rather than an accident, and the hash makes any later edit to the pre-registration detectable. The lock records a start and a completion four minutes apart on 2026-09-27, with verdicts "supported" and "not supported" (results/holdout/LOCK.json).

problems

None of these broke the build. Most of them would have produced plausible numbers.

the API was blocked, and the replacement was better

The Binance REST API returns HTTP 451 from the US. I switched to the static archive at data.binance.vision, which turned out to be the better research source anyway, because it keeps the monthly files of pairs that have since been delisted.

timestamps changed units mid-archive

The archive switched its kline timestamps from milliseconds to microseconds starting in 2025. Parsed naively as milliseconds, every bar from 2025 on lands tens of thousands of years in the future. I knew about the switch before writing the parser, so parse_kline_csv checks magnitude row by row.

t = pd.to_numeric(df["open_time"]).astype("int64").to_numpy()
unit_us = t > 10**14   # Binance spot archives use microseconds from 2025-01 onward
t_ms = np.where(unit_us, t // 1000, t)