# Farhan's Projects

Projects by Farhan Sadeek: storage engines, exchanges, trading research, compilers, kernels and ML systems, each built from scratch and written up with its measurements.

Author: Farhan Sadeek
Site: https://projects.farhansadeek.com/
Feed: https://projects.farhansadeek.com/atom.xml
Source: https://github.com/SadeekFarhan21/projects.farhansadeek.com
Projects: 17

## About

I'm [Farhan](https://farhansadeek.com), a computer science undergraduate at [Columbia](https://www.cs.columbia.edu). I spend most of my time tinkering with large language models, robots and computer chips, and this site is where I write up those projects: what I wanted to build, how it works, what broke, and what the measurements say.

The code for every project lives in one repository, [SadeekFarhan21/projects.farhansadeek.com](https://github.com/SadeekFarhan21/projects.farhansadeek.com). Essays and venture memos live on [writing.farhansadeek.com](https://writing.farhansadeek.com).

Reach me at [farhan@farhansadeek.com](mailto:farhan@farhansadeek.com), on [GitHub](https://github.com/SadeekFarhan21) or on [LinkedIn](https://www.linkedin.com/in/farhansadeek/).

## Index

1. GRPO Only Learns When the Rewards Disagree (2026-09-13)
   https://projects.farhansadeek.com/posts/rlvr-grpo-reward-variance/
   An RLVR pipeline on Qwen2.5-0.5B-Instruct, published before the training run. GRPO learns from the spread of rewards rather than their level, and four bugs never raised an exception.
2. Finding the Induction Heads in GPT-2 Small (2026-09-13)
   https://projects.farhansadeek.com/posts/reverse-engineering-gpt-2s-induction-circuit/
   All 144 attention heads of GPT-2 small scored on the induction diagonal in one cached forward pass on a laptop, recovering the five canonical heads. An induction score is a correlation, not a cause.
3. Every Line of a Small GPT, Written by Hand (2026-09-11)
   https://projects.farhansadeek.com/posts/building-a-gpt-from-scratch/
   A 5.26M-parameter GPT built from scratch in PyTorch and trained on TinyStories to a cross-entropy of 2.14 on a laptop, where the loss is a receipt that the pipeline works rather than a result.
4. Keeping My Own Copy of WakaTime (2026-06-29)
   https://projects.farhansadeek.com/posts/wakafree/
   WakaFree is a solo WakaTime-compatible coding tracker (a VS Code extension, a Next.js and Supabase server that archives one row per day, and a 15-tool MCP server) where the hard part was that a local day can last 23, 24, 25 or more hours.
5. Learning Chinese by Ear From Sun Tzu (2026-05-23)
   https://projects.farhansadeek.com/posts/listenting/
   A weekend hackathon PWA that plays Sun Tzu in Chinese with a synced transcript, 169 idiom tags across 290 sentences, and a spoken comprehension check-in that Claude writes and grades.
6. Keeping Columbus's Lead-Pipe Loan Fund From Running Dry (2026-03-28)
   https://projects.farhansadeek.com/posts/quantathon-leap-reserve/
   A two-person Quantathon 2026 entry sizes a $2.73M Columbus lead-pipe loan reserve (against $4.01M for grants) with a cohort Monte Carlo, and finds that one pilot-estimated uptake rate moves it by nearly $1M.
7. Erasing Artists From Stable Diffusion at Right Angles (2026-02-27)
   https://projects.farhansadeek.com/posts/cure-improve/
   An orthogonal subspace bank on top of training-free CURE erasure cuts collateral drift in Stable Diffusion v1.4 from 5 to 50 sequential erasures (LPIPS_u 0.7375 against 0.7735 at k=5), then reverses at 100.
8. Fraud Triage That Spares the LLM 97% of Transactions (2026-02-06)
   https://projects.farhansadeek.com/posts/aegis-fraud-triage/
   A 20-hour TartanHacks pipeline that scores every card transaction with an Isolation Forest and rules, then hands only 2.9 to 3.2 percent of synthetic traffic to a local LLM agent, with precision 1.000 in all 6 runs.
9. O'Hare or Atlanta Tops Every Centrality Ranking (2026-02-05)
   https://projects.farhansadeek.com/posts/airport-network-analysis/
   A Quarto notebook that turns 5,065 sampled flights into a 1,909-airport network and finds Chicago O'Hare or Atlanta at the top of all six centrality rankings.
10. Trial Emulation With a Critic That Guards Time Zero (2026-01-25)
   https://projects.farhansadeek.com/posts/clinova-trial-emulation/
   Clinova is a team-built agent pipeline that drafts a target trial emulation from a free-text clinical question, and on the Barrett 2006 benchmark its LLM validator rejected two designs over time-zero alignment before accepting the third, a loop that took 490 of the run's 837 seconds.
11. Graphing What Biomedical Papers Mention Together (2025-10-25)
   https://projects.farhansadeek.com/posts/empirica-biomedical-graphs/
   Empirica, a two-day hackathon app that turns biomedical PDFs into a NetworkX knowledge graph; on three open-access papers co-occurrence supplied 44 to 190 edges against 1 to 14 from verb patterns, and about 98 percent of the time was scispaCy NER.
12. The Encoder Sets the Scale of Commit Drift (2025-06-21)
   https://projects.farhansadeek.com/posts/diffsense/
   DiffSense scores how far each commit drifts semantically by embedding its diff and message. On its own 17 commits, CodeBERT gives a median drift of 0.0242 and MiniLM gives 0.679, and the two ranges never overlap.
13. A Pill Dispenser Built Around One Dose Record (2025-03-29)
   https://projects.farhansadeek.com/posts/pillifyai/
   A Make2025 hackathon build in about 19.5 hours: a prescribing app, a Node server and an ESP32 dispenser that polls every 30 seconds, all sharing one dose record.
14. Beating Buy-and-Hold by Holding Less Stock (2025-02-26)
   https://projects.farhansadeek.com/posts/quantifyai-market-regimes/
   A five-day Quantathon team pipeline that labels Bear, Bull and Static markets and moves money between stocks and bonds reached a Sharpe of 1.10 against 0.58 for buy-and-hold over 2019 to 2022, mostly by cutting volatility from 22.9% to 10.7% rather than by forecasting.
15. An ASL and Emotion Overlay for Any Video Call (2025-02-22)
   https://projects.farhansadeek.com/posts/signifyai/
   SignifyAI is a three-day team build that labels ASL letters, hand gestures and facial emotion over any window, including a video call, with three small CNNs that score 99.8% on gestures, 68.2% on FER2013 emotion and about 98% on ASL.
16. Casual Riders Keep Divvy Bikes Twice as Long (2025-01-18)
   https://projects.farhansadeek.com/posts/divvy-bike-analysis/
   A Quarto and R exploratory analysis of Chicago's 2024 Divvy bike-share trips finds casual riders average 25.2 minutes a ride against 12.8 for members, with a 17:00 commuter peak and a September high.
17. Timestamps and Hashes Did Most of My Photo Culling (2024-11-18)
   https://projects.farhansadeek.com/posts/photography-site/
   A curation pipeline for my photo portfolio scored 1,803 images and kept 217, and most of the cuts came from grouping near-duplicates by capture time and perceptual hash, not from the vision model's aesthetic score.


================================================================================

# GRPO Only Learns When the Rewards Disagree

Date: 2026-09-13
URL: https://projects.farhansadeek.com/posts/rlvr-grpo-reward-variance/
Tags: llm, reinforcement-learning, grpo
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/math-rlvr
Summary: An RLVR pipeline on Qwen2.5-0.5B-Instruct, published before the training run. GRPO learns from the spread of rewards rather than their level, and four bugs never raised an exception.

This RLVR pipeline produced a diagnosis and four post-mortems. The diagnosis is that the learning signal in GRPO[1] is the within-group *spread* of rewards rather than their level, which makes "my dataset is too easy" and "my dataset is too hard" indistinguishable from outside the reward function. The post-mortems cover four engineering failures (a token-id collision, non-terminating rollouts, generation running in train mode, and a silently frozen adapter), none of which raised an exception, and all of which produced a training run that looked healthy.

We also measured hardware instead of assuming it. For this workload a rented Modal L4 ran about 5% faster per step than an M4 Pro laptop, because per-step time is dominated by serial decode overhead rather than arithmetic. That is one paired observation, not a benchmark.

*Reading note.* The argument is the variance diagnostic and why the four bugs share a shape.

## Why It Matters

Reinforcement learning from verifiable rewards[2] is the cleanest setup in post-training. There is no reward model to train, no human preference data, and no judge model to be gamed. A math problem has a right answer, a program checks it, and the check *is* the reward. It is the recipe behind the reasoning-model results of the last two years[3], and at 500M it fits on one GPU.

That cleanliness is also what makes the failure modes legible. When the only moving parts are sample, score, and update, anything that goes wrong is in one of three places, which is why this project is worth writing up even without a training curve.

## Technical Details

The setup is GRPO via TRL 0.16[4] on **Qwen2.5-0.5B-Instruct**[5], LoRA[6] ($r = 32$, $\alpha = 64$, dropout 0.05, all projection modules) over a frozen base, `math_verify`[7] as the reward, 8 generations per prompt, learning rate $10^{-6}$, $\beta = 0.04$, temperature 0.9, completions capped at 640 tokens.

[Diagram] GRPO pipeline on one Modal L4 GPU with no vLLM: prompts from the data.py loaders and a frozen Qwen2.5-0.5B-Instruct with LoRA (r=32, alpha=64) feed a rollout of 8 completions per prompt; correctness_reward (last boxed answer checked by math_verify, which prints reward mean and std) and format_reward feed a group advantage that is zero when std is 0, then a GRPO update of the LoRA weights only (lr 1e-6, KL beta 0.04) loops back to the policy, save_model writes the adapter to the volume, a dashed arrow reloads it with is_trainable=True, and step4_eval.py compares base against adapted.

### GRPO, and Where the Signal Comes From

For each prompt, sample a group of $k$ completions and score them all. The advantage of completion $i$ is its reward minus the group mean, normalized by the group's spread.

$$
A_i = \frac{r_i - \mu}{\sigma}, \qquad
\mu = \frac{1}{k}\sum_{j=1}^{k} r_j, \qquad
\sigma = \sqrt{\frac{1}{k}\sum_{j=1}^{k}(r_j - \mu)^2}
$$

The policy is then pushed toward the above-average members of its own group. There is no value network and no critic to train. The group average *is* the baseline. That is the entire simplification, and at this scale it is the reason to use GRPO rather than PPO[8].

Read the numerator once more, because the dominant failure mode falls directly out of it. If every completion in a group receives the same reward, then $r_i = \mu$ for all $i$, every advantage is zero, and the gradient contribution of that prompt is exactly zero. You spend a GPU-minute generating 8 completions and learn nothing from them.

The condition for learning is therefore $\sigma > 0$ within the group, and that is a property of the *difficulty distribution of the prompts*, not of the model's competence in any absolute sense.

Zero within-group variance arrives from both directions, and the two are indistinguishable downstream.

| Training set | What happens | Reward std | Learning signal |
| --- | --- | --- | --- |
| GSM8K[9] | The Instruct model aces the problem 8/8 | $\approx 0$ | None |
| AIME | 90 olympiad problems, 0/8 on nearly all | $\approx 0$ | None |
| MATH[10] levels 3–5 | Sometimes right, sometimes wrong | $> 0$ | Yes |

"Too easy" and "too hard" produce the *identical* symptom. Flat reward, no movement, and a run that otherwise looks completely healthy. Wall-clock burns, the loss curve is plausible, and nothing improves. Any diagnosis that only asks whether reward is increasing cannot separate them, and the two have opposite fixes.

### The Verifier Is Part of the Objective

The other half of RLVR is deciding whether a completion's answer matches the gold answer. Completions are prompted to end in `\boxed{...}`, so extraction is a regex on the last match. Comparison is where the subtlety lives.

**String Equality vs. Symbolic Comparison**

| Model's boxed answer | Gold | String equality | `math_verify` |
| --- | --- | --- | --- |
| `72` | `72` | correct | correct |
| `0.5` | `1/2` | wrong | correct |
| `\frac{1}{2}` | `1/2` | wrong | correct |
| `40` | `72` | wrong | wrong |
| *(no `\boxed{}`)* | `72` | none | wrong |

Rows two and three are the reason this matters. A naive string check produces a **biased** reward. It systematically punishes correct answers for being written in a different form, and therefore trains the model toward the dataset's formatting conventions rather than toward being right. Parsing both sides symbolically and checking mathematical equality is the only version that rewards the intended thing.

Stated generally, the verifier *is* the objective. Every systematic error in it becomes a systematic pressure on the policy.

## Implementation

### The Diagnostic

So the reward function prints its own variance on every batch.

```python
n = len(rewards)
mean = sum(rewards) / n if n else 0.0
var = sum((r - mean) ** 2 for r in rewards) / n if n else 0.0
print(f"[reward] n={n} mean={mean:.3f} std={var ** 0.5:.3f} "
      f"(std~0 => no learning signal)")
```

Ten lines of arithmetic that convert a silent failure into a visible one. It is the single most useful thing in the pipeline. The general form is that **the learning signal in policy-gradient RL comes from the spread of outcomes, not their level.** That makes curriculum design a precondition here.

### Training Configuration

| Parameter | Value |
| --- | --- |
| Base model | Qwen2.5-0.5B-Instruct |
| Adapter | LoRA, $r = 32$, $\alpha = 64$, dropout 0.05, all projection modules |
| Generations per prompt ($k$) | 8 |
| Learning rate | $10^{-6}$ |
| KL coefficient ($\beta$) | 0.04 |
| Sampling temperature | 0.9 |
| Max completion length | 640 tokens |
| Correctness reward | 1.0 if `math_verify` accepts, else 0.0 |
| Format shaping | 0.2 for exactly one well-formed `\boxed{}` |
| Planned horizon | 300 steps (not launched) |

The format shaping term exists so that early in training there is a gradient toward parseable output before there is any gradient toward correctness. It is deliberately small relative to the correctness reward, so it cannot dominate once completions parse.

## Problems

None of these raised an exception. All four produced a training run that appeared to work. They share one shape. **Two components with defensible independent defaults must agree, and don't.** That is the most common category of bug we hit in this project.

[Diagram] Diagram of one GRPO training step from tokenizer through rollout, completion mask, reward and LoRA update, with four numbered red badges marking the seams where the eos/pad mismatch, missing stop criterion, no-KV-cache generate, and frozen adapter each failed silently.

### 1. The EOS/Pad Collision

Instruct models set `eos_token` to `<|im_end|>`. HuggingFace Transformers[11] pads stopped rollouts with `pad_token`. TRL masks each completion at the first `eos`. **If those two tokens disagree, the completion mask is wrong, and nothing tells you, because a wrong mask is still a valid mask.**

```python
tokenizer = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
tokenizer.eos_token = tokenizer.pad_token  # both <|endoftext|>
```

One line, several hours of diagnosis.

### 2. Rollouts That Never Terminate

We started on the Qwen2.5-Math-1.5B[12] *base* model. Its chat tokens are untrained, so it never emits `<|im_end|>` at all, so every rollout ran to the completion-length cap. At 8 generations per prompt and 640 tokens each, **nearly all of the compute was spent generating text after the answer had already been given.**

Two fixes followed: switching to the 500M Instruct model, which also roughly halved per-step time, and adding a stopping criterion that halts a rollout as soon as it contains a closed `\boxed{}`.

```python
class _StopAfterBoxed(StoppingCriteria):
    def __call__(self, input_ids, scores, **kwargs):
        texts = self.tokenizer.batch_decode(
            input_ids[:, self.prompt_len:], skip_special_tokens=True
        )
        return torch.tensor([BOXED_RE.search(t) is not None for t in texts],
                            device=input_ids.device)
```

The prompt itself mentions `\boxed{}`, so the criterion has to skip the prompt prefix, the kind of detail that turns a clean idea into a debugging session.

### 3. Generation Running in Train Mode

TRL 0.16 calls `generate()` without switching the model to eval. With gradient checkpointing enabled, that forces `use_cache=False`. **No KV cache means decoding is quadratic in sequence length rather than linear, on the single most expensive part of the step.**

```python
was_training = model.training
model.eval()
try:
    return orig_generate(input_ids, *args, **kwargs)
finally:
    if was_training:
        model.train()
```

A large speedup for a small patch, and again not something that surfaces as an error. It surfaces as "training is slower than I expected," which is indistinguishable from "training is this slow."

### 4. The Adapter That Loaded Frozen

If an adapter already exists at the output directory, the trainer loads it through PEFT[13] with `is_trainable=True` and continues, rather than starting a fresh LoRA.

```python
model = PeftModel.from_pretrained(base, args.output_dir, is_trainable=True)
```

**Omitting `is_trainable` loads the adapter frozen.** Training then updates nothing, reports no error, and produces a complete run with a plausible log.

A fifth issue is unresolved rather than fixed. TRL 0.16's vLLM[14] integration is server-based and wants a second GPU, and single-GPU colocation requires TRL 0.18 or later. The runs therefore use HuggingFace `generate` on one L4 and accept the throughput. That decision is what the next section measures.

## Experiments

### The Hardware Measurement

| Setup | Time per step |
| --- | --- |
| M4 Pro laptop (MPS) | ~77 s |
| Modal L4 (CUDA, bf16) | ~73 s |

The rented datacenter GPU was roughly 5% faster than the laptop. At 300 steps that is about 6 to 6.5 hours either way.

This is a single paired observation (one configuration, one run on each device, no repetitions), so it carries no interval and should not be read as a benchmark. What it does support is a directional claim with a mechanism behind it. **Per-step time is dominated by generation overhead in HuggingFace `generate`, not by matrix multiplication.** Sampling 8 completions of up to 640 tokens is a long serial sequence of small, latency-bound decode steps, and a wider GPU does not shorten a serial loop.

The consequences follow from the mechanism rather than from the 5%. Larger GPUs cost proportionally more per hour for roughly the same wall-clock, and a T4 is a false economy because it lacks bf16 and loses more time than it saves in price. The real fix is batched inference through vLLM, continuous batching[15] and paged attention[14], which is blocked on the TRL version above.

## Results

### What Exists

**We have no before/after RL numbers.** The pipeline runs end to end, the 300-step training run has not been launched, and we are publishing before it deliberately. Nothing below is contingent on what the final $\text{pass@}1$ turns out to be.

| Component | State |
| --- | --- |
| Data loaders (GSM8K, MATH, AIME → prompt/answer pairs) | Done |
| Verifier and reward-variance diagnostic | Done |
| GRPO training with LoRA, adapter-resume across runs | Done |
| $\text{pass@}1$ / $\text{pass@}k$ evaluation, base and adapted | Done |
| Modal deployment, persistent volume, wandb | Done |
| 300-step training run | Not launched |
| Before/after results | None |

### Conclusions

The stack is built but not yet tested against its own objective, and everything of value so far came out of it refusing to work.

- **Reward variance is the metric to watch, not reward mean.** Printing $\sigma$ every batch is ten lines and converts the most likely failure from invisible to obvious.
- **Too-easy and too-hard are the same observation from outside.** Any diagnosis that checks only whether reward is increasing cannot separate them, and they have opposite fixes.
- **The verifier is the objective.** A string-equality check is a biased reward function, not an approximate one.
- **Every failure here was silent.** EOS/pad mismatch, non-terminating rollouts, generation in train mode, a frozen adapter. Four bugs, zero exceptions.
- **Measure before renting.** For this workload the GPU bought about 5%, on one paired observation, because the bottleneck is serial decode rather than arithmetic.
- **The difficulty band is unmeasured.** Levels 3–5 produce nonzero variance; we have not swept difficulty against $\text{pass@}8$, so the curriculum is a working choice rather than a tuned one.

The open question is whether 300 steps on a 500M model moves $\text{pass@}1$ measurably at all. The frame we will judge it against is the $\text{pass@}k \gg \text{pass@}1$ gap. A model that solves a problem 1 time in 8 already contains the capability, and RL's job is to shift probability mass onto reasoning it can already occasionally produce rather than to teach it something new[16]. If the gap does not narrow, the report will say that it did not.

Find the code at [github.com/SadeekFarhan21/projects.farhansadeek.com/math-rlvr](https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/math-rlvr).

## What I Would Change

The claim above is that MATH levels 3–5 produce nonzero within-group variance and the alternatives did not. That is an observation from the diagnostic, not a characterization of the difficulty band.

We did not measure where the band actually sits for this model. The principled version sweeps difficulty against measured $\text{pass@}8$[17] and selects problems whose success probability is near $0.5$, which is where the expected within-group spread is largest. For a binary reward with success probability $p$, the group variance is maximized at $p = 0.5$, since

$$
\sigma^2 = p(1-p).
$$

That sweep has not been run, so "levels 3–5" is a working choice supported by a binary observation, not a tuned curriculum. It is the first thing we would run given more time, and it is cheap relative to training.

## References

1. Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv preprint, 2024. [arXiv:2402.03300](https://arxiv.org/abs/2402.03300)
2. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. *Tulu 3: Pushing Frontiers in Open Language Model Post-Training*. COLM, 2025. [arXiv:2411.15124](https://arxiv.org/abs/2411.15124)
3. DeepSeek-AI, Daya Guo, Dejian Yang, et al. *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. Nature, 2025. [arXiv:2501.12948](https://arxiv.org/abs/2501.12948)
4. Leandro von Werra, Younes Belkada, Lewis Tunstall, et al. *TRL: Transformers Reinforcement Learning*. Hugging Face, 2020. [link](https://github.com/huggingface/trl)
5. Qwen: An Yang, Baosong Yang, Beichen Zhang, et al. *Qwen2.5 Technical Report*. arXiv preprint, 2024. [arXiv:2412.15115](https://arxiv.org/abs/2412.15115)
6. Edward J. Hu, Yelong Shen, Phillip Wallis, et al. *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR, 2022. [arXiv:2106.09685](https://arxiv.org/abs/2106.09685)
7. Hynek Kydlíček. *Math-Verify: Math Verification Library*. Hugging Face, 2025. [link](https://github.com/huggingface/Math-Verify)
8. John Schulman, Filip Wolski, Prafulla Dhariwal, et al. *Proximal Policy Optimization Algorithms*. arXiv preprint, 2017. [arXiv:1707.06347](https://arxiv.org/abs/1707.06347)
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. *Training Verifiers to Solve Math Word Problems*. arXiv preprint, 2021. [arXiv:2110.14168](https://arxiv.org/abs/2110.14168)
10. Dan Hendrycks, Collin Burns, Saurav Kadavath, et al. *Measuring Mathematical Problem Solving With the MATH Dataset*. NeurIPS Datasets and Benchmarks, 2021. [arXiv:2103.03874](https://arxiv.org/abs/2103.03874)
11. Thomas Wolf, Lysandre Debut, Victor Sanh, et al. *Transformers: State-of-the-Art Natural Language Processing*. EMNLP System Demonstrations, 2020. [doi:10.18653/v1/2020.emnlp-demos.6](https://doi.org/10.18653/v1/2020.emnlp-demos.6)
12. An Yang, Beichen Zhang, Binyuan Hui, et al. *Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement*. arXiv preprint, 2024. [arXiv:2409.12122](https://arxiv.org/abs/2409.12122)
13. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, et al. *PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods*. Hugging Face, 2022. [link](https://github.com/huggingface/peft)
14. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. *Efficient Memory Management for Large Language Model Serving with PagedAttention*. SOSP, 2023. [arXiv:2309.06180](https://arxiv.org/abs/2309.06180)
15. Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, et al. *Orca: A Distributed Serving System for Transformer-Based Generative Models*. OSDI, 2022. [link](https://www.usenix.org/conference/osdi22/presentation/yu)
16. Yang Yue, Zhiqi Chen, Rui Lu, et al. *Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?* NeurIPS, 2025. [arXiv:2504.13837](https://arxiv.org/abs/2504.13837)
17. Mark Chen, Jerry Tworek, Heewoo Jun, et al. *Evaluating Large Language Models Trained on Code*. arXiv preprint, 2021. [arXiv:2107.03374](https://arxiv.org/abs/2107.03374)


================================================================================

# Finding the Induction Heads in GPT-2 Small

Date: 2026-09-13
URL: https://projects.farhansadeek.com/posts/reverse-engineering-gpt-2s-induction-circuit/
Tags: transformers, interpretability, research
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/tiny-circuits
Summary: All 144 attention heads of GPT-2 small scored on the induction diagonal in one cached forward pass on a laptop, recovering the five canonical heads. An induction score is a correlation, not a cause.

We score all 144 attention heads of GPT-2 small on the induction diagonal and recover the five heads the literature names[1]. They are **L5H5 (0.925), L6H9 (0.915), L7H10 (0.905), L5H1 (0.902), and L7H2 (0.820)**. The sixth-ranked head scores 0.529. That gap of 0.290 is what makes "the induction heads" a set rather than a cutoff someone chose.

The measurement is one cached forward pass over eight sequences of 101 tokens, and it runs in under a minute on an M4 Pro laptop. The model is GPT-2 small (124M)[2] via [TransformerLens](https://github.com/TransformerLensOrg/TransformerLens)[3].

*Reading note.* The argument is the gap and what it does not license. The configuration details, the TransformerLens processing notes, and the experiment plan are self-contained. Skip to [Conclusions](#conclusions) if you only want the claim and its limits.

## Why It Matters

Show a language model $\dots A\,B \dots A$ and it predicts $B$. It has not seen that pair in training; it learned the rule, which is to look back for where this token last occurred and emit whatever followed it. The standard argument is that this is the substrate of in-context learning generally[4], which is why the circuit that implements it is the usual first target.

This is experiment 01 of six, and it is the weakest kind of evidence in the plan it opens. An induction score is a correlation between a head's attention pattern and a behavior. It cannot separate a head that performs in-context copying from a head that attends to the right position for some unrelated reason while another component does the copying. We report these scores as a validated instrument reading, not as a circuit claim. The four experiments that would make it a circuit claim are specified below and have not been run.

## Technical Details

### The Behavior

The claimed implementation is two heads composed across layers[5].

1. A *previous-token head*, typically in layer 0, writes "the token before me was $A$" into each position's residual stream.
2. An *induction head*, in a middle layer, queries for that signature. It finds the position whose *previous* token matches the current token, and its OV circuit copies what that position holds into the output.

Two heads and one edge between them. Every part of that claim is separately testable, and this experiment tests the weakest part of it, whether heads exist whose attention goes where the story says it should.

[Diagram] Token strip '... A B ... A' where a previous-token head (typically layer 0) writes 'previous token = A' into the residual stream at B's position, and an induction head uses QK (W_Q W_K^T) to match the final A against that position and OV (W_O W_V) to copy B into the output, raising its logit; QK is bracketed as what the induction score measures, OV as not measured here (experiment 05), and the composition edge as assumed and tested by path patching in experiment 04.

### QK and OV, Where a Head Attends versus What It Moves

An attention head factors into two circuits that can be analyzed separately[5], and the separation is what makes "which head does what" a tractable question.

The *QK circuit* decides where to attend. It is the bilinear form $W_Q W_K^{\top}$ acting on pairs of residual-stream vectors. Given a query position and a key position, it produces the pre-softmax score[5][6]. Everything about attention placement is in this matrix.

The *OV circuit* decides what gets moved once a position is attended to. It is $W_O W_V$, and composed with the embedding and unembedding it becomes $W_U W_{OV} W_E$, a map from "the token at the attended position" to "the change in output logits." A head that copies is one whose OV circuit is approximately diagonal in token space. Attending to token $t$ raises the logit of $t$.

The split matters for this project because an induction score only measures the QK side. It says the head looks in the right place. It says nothing about whether the OV side moves anything useful, which is why experiment 05 examines $W_U W_{OV} W_E$ directly from the weights.

### Why a Repeated Random Sequence

Measuring induction on natural text conflates the thing we want with three things we don't. A head that attends "correctly" on English may be exploiting grammar, semantics, or a memorized n-gram rather than repeat structure, and the attention pattern looks the same in all four cases.

The input removes every alternative by construction. The test on repeated random tokens follows Olsson et al.[4].

```python
rand = torch.randint(0, vocab, (batch, seq_len), generator=g)
tokens = torch.cat([bos, rand, rand], dim=1)
```

`[BOS][50 random tokens][the same 50 tokens]`. There is no grammar, no semantics, and no n-gram statistics worth exploiting. The only structure available is that the second half repeats the first, so a head scoring highly here cannot be scoring highly for a reason we failed to think of.

This is the load-bearing design decision in the experiment. It buys a clean reading. The score means one thing. It costs external validity. Every conclusion below is about behavior on inputs GPT-2 never saw in training, and a head specialized for this diagonal on random tokens is not yet shown to do anything on English.

### The Induction Score

Under that construction, a head doing induction at position $i$ in the second copy must attend to position $i - 50 + 1$, the token that followed the previous occurrence of the current token. That is one fixed diagonal of the attention pattern, so the score is the mean weight along it.

For a head with attention weights $A^{(b)}_{q,k}$ on sequence $b$ of length $2N+1$, the induction score is the mean weight on the induction diagonal, taken over the batch $B$ and over the $N$ query positions in the second copy.

$$
\text{score} = \frac{1}{|B|\,N}
\sum_{b \in B} \sum_{j=0}^{N-1} A^{(b)}_{\,1+N+j,\; 2+j}
$$

Query $1+N+j$ sits at offset $j$ into the repeated half; it holds the same token as position $1+j$, so the key it must attend to is the token that *followed* that first occurrence, at position $2+j$. The difference between those indices is constant, which is why the whole measurement is one diagonal at offset $-(N-1)$.

[Diagram] Diagram of the induction-score setup: a [BOS][50 random][same 50] token strip above a 101x101 causal attention matrix with the single offset -49 diagonal (query 1+N+j to key 2+j) highlighted, the two first-half cells excluded, and the pipeline from one cached forward pass to 144 head scores.

## Implementation

The configuration is sequence length 50, batch 8, seed 1337, one cached forward pass, 144 numbers out.

```python
pattern = cache["pattern", layer]                     # [batch, head, query, key]
diag = pattern.diagonal(offset=-(seq_len - 1), dim1=-2, dim2=-1)
diag = diag[..., -seq_len:]                           # second-half queries only
scores[layer] = diag.mean(dim=(0, -1))                # mean over batch and diagonal
```

### What TransformerLens Changes About the Model Before You Measure It

TransformerLens loads GPT-2 with LayerNorm folding, centered writing weights, and centered unembedding on by default. These change the parameterization without changing the function the model computes, and they are what make residual-stream and logit-lens[11] analysis clean. A score computed on an unprocessed model is not being compared against the same object the published results describe.

This is why reproducing a known result was the point of experiment 01 rather than a formality. L5H5 and L6H9 are in the literature[1]; we did not find them, we recovered them, with our own metric implementation, our own input construction, and our own weight processing. A pipeline that recovers the canonical heads from scratch is a pipeline that can be pointed at questions with no answer key. That validation step is usually invisible in the write-up, and it is the step that decides whether any later number means anything.

## Results

| Head | Induction score |
| --- | --- |
| L5H5 | 0.925 |
| L6H9 | 0.915 |
| L7H10 | 0.905 |
| L5H1 | 0.902 |
| L7H2 | 0.820 |
| L10H1 | 0.529 |
| L9H6 | 0.509 |
| L9H9 | 0.500 |
| L10H7 | 0.459 |
| L5H0 | 0.436 |

We do not report an interval on these scores. With a single seed and a batch of 8, the run supports the ranking and the size of the rank-5 to rank-6 gap; it does not support the third decimal place. The per-head scores are stable across seeds in informal checks, but a seed sweep is a ten-line change that has not been run, so that stability is an impression rather than a measurement. Read the table as an ordering.

*Correction, September 26, 2026.* An earlier version averaged the diagonal over two extra first-half query positions, which have no earlier copy to look back to. That diluted every score by about 2% and put L5H1 above L7H10. The five-head set and the size of the gap did not change. The formula above also had an off-by-one in the key index, now fixed; the code was already correct on that point.

### What the Gap Means

Attention is a probability distribution over the entire 101-token context. A score of 0.925 means L5H5 places about 92% of its attention mass on a single position, and that position is determined purely by a structural rule in an input with no content to attend to. This is not a tendency toward induction. **It is a head that does one job.**

Five heads sit above 0.80 and the sixth sits at 0.53. **The 0.290 gap between rank 5 and rank 6 is the reason the set is well defined.** No threshold has to be chosen, because the data separates. Had the scores decayed smoothly from 0.9 to 0.4, "the induction heads" would name whatever cutoff we picked, and every downstream experiment would inherit that arbitrariness.

The second feature of the table is location. The five heads sit in layers 5 through 7 of a 12-layer model, downstream of the layer-0 previous-token heads the circuit needs as input, with enough depth remaining to influence the output. **The theory predicts where these heads should live, and that is where they are.** A prediction that could have failed and did not is worth more than the scores themselves, because the scores were always going to be high for *something*.

## What I Would Change

### What the Ranking Does Not Establish

Everything above is consistent with the five heads being passengers. A head can attend to exactly the right position for an unrelated reason while some other component performs the copy, and no attention-pattern measurement can tell the difference. Four objections remain open.

| Objection | Experiment | Method |
| --- | --- | --- |
| Correlation, not causation | 03 | Activation patching[7][8], clean vs. corrupted repeated sequences |
| Composition is assumed, not shown | 04 | Path patching[1][9] on the prev-token → induction edge |
| The behavior may be input-specific | 05 | Eigenvalues and diagonal dominance of $W_U W_{OV} W_E$ |
| Other heads may be redundant backups | 06 | Zero and mean ablation, head knockout sweep |

The distinction between 03 and 05 is the one that matters most. Activation patching is a statement about this model on these inputs; showing that the OV circuit is approximately a copy matrix is a statement about the *parameters*, independent of whatever inputs we happened to construct. Input-level evidence and weight-level evidence fail in different ways, and a circuit claim is only strong when both hold.

Experiment 06 is the one most likely to produce an awkward result. Ablation studies in this literature routinely find that knocking out a "necessary" component degrades behavior far less than the discovery evidence suggests, because other heads absorb the loss[1][10]. If that happens here it is a finding about the circuit rather than a defect in the measurement, but it would complicate the clean two-head story considerably. We would rather name that possibility now than after the fact.

### The Remaining Five Experiments

The plan, with the evidence each step is meant to produce.

| Experiment | Question | Evidence it produces |
| --- | --- | --- |
| 02 | Does the attention pattern look the way the score implies? | Per-head attention visualization on the repeated sequence |
| 03 | Do these heads cause the copying? | Recovery of logit difference under activation patching |
| 04 | Is the prev-token → induction edge real? | Path patching isolating that specific edge |
| 05 | Is the OV circuit a copy matrix? | Spectrum and diagonal dominance of $W_U W_{OV} W_E$ |
| 06 | Are the heads necessary? | Behavior collapse under zero and mean ablation |

Only 02 is a refinement of what is already measured. Experiments 03 through 06 are the ones that can overturn the reading above, and none of them has been run.

## Conclusions

Had we stopped at the score table, this post would have told a circuit-discovery story. Recovering L5H5 and L6H9 with an independent implementation is a validated instrument reading and nothing more. It establishes that the measurement is correct on a case with a published answer, which is the precondition for trusting it anywhere else.

- The five-head set is sharply defined. The 0.290 gap between rank 5 and rank 6 is in the data, not in a chosen threshold.
- The heads sit in layers 5 through 7, where the composition story says they must. That prediction could have failed.
- Nothing here separates "these heads participate in induction" from "these heads cause induction." Patching and ablation separate them, and they are not run.
- The reported scores support an ordering, not three decimal places. No seed sweep has been run.
- Laptop-scale interpretability on a real pretrained model is genuinely accessible. The whole experiment is one cached forward pass over eight sequences of 101 tokens.

Find the code at [github.com/SadeekFarhan21/projects.farhansadeek.com/tiny-circuits](https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/tiny-circuits).

## References

1. Kevin Wang, Alexandre Variengien, Arthur Conmy, et al. *Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small*. ICLR, 2023. [arXiv:2211.00593](https://arxiv.org/abs/2211.00593)
2. Alec Radford, Jeffrey Wu, Rewon Child, et al. *Language Models are Unsupervised Multitask Learners*. OpenAI, 2019. [link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
3. Neel Nanda, Joseph Bloom. *TransformerLens*. Software, 2022. [link](https://github.com/TransformerLensOrg/TransformerLens)
4. Catherine Olsson, Nelson Elhage, Neel Nanda, et al. *In-context Learning and Induction Heads*. Transformer Circuits Thread, 2022. [arXiv:2209.11895](https://arxiv.org/abs/2209.11895)
5. Nelson Elhage, Neel Nanda, Catherine Olsson, et al. *A Mathematical Framework for Transformer Circuits*. Transformer Circuits Thread, 2021. [link](https://transformer-circuits.pub/2021/framework/index.html)
6. Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. *Attention Is All You Need*. NeurIPS, 2017. [arXiv:1706.03762](https://arxiv.org/abs/1706.03762)
7. Kevin Meng, David Bau, Alex Andonian, et al. *Locating and Editing Factual Associations in GPT*. NeurIPS, 2022. [arXiv:2202.05262](https://arxiv.org/abs/2202.05262)
8. Stefan Heimersheim, Neel Nanda. *How to use and interpret activation patching*. arXiv preprint, 2024. [arXiv:2404.15255](https://arxiv.org/abs/2404.15255)
9. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, et al. *Localizing Model Behavior with Path Patching*. arXiv preprint, 2023. [arXiv:2304.05969](https://arxiv.org/abs/2304.05969)
10. Thomas McGrath, Matthew Rahtz, Janos Kramar, et al. *The Hydra Effect: Emergent Self-repair in Language Model Computations*. arXiv preprint, 2023. [arXiv:2307.15771](https://arxiv.org/abs/2307.15771)
11. nostalgebraist. *interpreting GPT: the logit lens*. LessWrong, 2020. [link](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)


================================================================================

# Every Line of a Small GPT, Written by Hand

Date: 2026-09-11
URL: https://projects.farhansadeek.com/posts/building-a-gpt-from-scratch/
Tags: transformers, pytorch
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/gpt-from-scratch
Summary: A 5.26M-parameter GPT built from scratch in PyTorch and trained on TinyStories to a cross-entropy of 2.14 on a laptop, where the loss is a receipt that the pipeline works rather than a result.

We trained a decoder-only GPT[1] written from scratch, with no `nn.Transformer`, no `F.scaled_dot_product_attention` and no HuggingFace model code, to a final cross-entropy of **2.14** on TinyStories[2], in 20,000 steps on a single Apple Silicon laptop. The shipped checkpoint is **5,263,848 parameters**, configured as 6 pre-LN blocks[3], 8 heads, a 256-dimensional residual stream, a 64-token context, and a 1,000-token byte-level BPE[4][1] vocabulary trained on the corpus itself.

The loss is not a result. It is a receipt, evidence that every stage of the pipeline, from tokenizer and cache through loader, model, loss and sampler, is wired correctly end to end, which was the entire goal. The run consumed $20{,}000 \times 8 \times 64 \approx 10.2\text{M}$ tokens against a cached corpus of roughly 632M tokens, so the model saw about 1.6% of the available data in a single pass. It is undertrained by construction, and no comparison to any published loss is meaningful. What the project did produce is a clear view of which lines are load-bearing. Two constants decide silently whether the model trains at all, and two bugs cost more time than the model code did.

*Reading note.* The argument is that the naive implementation makes the mechanism visible, and that the expensive failures were all silent.

## Why It Matters

Reading the transformer paper[5] and reading a reference implementation both leave the same gap. You can follow every line and still not know which lines are *load-bearing*. Typing it out closes that gap by force. Every constant omitted and every shape gotten wrong produces either a crash or, worse, a model that trains to nothing while looking fine.

## Technical Details

### Attention Is Three Linear Maps and a Mask

The single-head forward pass in full.

```python
keys, queries, values = self.key(x), self.query(x), self.value(x)
scores = queries @ keys.transpose(-2, -1)
scores = scores * (self.head_size ** -0.5)
scores = scores.masked_fill(self.mask[:T, :T] == 0, float("-inf"))
return F.softmax(scores, dim=-1) @ values
```

Four operations. A token's query is a question, every other token's key is an advertisement, the dot product scores the match, and the value is what actually gets moved. The causal mask is not a deep architectural property. It is a lower-triangular matrix of ones the size of the context, built with `torch.tril` and registered as a buffer, plus a `masked_fill`.

[Diagram] Inside one SelfAttention head, x passes through self.query, self.key and self.value; queries @ keys.transpose(-2, -1) is scaled by head_size ** -0.5, positions where the lower-triangular self.mask is 0 are set to -inf with masked_fill, F.softmax turns the scores into weights, and weights @ values gives the head's output; MultiHeadAttention runs x through heads[0] through heads[7] (the default 8 heads), joins the results with torch.cat(dim=-1), and passes them through self.proj.

### The Scale Factor Is Not a Detail

Multiplying the scores by one over the square root of the head size looks like a detail, and it decides whether the model trains.

Take a query $q$ and key $k$ in $\mathbb{R}^{d}$ with independent, zero-mean, unit-variance entries. Their dot product is a sum of $d$ independent terms, so

$$
\operatorname{Var}(q \cdot k) = \sum_{i=1}^{d} \operatorname{Var}(q_i k_i) = d,
$$

and the logits entering the softmax have standard deviation $\sqrt{d}$. Dividing by $\sqrt{d}$ restores unit variance and keeps the softmax in a regime where it is not saturated[5].

Without the scale, logits grow with head size, the softmax collapses toward one-hot, and the gradient through it vanishes, because $\partial \operatorname{softmax}$ is proportional to $p_i(\delta_{ij} - p_j)$, which goes to zero as any $p_i \to 1$. The model still trains, the loss barely moves, and nothing in the code looks wrong. That combination is what makes it dangerous. The failure has no error message and no obviously guilty line.

At $d = 32$ the factor is $1/\sqrt{32} \approx 0.177$, so the unscaled logits would be roughly $\sqrt{32} \approx 5.7$ times larger, enough to saturate and not enough to look absurd if printed.

### The Residual Stream Is the Actual Abstraction

```python
x = x + self.attention(self.layer_norm1(x))
x = x + self.feed_forward(self.layer_norm2(x))
```

Two `x + ...` lines, and they are the reason depth works at all. Each block *proposes an edit* to a running representation rather than replacing it. Normalization goes before the sublayer (pre-LN), so the residual path from input to output is unnormalized and gradients reach layer 0 intact[3].

Once the residual stream reads as a shared bus that every layer writes to and reads from[7], the later literature of logit lens[8], activation patching[9] and circuits[7] stops being exotic and starts being the obvious next question. That framing is what sent this project toward induction heads[10] next.

### Architecture

[Diagram] GPT forward pass as one residual stream: token ids through a 1,000 × 256 token embedding plus a learned 64 × 256 position embedding give x of shape (8, 64, 256), which runs down a spine through six pre-LN blocks, where LayerNorm then 8-head causal self-attention and LayerNorm then a 256 to 1,024 to 256 ReLU MLP each branch off and add back; the spine then goes through a final LayerNorm and a 256 to 1,000 LM head, and the (8, 64, 1,000) logits meet the shifted targets in cross-entropy to give a scalar loss.

**Full Architecture, as Configured in the Shipped Checkpoint**

| Component | Value |
| --- | --- |
| Blocks | 6 |
| Attention heads | 8 |
| Residual width | 256 |
| Head size | 32 (residual width / heads) |
| Context | 64 |
| Vocabulary | 1,000 (byte-level BPE) |
| MLP | $4\times$ expansion, ReLU |
| Position encoding | Learned embeddings |
| Normalization | Pre-LN, two `LayerNorm`s[6] per block |
| Parameters | 5,263,848 |

The parameter count breaks down as 272,384 in the embeddings (256,000 token + 16,384 positional), 4,733,952 across the six blocks, 512 in the final `LayerNorm`, and 257,000 in the unembedding. The causal masks are registered buffers and are not counted, which is why the usual count over the model's parameters excludes the 196,608 mask entries in the state dict.

[Diagram] The pipeline runs top to bottom: TinyStories text trains a 1,000-token byte-level BPE tokenizer.json, and tokenize_file() streams the text into a uint16 token cache that is rebuilt only when it is older than the text or the tokenizer. LanguageModelDataset memory-maps that cache and feeds (x, y) batches to the train.py loop (GPT forward, cross-entropy, backward, AdamW step), which writes gpt-tinystories.pt every 1,000 steps for generate.py and chat.py to load. Separately, launch_modal.py spawns modal_app.py, which runs the same train.py on an L4 GPU with a prebuilt token cache and saves its own checkpoint to the Modal volume every 500 steps.

## Implementation

The project covers the whole path: a tokenizer, a streaming data pipeline, the model, a training loop, a sampler, a chat REPL, and a harness for the GPU runs on Modal.

### Multi-Head, Implemented the Slow Way on Purpose

```python
self.heads = nn.ModuleList([SelfAttention(...) for _ in range(num_heads)])
return self.proj(torch.cat([head(x) for head in self.heads], dim=-1))
```

Production implementations fold all heads into one batched matmul. This one keeps them as independent modules and concatenates. That is measurably slower, and we kept it. Heads genuinely are independent subspaces, and writing them as separate objects makes that structural fact impossible to forget. Fusing is an optimization, not a concept, and the independence is exactly the property that later interpretability work depends on. The [induction-circuit project](https://projects.farhansadeek.com/posts/reverse-engineering-gpt-2s-induction-circuit/) scores heads individually, which only means something because they *are* individual.

At 3.8M–5.3M parameters on a laptop the trade is free. At any serious scale it is not, and the right move is to fuse and keep a slow reference implementation for testing against.

### The Tokenizer and the Data Pipeline

The least interesting part took the most iterations.

Byte-level BPE[4][1] with a ByteLevel pre-tokenizer and decoder means **no unknown tokens are possible**, since every byte sequence encodes. That sounds like a footnote and is what makes the model robust to whatever the corpus contains. The vocabulary is trained on the corpus itself rather than borrowed, which at 1,000 merges over children's stories gives a tokenizer specialized to exactly this distribution.

**Streaming Tokenization and Cache Invalidation**

The corpus is 1.9 GB of text and stopped fitting comfortably in memory. Tokenization became a streaming pass that writes a `uint16` token cache to disk, invalidated by comparing modification times against both the source text and the tokenizer.

```python
cache_is_stale = not token_cache.exists() or token_cache.stat().st_mtime_ns < max(
    args.data.stat().st_mtime_ns,
    tokenizer_path.stat().st_mtime_ns,
)
```

`uint16` because a 1,000-token vocabulary fits in 16 bits, which halves the cache versus `int32`. The resulting cache is 1.26 GB, or roughly 632M tokens.

The staleness check exists because the tokenizer was retrained once, and an hour of training then ran against tokens from the *previous* vocabulary, a corruption that produces no error, just a model learning a scrambled language.

### Training, and How Much Data the Run Actually Saw

AdamW[11] at its default weight decay of 0.01, learning rate $3 \times 10^{-4}$, batch size 8, context 64, seed 1337, 20,000 steps, checkpointing every 1,000. Checkpoints store the model state, optimizer state, the full config, the training arguments, and the final loss, so a resume restores the run rather than approximating it.

No learning-rate schedule, no warmup, no gradient clipping. At this scale none were necessary, and leaving them out kept the loop readable. At larger scale each is the next thing to add.

The number worth stating plainly is the token budget.

$$
20{,}000 \text{ steps} \times 8 \text{ sequences} \times 64 \text{ tokens} = 10.24\text{M tokens},
$$

against a cache of roughly 632M. The run therefore saw about **1.6%** of the corpus, once. The final loss of 2.14 is a number from a model that is data-starved by design, and the correct reading of it is "the pipeline learns" rather than "the model is good." Anyone comparing it to a published TinyStories loss is comparing against a different experiment.

### Sampling

Greedy decoding on a model this size produces loops almost immediately[12]. Temperature and top-$k$[13] were about thirty lines and changed the output more than any architectural change we made.

```python
logits = logits[:, -1, :] / temperature
if top_k is not None:
    cutoff = torch.topk(logits, min(top_k, logits.size(-1))).values[:, -1, None]
    logits = logits.masked_fill(logits < cutoff, float("-inf"))
next_token = torch.multinomial(F.softmax(logits, dim=-1), num_samples=1)
```

The thing worth internalizing is that the model is a distribution, not a speaker. Every "the model said X" is really "this decoding strategy, at this temperature, said X." Two of the qualitative judgments we made early about model quality turned out to be judgments about decoding.

### Scaling to a GPU, and What It Bought

Once the laptop run worked, training moved to a Modal L4 with a larger configuration of 8 layers, 320-dimensional residual, 128-token context, mixed precision, and an auto batch-size search that doubles the batch until CUDA runs out of memory.

The observation worth recording is how little of that work was model code. The same training script runs unchanged on CPU, Apple's MPS backend and CUDA, picking the device automatically. What the GPU run required was infrastructure.

**The Supervising Loop**

The GPU run needed three things: a persistent volume so a preempted container does not lose the checkpoint, a resume path that points at the same checkpoint file the run writes to, and a supervising loop that commits the volume every 60 seconds.

```python
while True:
    try:
        return_code = process.wait(timeout=60)
        ...
    except subprocess.TimeoutExpired:
        storage.commit()
```

Six hours of GPU time is worth nothing if the container dies at hour five with the checkpoint in a tmpfs.

The model itself is 182 lines. The tokenizer, data pipeline, training loop, sampler, chat REPL, and Modal harness together are roughly four times that, and that is where all of the bugs were. The transformer was the easy half.

## Problems

| Problem | Symptom | Cause |
| --- | --- | --- |
| Stale token cache | Model trains, output is gibberish, loss looks plausible | Tokenizer retrained without invalidating the cached token file |
| Dataset target alignment | Off-by-one, loss plateaus higher than expected | Dataset pre-shifts targets; the loss shifts them again |

**Both are silent. Neither raises.** Both are the same category, **a pipeline that is internally consistent and describing the wrong thing.** That category is, we now believe, the dominant failure mode in small-scale ML work, and it is why the receipt matters more than the number on it.

## Results

Had we reported only the final loss, this post would have described a trained model. **The accurate version is that it describes a *validated pipeline* with a model attached that has seen 1.6% of its corpus once.**

- **The naive version is the point.** Every optimization skipped is a concept left visible. Unfused multi-head attention costs throughput and buys understanding, and at this scale that trade is free.
- **The scale factor and the mask are not details.** $1/\sqrt{d}$ is the difference between a model that trains and one that silently does not, and the failure has no error message.
- **Most of the code is not the model.** 182 lines of model against roughly four times that in everything else, and the bugs were all in the everything else.
- **Silent correctness bugs dominate.** Nothing in this project crashed in an informative way. Both expensive bugs produced a running system quietly learning the wrong function.
- **The loss number is not comparable to anything.** One pass over 1.6% of the data is not a training run anyone should benchmark against.

## References

1. Alec Radford, Jeffrey Wu, Rewon Child, et al. *Language Models are Unsupervised Multitask Learners*. OpenAI technical report, 2019. [link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
2. Ronen Eldan, Yuanzhi Li. *TinyStories: How Small Can Language Models Be and Still Speak Coherent English?* arXiv preprint, 2023. [arXiv:2305.07759](https://arxiv.org/abs/2305.07759)
3. Ruibin Xiong, Yunchang Yang, Di He, et al. *On Layer Normalization in the Transformer Architecture*. ICML, 2020. [arXiv:2002.04745](https://arxiv.org/abs/2002.04745)
4. Rico Sennrich, Barry Haddow, Alexandra Birch. *Neural Machine Translation of Rare Words with Subword Units*. ACL, 2016. [arXiv:1508.07909](https://arxiv.org/abs/1508.07909)
5. Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. *Attention Is All You Need*. NeurIPS, 2017. [arXiv:1706.03762](https://arxiv.org/abs/1706.03762)
6. Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton. *Layer Normalization*. arXiv preprint, 2016. [arXiv:1607.06450](https://arxiv.org/abs/1607.06450)
7. Nelson Elhage, Neel Nanda, Catherine Olsson, et al. *A Mathematical Framework for Transformer Circuits*. Transformer Circuits Thread, 2021. [link](https://transformer-circuits.pub/2021/framework/index.html)
8. nostalgebraist. *interpreting GPT: the logit lens*. LessWrong, 2020. [link](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens)
9. Stefan Heimersheim, Neel Nanda. *How to use and interpret activation patching*. arXiv preprint, 2024. [arXiv:2404.15255](https://arxiv.org/abs/2404.15255)
10. Catherine Olsson, Nelson Elhage, Neel Nanda, et al. *In-context Learning and Induction Heads*. Transformer Circuits Thread, 2022. [arXiv:2209.11895](https://arxiv.org/abs/2209.11895)
11. Ilya Loshchilov, Frank Hutter. *Decoupled Weight Decay Regularization*. ICLR, 2019. [arXiv:1711.05101](https://arxiv.org/abs/1711.05101)
12. Ari Holtzman, Jan Buys, Li Du, et al. *The Curious Case of Neural Text Degeneration*. ICLR, 2020. [arXiv:1904.09751](https://arxiv.org/abs/1904.09751)
13. Angela Fan, Mike Lewis, Yann Dauphin. *Hierarchical Neural Story Generation*. ACL, 2018. [link](https://aclanthology.org/P18-1082/)


================================================================================

# Keeping My Own Copy of WakaTime

Date: 2026-06-29
URL: https://projects.farhansadeek.com/posts/wakafree/
Tags: wakatime, time-tracking, timezones, dashboards
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/wakafree
Summary: WakaFree is a solo WakaTime-compatible coding tracker (a VS Code extension, a Next.js and Supabase server that archives one row per day, and a 15-tool MCP server) where the hard part was that a local day can last 23, 24, 25 or more hours.

WakaFree keeps a permanent copy of my coding time. WakaTime already measures it, but its free plan only shows the last week, so I built a tracker that speaks WakaTime's API and saves WakaTime's answer for every day into my own database. Most of the work turned out to be about calendars. **Archiving coding time means agreeing with WakaTime on where each day starts and how long it lasts, and a local day can run 23, 24, 25 or more hours.**

WakaFree is a coding-time tracker I built on my own and run at [wakafree.farhansadeek.com](https://wakafree.farhansadeek.com). It has three parts. A VS Code extension sends WakaTime-style heartbeats, a Next.js 15 and Supabase server accepts them on WakaTime's `/api/v1` routes and copies WakaTime's daily summaries and timelines into Postgres (one row per day, refreshed every 20 minutes and backfilled over a full year every night), and a FastMCP server exposes the WakaTime API to an LLM client as 15 tools. It's a solo summer project from 2026, and it has one user, me.

## Why It Matters

WakaTime is a coding-time tracker. You install a plugin in your editor, and the plugin reports small events called heartbeats while you work. WakaTime's servers turn those heartbeats into time per project, language, editor and machine, and show it on a dashboard[1].

What I wanted was my own copy. The free plan shows one week of dashboard history[2], and although WakaTime keeps everything and opens up the older history if you upgrade[3], I'd still be looking at my own work through someone else's window. I wanted every day in a database I control, charts I choose (timelines segmented six ways, AI coding metrics, per-project detail), and the same data available to an LLM.

So WakaFree does two jobs. It speaks WakaTime's API, so an editor plugin can send heartbeats to it the same way it would to WakaTime. And it archives WakaTime's computed answers every day, so the history piles up in my Postgres instead of scrolling out of a dashboard view.

## Technical Details

### Heartbeats Are Cheap Facts

A heartbeat is one small record saying "this file was active at this moment". WakaTime's plugin guide says a plugin should send one when the focused file changes, when more than 2 minutes have passed on the same file, and on every save[4], and my extension follows those rules. This is the payload it sends, from the source with comments added.

```ts
export interface HeartbeatPayload {
  entity: string        // the file path
  type: 'file'
  time: number          // epoch seconds
  project?: string
  language?: string
  branch?: string
  is_write: boolean     // true on save
  editor: string
  operating_system: string
  machine: string
  lines?: number
  lineno?: number
  cursorpos?: number
}
```

Heartbeats go out in batches to `POST /api/v1/users/current/heartbeats.bulk`, with the API key base64-encoded in a Basic `Authorization` header. That's the request shape WakaTime documents[1], which is why a server that accepts it can take heartbeats from a WakaTime plugin.

### Time Comes From the Gaps Between Heartbeats

Since a heartbeat is only a timestamp, time has to come from the spacing between them. Sort the heartbeats, and if two neighbors are closer than a timeout, count the gap as coding time; if they're further apart, you walked away and the gap counts for nothing. WakaTime calls this the keystroke timeout, and its default is 15 minutes[3]. WakaFree's own computation uses the same 15 minutes.

```ts
const TIMEOUT_SECONDS = 15 * 60

for (let i = 1; i < sorted.length; i++) {
  const diff = sorted[i].time - sorted[i - 1].time
  if (diff < TIMEOUT_SECONDS) total += diff
}
```

On synthetic data the edges behave as you'd hope. Heartbeats every 2 minutes for an hour, then a 20-minute break, then another hour, make 62 heartbeats and **exactly 7,200 seconds, with the break dropped entirely**. A gap of 899 seconds counts in full, a gap of exactly 900 counts zero, and a single heartbeat is worth nothing. Because the rule only ever looks at neighbors, a day's total depends entirely on which heartbeats fall inside that day, so the day's boundaries matter as much as the rule.

WakaTime also joins heartbeats into sessions it calls durations, and it can split a day's durations by a field with `slice_by` (project, language, editor, operating system, machine, category and a few more)[1]. The timelines in WakaFree are built from those.

### A Day Is Local, and Its Length Varies

WakaTime's API returns a day's durations and heartbeats from midnight to 11:59 PM in the user's timezone[1]. That sounds like a detail until you write code that asks for "today". Anywhere west of UTC, the UTC date rolls over before local midnight. In a zone five hours behind, 19:00 on January 15 is already January 16 in UTC, so every evening has a five-hour window where UTC is a day ahead.

Days also come in different lengths. Measured from one local midnight to the next with WakaFree's own date helper, an ordinary New York day is 24 hours, **the US spring-forward day, 2026-03-08, is 23 hours, and the fall-back day, 2026-11-01, is 25**. A day that starts at midnight in one zone and ends at midnight in a zone two hours west is 26 hours, and one that ends two hours east is 22. The start of each day has to be computed on its own, because adding 86,400 seconds to the start of March 8 lands an hour after the real start of March 9.

[Diagram] Two panels built from the repo's date helpers: the first lines up local time five hours behind UTC against UTC for January 15 to 16 and shows that at 19:00 local UTC is already January 16, which is why offsetDate takes today in the local zone anchored at 12:00 UTC; the second draws bars measured from local midnight to the next local midnight, 24 hours on an ordinary day, 23 on the US spring-forward day 2026-03-08, 25 on the fall-back day 2026-11-01 and 26 on a day that ends in a zone 2 hours west.

### Architecture

Data comes in two ways. My extension posts heartbeats to WakaFree's own API, which stores them raw, and a sync job reads WakaTime's computed summaries and timelines and upserts them into the same database, one row per date. The dashboard reads the archive, and the MCP server talks to WakaTime's API directly.

[Diagram] WakaFree architecture with two ways in: the VS Code extension (heartbeat on file switch, on edit once per file per 2 minutes, and on every save, queued and flushed every 30 s) posts to heartbeats.bulk on the Next.js 15 server, while syncRange pulls /summaries and /durations from the WakaTime API (every 20 minutes for 2 days, nightly for 365 days, or from the Refresh button) and upserts one waka_daily row per date into Supabase Postgres beside the heartbeats and waka_meta tables; the Recharts dashboard reads 365 slim rows plus 3 rows with full timelines, and a 15-tool FastMCP server calls the WakaTime API for an LLM client.

## Implementation

### The Extension

The extension is 318 lines of TypeScript across five files. It hooks three VS Code events[5] (active editor changed, document changed, document saved) and sends one heartbeat for the file that's open when it starts. Each heartbeat gets the workspace folder as its project, the git branch from `git branch --show-current` with a one-second timeout, and a language name mapped from VS Code's language IDs to WakaTime's names across 30 entries.

Nothing is sent right away. Heartbeats go into a queue that flushes every 30 seconds, and a failed flush puts the batch back at the front.

```ts
function needsHeartbeat(filePath: string, isWrite: boolean): boolean {
  if (isWrite) return true
  const last = lastSentAt.get(filePath) ?? 0
  return Date.now() - last >= DEBOUNCE_MS   // 2 minutes
}

async function flush(): Promise<void> {
  if (queue.length === 0) return
  const batch = queue.splice(0, queue.length)
  try {
    await sendHeartbeats(batch)
  } catch (err) {
    queue.unshift(...batch)   // retried on the next flush
  }
}
```

Driving the tracker with synthetic edits under a fake clock shows how much the debounce saves.

| Scenario | Heartbeats sent |
|---|---|
| One edit per second on one file for 100 s | 1 |
| One edit per second on one file for 600 s | 5 |
| Five saves in 5 s | 5 |
| Three files, switching every 10 s for 600 s | 15 |

**Six hundred keystrokes turn into five heartbeats**, while saves always go through because they mark actual writes. That keeps the traffic tiny, since a one-heartbeat batch is 261 bytes, and the retry path holds up too: when a stub server answered with a 500, the same two-heartbeat batch went out again on the next flush. The status bar shows today's total, refreshed every 5 minutes from the server's `/summaries`, and clicking it opens the dashboard.

### Speaking WakaTime's API

The server implements the routes an editor plugin and the status bar need, which are `heartbeats`, `heartbeats.bulk`, `summaries`, `projects`, `stats/[range]` and the current user. The bulk route accepts an array or a single object, takes the machine name from the `X-Machine-Name` header, defaults the category to `coding`, and inserts the rows into a `heartbeats` table indexed on `(user_id, time)`. It then replies with a `responses` list that pairs each stored heartbeat with a 201 status.

```ts
const responses = (data ?? []).map((hb) => [
  { data: { id: hb.id, entity: hb.entity, time: hb.time } },
  201,
])
return NextResponse.json({ responses }, { status: 201 })
```

Auth accepts either a Basic header with the base64 key, which is what WakaTime plugins send, or a Bearer token.

### Archiving One Row per Day

The archive is the center of the project. `waka_daily` has one row per date, and the whole upstream summary for that day goes in a JSONB column[6] next to every timeline. The schema comment says it plainly, "Nothing is discarded", so projects, languages, editors, dependencies, AI agent costs and the daily average all come along without me picking fields in advance.

A sync is one batched `/summaries` call for the whole range, then six `/durations` calls per day, fired in parallel. One is the unsliced timeline, and five are sliced by category, language, editor, OS and machine.

```ts
const TIMELINE_SLICES = ['category', 'language', 'editor', 'os', 'machine'] as const

days.flatMap((day) => [
  fetchWakaFull(`/durations?date=${day.range.date}`),
  ...TIMELINE_SLICES.map((slice) =>
    fetchWakaFull(`/durations?date=${day.range.date}&slice_by=${slice}`)),
])
```

Every row is upserted on its date[7], so syncing the same day twice just replaces it, and that's what lets the schedules overlap without coordination. A GitHub Actions workflow[8] runs every 20 minutes and refreshes the last 2 days, another runs nightly and refreshes the last 365, and a Vercel cron runs once a day as well. A Refresh button on the dashboard calls a Next.js server action[9] that syncs the last 2 days on the server, so the sync's shared secret never reaches the browser. A sync can also snapshot 9 account-level endpoints (profile, all-time total, goals, projects, machines, user agents and stats for 7 days, 30 days and all time) into a `waka_meta` table.

With a counting `fetch` and a stubbed database, `syncRange` shows what each schedule costs upstream.

| Sync | Requests to WakaTime |
|---|---|
| 2 days, no snapshots (every 20 minutes) | 13 |
| 2 days with snapshots (Refresh button) | 22 |
| 7 days, no snapshots | 43 |
| 365 days with snapshots (nightly) | 2,200 |

**The timelines are almost all of it.** The count grows by 6 for every day in the range, so the nightly year is 2,190 durations requests, 1 summaries request and 9 snapshots.

### Timelines Segmented Six Ways

Because every slice is already in the row, the "Segment By" picker on the timeline is a client-side switch with no extra fetch. The only wrinkle is that each slice names its label field differently (`category`, `language`, `editor`, `machine`, `os` or `project`), so one function reads whichever is present and turns every slice into the same `{time, duration, label}` block. Older rows that only stored the category slice in its own column still map through the same path.

The dashboard around it has an overview (range total, current day, daily average, most active day), a range picker for 7, 14, 30, 90 and 365 days, a weekday chart, stacked daily charts by project and category, breakdown pies, goals, all-time stats, per-project detail and an AI coding section. That section adds up WakaTime's per-day AI fields, such as AI and human line additions and deletions, input and output tokens, prompts, sessions and per-agent cost. It's drawn with Recharts in a muted Nord-style palette.

### Drawing Each Day in Its Own Zone

The timeline's hour axis has to start at the midnight I actually lived through. `timezone.ts` keeps an ordered list of `{from, tz}` segments with IANA zone names[10], and `timezoneForDate` picks the last segment that has started. The IANA rules handle daylight saving, so only travel or a timezone change needs a new line. Local midnight comes from measuring the zone's offset with `Intl.DateTimeFormat`[11].

```ts
export function localMidnightEpochSeconds(dateStr: string, tz: string): number {
  const utcMidnight = new Date(`${dateStr}T00:00:00Z`)
  const offsetMin = tzOffsetMinutes(utcMidnight, tz)
  return Math.round((utcMidnight.getTime() - offsetMin * 60000) / 1000)
}
```

For New York, local midnight on January 15 comes out 5 hours after UTC midnight and on July 15 only 4 hours after, so summer time shows up without any special case.

### The MCP Server

The third part is a 282-line Python server built on FastMCP[12], which speaks the Model Context Protocol[13]. It wraps WakaTime's API as 15 tools, covering the current user, all-time stats, today's status bar, stats by range, summaries, durations with `slice_by`, heartbeats for a date, projects, commits per project, goals, insights, leaderboards, machine names, editors and languages. That puts the same data in front of an LLM client, which can look up a day's work through a tool like `get_durations`.

## Problems

### 1. Asking WakaTime for the Right Day

The first version of the sync computed "today" and "yesterday" as UTC dates. Every evening, once UTC had rolled over, the frequent sync asked WakaTime for a date that hadn't started locally, got an empty day back and stored it, and the dashboard showed that empty day as today. **Every date the sync asks for has to be a local date**, because that's the only kind of day WakaTime answers for.

The fix was to do date math in the local zone. `offsetDate` takes today's date in the zone I'm in, then anchors it at 12:00 UTC before subtracting whole days.

```ts
const [y, m, d] = currentLocalDate().split('-').map(Number)
const dt = new Date(Date.UTC(y, m - 1, d, 12))
dt.setUTCDate(dt.getUTCDate() - daysAgo)
```

Noon keeps the timestamp far from either midnight, so stepping back whole days always lands on the intended calendar date. The dashboard now reads the row for today's local date instead of the newest row, ignores any row dated after today, and picks Most Active as the day with the largest tracked total. The once-a-day Vercel cron was also rescheduled to run after the local day has rolled over.

### 2. Drawing a Day Across a Timezone Change

A single fixed zone draws every day in the wrong place once you travel. **Each day needs the zone you were in that day**, and its axis has to start at that zone's midnight. That's why the schedule is per date and why each day's start is computed on its own. Adding 86,400 seconds to the previous start would be off by an hour after each DST change, and by the full offset difference after moving between zones.

### 3. A Year of Timelines on Every Page Load

The dashboard first loaded all 365 rows with every column, including the heavy JSONB timelines, even though **only the latest days need a timeline when the page opens**. Now the page runs three queries in parallel, for the slim `date, data` columns of up to 365 rows, the account snapshots, and the full timeline columns for just the last 3 rows. Moving the timeline to an older date fetches that one day from `/api/timeline?date=`.

## What I Would Change

### More Editors

WakaTime's heartbeat rules fit in a paragraph, and WakaFree already accepts the bulk format a WakaTime plugin sends. A JetBrains or Neovim plugin would be a small project, and it would make WakaFree's own heartbeat data cover all of my editing instead of only VS Code.

## References

1. WakaTime. *API Docs*. WakaTime developer documentation. [link](https://wakatime.com/developers)
2. WakaTime. *Pricing*. [link](https://wakatime.com/pricing)
3. WakaTime. *FAQ*. [link](https://wakatime.com/faq)
4. WakaTime. *Creating a Plugin*. WakaTime help. [link](https://wakatime.com/help/creating-plugin)
5. Microsoft. *VS Code API*. Visual Studio Code extension documentation. [link](https://code.visualstudio.com/api/references/vscode-api)
6. PostgreSQL Global Development Group. *JSON Types*. PostgreSQL documentation. [link](https://www.postgresql.org/docs/current/datatype-json.html)
7. Supabase. *Upsert data*. Supabase JavaScript reference. [link](https://supabase.com/docs/reference/javascript/upsert)
8. GitHub. *Events that trigger workflows*. GitHub Actions documentation. [link](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows)
9. Vercel. *Updating Data*. Next.js documentation. [link](https://nextjs.org/docs/app/getting-started/mutating-data)
10. IANA. *Time Zone Database*. [link](https://www.iana.org/time-zones)
11. MDN. *Intl.DateTimeFormat.prototype.formatToParts()*. MDN Web Docs. [link](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/formatToParts)
12. Model Context Protocol. *MCP Python SDK*. GitHub repository. [link](https://github.com/modelcontextprotocol/python-sdk)
13. Model Context Protocol. *Introduction*. [link](https://modelcontextprotocol.io/)


================================================================================

# Learning Chinese by Ear From Sun Tzu

Date: 2026-05-23
URL: https://projects.farhansadeek.com/posts/listenting/
Tags: llm, nextjs, audio
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/listenting
Summary: A weekend hackathon PWA that plays Sun Tzu in Chinese with a synced transcript, 169 idiom tags across 290 sentences, and a spoken comprehension check-in that Claude writes and grades.

Heritage speakers understand spoken Chinese but can't read it well, and most learning apps start from characters. ListenTing starts from listening. It plays Sun Tzu's *Art of War*[1] in Chinese with a transcript that highlights the current sentence, flags idioms in the text, and every three minutes by default pauses to ask a spoken comprehension question that Claude writes and grades. Three of us built it over one weekend, 2026-05-23 to 05-25, for the Anthropic Cultural Exchange Hackathon.

The design idea is to do the expensive work once, offline, so it isn't paid for by every listener. A pipeline segments the source into **13 chapters, 290 sentences and 169 idiom tags**, translates it with Claude Haiku 4.5, and narrates it with ElevenLabs one sentence per call, so each sentence's place in the audio comes from its own clip. The app is a Next.js PWA that loads those files, plays the chapter, and uses Haiku to write each check-in question and Sonnet to grade the answer.

## Why It Matters

A heritage speaker can follow the sound of Chinese and stalls at the page. I wanted an app where the audio is the main thing and the text is scaffolding you can lean on.

Two ideas sit behind that. Heritage language learners are a distinct group whose listening and speaking outrun their literacy[2]. And the comprehensible-input hypothesis argues that acquisition is driven by understanding messages slightly above your current level[3]. Put together, they point to a product where you hear real Chinese at natural pace and the transcript is a fallback.

We picked Sun Tzu because it is short, ancient, and full of four-character idioms that people still use. A phrase like 知己知彼, "know yourself and know your opponent", is both a line in the book and something a heritage speaker may have heard at a dinner table. Tapping it to see the literal reading and the modern meaning connects two things they half know.

The weekend scope was a player with a highlighted transcript, idiom cards, a vault for saved words, a scheduled spoken check-in, an installable PWA, and an offline pipeline, because translating at runtime would cost money for every listener.

## Technical Details

### A Synced Transcript Is a Timeline

A synced transcript has one core data structure: a list of sentences, each with a start and end time in the audio. The player asks which sentence contains the current audio time and highlights it. If the times are wrong, the highlight is wrong, however good everything else is.

The cleanest way to get the times is to measure them from the audio, one sentence at a time. That is what the narration script does. It asks ElevenLabs for one clip per sentence, measures each clip, and writes the start and end of every sentence from those measured lengths.

### Measuring MP3 Duration Without ffmpeg

The narration script measures each generated clip by walking the MP3 frame headers. For MPEG-1 Layer III every frame holds 1,152 samples, and its length in bytes is

$$
\text{frame length} = \left\lfloor \frac{1152}{8} \cdot \frac{\text{bitrate}}{\text{sample rate}} \right\rfloor + \text{padding}
$$

so summing frames and dividing the total samples by the sample rate gives the duration. That is exact for constant-bitrate audio, and the script requests `mp3_44100_128` from ElevenLabs.[6] That citation covers the output format; the frame formula itself is standard MP3 framing.

### Architecture

The system has an offline half and an online half, joined by static files.

[Diagram] ListenTing architecture in two halves: an offline pipeline where qinkan.net text goes through prepare-chapters.ts into public/chapters/1..13.json (13 chapters, 290 sentences, 169 idiom tags), which feeds enrich-translations.ts (Haiku 4.5 fills en) and generate-audio.ts (ElevenLabs per sentence, rewrites timestamps from real durations), plus ch1-4.mp3 ElevenLabs narration uploaded by Mulong, feeding a browser PWA (Player, CheckinModal, IdiomCard with 28 curated entries, VaultSheet on device) that calls the Next.js API routes /api/question, /api/grade, /api/speak, /api/transcribe and /api/translate.

In the offline half, `prepare-chapters.ts` reads the source text and writes one JSON per chapter into `public/chapters`. `enrich-translations.ts` fills an English field on each sentence with Claude Haiku 4.5, and `generate-audio.ts` narrates each sentence with ElevenLabs and writes the timestamps from the real durations.

The online half is a Next.js 15 and React 19 app with Tailwind, packaged as a PWA with a small service worker. The player loads a chapter's JSON as a static file, asks the server whether `/audio/chN.mp3` exists, and plays it if it does. If not, it falls back to the browser's speech synthesis and walks the sentences one by one. The API routes hold the keys: Haiku 4.5 writes the check-in question, Sonnet 4.5 grades the answer, ElevenLabs reads the question aloud, and `/api/translate` translates on demand.

The check-in has its own small state machine, and it is where the model calls live.

[Diagram] The spoken check-in as a vertical chain of phases in CheckinModal.tsx: the player pauses after 3 minutes of audio, loading calls /api/question (Haiku 4.5, last 12 sentences), asking calls /api/speak with a speechSynthesis fallback, listening uses browser SpeechRecognition or typed input, grading calls /api/grade (Sonnet 4.5) and the result phase shows a verdict of great, good, partial or off with feedback under 30 words.

## Implementation

I wrote the pipeline scripts and the Next.js app. Alex converted the source text to UTF-8, and Mulong supplied the source files and the four chapter recordings. The scripts, API routes, lib and components come to 2,841 lines.

### Segmenting the Text

The source is a single third-party file with the classical text and a modern-Chinese translation for each of the 13 chapters, plus editorial annotations and a promotional footer. `prepare-chapters.ts` finds each chapter by its header, separates the classical block from the translation block, strips annotations, and splits paragraphs into sentences after 。, ！ and ？.

```ts
function splitSentences(paragraph: string): string[] {
  const cleaned = cleanLine(paragraph);
  if (!cleaned) return [];
  // Keep punctuation attached, split after 。！？
  const parts = cleaned
    .split(/(?<=[。！？])/)
    .map((p) => p.trim())
    .filter(Boolean);
  return parts.length ? parts : [cleaned];
}
```

Each sentence keeps its classical text in `zh` and the matching modern-Chinese translation from the source in `gloss`.

### Tagging Idioms

Idioms are a hand-written list of 28 entries in `idioms.ts`. Each has the term, pinyin, a literal character-by-character reading, a modern meaning, an optional origin, and a kind: `chengyu`, `reference` or `concept`. Coverage is selective on purpose. Tagging is a substring match of each term against the sentence, so one sentence can carry several tags.

```ts
{
  term: "知己知彼",
  pinyin: "zhī jǐ zhī bǐ",
  literal: "know-self know-other",
  kind: "chengyu",
  // meaning and origin omitted here
},
```

### Choosing Audio or Speech Synthesis

`usePlayer` decides at load time whether real audio exists by asking for the file.

```ts
const head = await fetch(chapter.audioSrc, { method: "HEAD" });
if (head.ok && head.headers.get("content-type")?.includes("audio")) {
  const a = new Audio(chapter.audioSrc);
  // ...
}
```

Dropping an MP3 into `public/audio` is all it takes to switch a chapter from speech synthesis to real narration.

### The Enrichment Script

`enrich-translations.ts` sends one batch per chapter to `claude-haiku-4-5` with a system prompt asking for a JSON array of modern English translations, one per input sentence. It strips a code fence if the model added one, and it checks that the array length equals the sentence count, throwing if not. It skips chapters where every sentence already has English, so it is safe to re-run.

### The Narration Script

`generate-audio.ts` makes one ElevenLabs call per sentence, so it knows each sentence's real duration, then concatenates the MP3 chunks by raw bytes. The defaults are the `eleven_multilingual_v2` model, `mp3_44100_128` output, stability 0.45 and similarity 0.7. It writes each sentence's `start` and `end` from the measured durations, using the frame walk above.

```ts
chunks.push(result.audio);
s.start = +cursor.toFixed(2);
s.end = +(cursor + result.durationSec).toFixed(2);
cursor += result.durationSec + SENTENCE_GAP_MS / 1000;
```

It avoids ffmpeg entirely, and per-sentence calls make the alignment exact by construction.

### The Check-in Loop

The player pauses after `checkinIntervalMin` minutes of playback, three by default, and opens `CheckinModal`. The modal asks `/api/question` for a question, its rubric and a one-sentence recap, using the last 12 sentences as context. It reads the question aloud and takes the answer from the browser's speech recognition[4], or as typed text when recognition isn't available. `/api/grade` then sends the last 10 sentences, the rubric and the answer to Sonnet 4.5 and gets back one of four verdicts, `great`, `good`, `partial` or `off`, with feedback under 30 words and a one-sentence model answer.

### The Vault and the PWA

Saved words are stored client-side with `idb-keyval` under a `vault:` prefix, so the vault works offline and never leaves the device. The service worker caches a small shell and serves stale-while-revalidate for same-origin GETs, skipping `/api/`.

## Problems

### 1. Pairing Classical Sentences With Their Translation

Each classical paragraph comes with a modern-Chinese translation paragraph, and the two don't always split into the same number of sentences. The script pairs paragraphs by index. **When the sentence counts match they align one to one; when they don't, the translation sentences are distributed by index ratio and leftovers fold into the last sentence.** The pairing is only as good as the punctuation in the source, but it keeps every sentence attached to some translation without hand alignment.

### 2. Timing Every Sentence Without an Audio Toolchain

A synced transcript needs the start and end of every sentence inside a chapter-long recording. **Narrating one sentence per call makes the boundaries known by construction, and walking the MP3 frame headers gets each clip's length without ffmpeg.** Concatenating the chunks by raw bytes then gives the chapter file, and the timestamps fall out of a running cursor.

### 3. Grading a Spoken Answer Kindly

A check-in that makes a heritage speaker feel stupid defeats the point of the app. The grading prompt tells the model to be warm and never shame, and to treat the transcription as possibly wrong, since browser speech recognition mishears. **I split the models by difficulty. Question writing is short and forgiving, so it goes to Haiku 4.5, while grading a free-form spoken answer against a rubric is the judgment step, so it goes to Sonnet 4.5.**

## Experiments

Two measurements describe what the offline pipeline produces.

**Segmentation.** The segmenter's output for all 13 chapters, counted as sentences and idiom tags per chapter.

**Narration.** The four narrated chapter MP3s, measured for duration, format and file size.

## Results

**Segmentation is deterministic and gives 13 chapters, 290 sentences and 169 idiom tags, or 0.583 tags per sentence.**

[Chart] 290 Sentences and 169 Idiom Tags Across 13 Chapters
Output of prepare-chapters.ts, by chapter number
Unit: Count
- Sentences: 1 22; 2 18; 3 18; 4 19; 5 20; 6 31; 7 23; 8 9; 9 21; 10 26; 11 48; 12 18; 13 17
- Idiom tags: 1 26; 2 7; 3 14; 4 9; 5 11; 6 9; 7 16; 8 10; 9 5; 10 22; 11 26; 12 6; 13 8
Sentences are split on the characters 。！？ and idioms are tagged by substring match against a hand-written list of 28 entries. Chapters 1 and 8 have more tags than sentences because one sentence can contain several terms. Regeneration is deterministic and needs no API key.

Chapter 11 is the longest at 48 sentences and chapter 8 the shortest at 9. **Tags run from 5 in chapter 9 to 26 in chapters 1 and 11, which reflects the hand-picked list of 28 entries more than how idiomatic a chapter is.**

**The four narrated chapters come to 944.33 s of audio**: 230.43, 213.84, 278.18 and 221.88 seconds for chapters 1 to 4. The files are 3,722,279, 3,456,875, 4,486,309 and 3,585,607 bytes, about 15.2 MB together, all mono at 44.1 kHz and roughly 129 kbps. Their embedded content credentials[5] name Eleven Labs as the generator, first as created by a trained algorithm and then as edited into a composite. Chapters 5 to 13 play through browser speech synthesis.

None of this measures whether the product teaches anything. Translation quality, grading accuracy, learning outcomes, cost and latency were outside what a weekend could measure.

## What I Would Change

### Evaluate the Grader

Grading a spoken answer against a rubric is the part that most needs evidence. A small hand-labelled set of question, rubric, answer and expected verdict, run through `/api/grade`, would give an agreement rate. Speech recognition errors are a confound, so I would start with typed answers.

### Measure Cost per Session

I would log token and character counts per session and compute the cost from the price list, so the case for an offline pipeline rests on a number.

## References

1. Sun Tzu (trans. Lionel Giles). *The Art of War*. Project Gutenberg, ebook 132. [link](https://www.gutenberg.org/ebooks/132)
2. Valdés, G. *Heritage Language Students: Profiles and Possibilities*. In Peyton, Ranard and McGinnis (eds.), *Heritage Languages in America: Preserving a National Resource*, Center for Applied Linguistics and Delta Systems, 2001.
3. Krashen, S. *Principles and Practice in Second Language Acquisition*. Pergamon Press, 1982.
4. *Web Speech API*. W3C draft specification. [link](https://webaudio.github.io/web-speech-api/)
5. Coalition for Content Provenance and Authenticity. *C2PA Technical Specification*. [link](https://c2pa.org/specifications/)
6. ElevenLabs. *Text to Speech API documentation*. [link](https://elevenlabs.io/docs)


================================================================================

# Keeping Columbus's Lead-Pipe Loan Fund From Running Dry

Date: 2026-03-28
URL: https://projects.farhansadeek.com/posts/quantathon-leap-reserve/
Tags: monte-carlo, simulation, public-policy
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/quantathon-leap-reserve
Summary: A two-person Quantathon 2026 entry sizes a $2.73M Columbus lead-pipe loan reserve (against $4.01M for grants) with a cohort Monte Carlo, and finds that one pilot-estimated uptake rate moves it by nearly $1M.

Columbus runs LEAP, a loan fund that pays for replacing lead and galvanized water service lines. It lends up to $10,000 at 0% for up to 99 years, and the loan is repaid in one lump sum when the house is sold or equity is withdrawn. For Quantathon 2026 (the SIAM-MTI challenge), Aaditya and I had to answer one question: how much should the city set aside in 2026 so the fund never runs dry before the 2037 federal replacement deadline?

We built a cohort Monte Carlo that simulates 34,269 eligible lines over twelve years on 3,000 paths and takes the 95th percentile of each path's worst cumulative deficit. The answer is **$2.73M** for loans, against **$4.01M** for an equivalent grant program. A Markov-chain benchmark built from the same inputs agrees with the simulation to within 0.015% once both measure the same quantity. The more useful finding is where the uncertainty lives. The sampling error on $2.73M is about $25k, while a plausible range for a single uptake rate estimated from the pilot moves it by nearly $1M.

## Why It Matters

Lead service lines have a hard federal deadline, and a loan fund that runs out of money halfway there stalls replacements for the households that depend on it. A grant program is simple to budget, because you spend what you give away. A loan fund is harder, because money comes back, but slowly and at times nobody controls. A loan is repaid when a house sells, and nobody knows when that will be.

The task asked for the 2026 upfront funding that sustains the program through 2037 at a 95%-sufficient level, a comparison of the loan structure with a grant, and a confidence interval on the recommendation. There are 34,269 eligible lines (whole-lead lines plus customer-side galvanized lines, minus the overlap) out of 288,389 in the system materials table. The pilot data is tiny, with 64 observed funded loan amounts with a mean of $7,386 and a cap-limited maximum of $10,000, and a snapshot of 62 active loans (42 completed, 20 in progress).

That set the design. A city cannot observe a fund's future, so the reserve has to come from a simulation of household behavior over twelve years, driven by parameters estimated from a pilot that is barely larger than a classroom. A single expected value would hide the risk, so the model produces a distribution and reports a high percentile of it. Everything else we built is there to show how far to trust that percentile.

This was a two-person team entry. Aaditya wrote the core cohort Monte Carlo and the first versions of the assumptions, charts, data loading and report driver. I built the cross-checks around it (the Markov chain, the deterministic benchmark, the bootstrap, the tornado analysis, stress tests and NPV), the ML-enhanced agent simulation, the paper and the slide deck.

## Technical Details

### Reserve as the Peak of a Cumulative Deficit

Let $O_t$ be the year-$t$ loan disbursements and $R_t$ the year-$t$ repayments on one simulated path. The fund is solvent if it never runs dry, so the reserve that path needs is the largest cumulative net outflow:

$$
D = \max_{t \in 2026..2037} \sum_{s \le t} (O_s - R_s), \qquad \text{reserve} = Q_{0.95}(D)
$$

For grants, $R_s = 0$. The maximum is over time and the percentile is over paths, and the difference between those two operations matters later.

### Loans as a Three-State Chain

Each loan is Active, Inherited or Repaid. Active loans repay on a sale (5.6% a year) or an equity withdrawal (2.0%), and become Inherited with probability 0.8%. Inherited loans repay at 8% (sale) plus 1% (equity). That is an absorbing chain, and its fundamental matrix $N = (I - Q)^{-1}$ gives expected time to repayment from each transient state[1]. I use it both as an analytic check and as the engine of the deterministic benchmark.

[Diagram] Three-state loan chain: Active moves to Inherited at 0.8% a year and to Repaid at 7.6% (sale 5.6%, equity 2.0%), Inherited moves to Repaid at 9%, Repaid is absorbing, and a side panel lists the fundamental matrix results of 12.96 expected years from Active, 11.11 from Inherited and a half-life of about 8.7 years.

### Jensen's Inequality for a Maximum

For any random process, $E[\max_t X_t] \ge \max_t E[X_t]$, because the maximum is convex[2]. A deterministic expected-value model computes the right-hand side, while a Monte Carlo average of per-path peaks computes the left, so the two differ whenever the peak moves between paths.

### Two Kinds of Uncertainty

The bootstrap[3] resamples the simulated paths and tells you how precisely a finite number of paths pins down a statistic. It says nothing about whether the parameters that generated the paths are right. Parameter uncertainty needs a different tool, such as one-at-a-time sweeps[4] or sampling parameters from priors.

### Architecture

[Diagram] Architecture of the LEAP reserve model: challenge data aggregates (34,269 eligible lines, 64 observed loan amounts, 62 active pilot loans) feed Aaditya's core cohort Monte Carlo of 3,000 paths, then per-year loan status and a per-path maximum cumulative deficit whose P95 is the reserve, $2.73M for loans and $4.01M for grants, with Farhan's cross-checks below and a dashed arrow to the PyTorch and agent simulation, P95 $3.60M, treated as an upper bound.

The core is Aaditya's chain of three steps: simulate originations and loan status per year, take each path's maximum cumulative deficit, and report the 95th percentile. My cross-checks sit around it. The ML-enhanced agent simulation is a separate, richer model that adds a regime-switching housing market and contractor capacity, and we treat its answer as an upper bound.

## Implementation

### The Cohort Monte Carlo

This part is Aaditya's, and I quote only the mechanism in condensed form. The 34,269 lines are spread across 2026 to 2037 by fixed schedule weights. Each year, every not-yet-replaced line in a later scheduled cohort originates a loan with probability $p_t = \min(0.20\% \cdot 1.05^{t-2026},\ 0.30\%)$.

```python
# condensed from solution/model.py (Aaditya): type hints removed, a local inlined
def _annual_trigger_probability(scenario, year):
    steps = year - START_YEAR
    probability = scenario.annual_trigger_start * ((1.0 + scenario.annual_trigger_growth) ** steps)
    return min(probability, scenario.annual_trigger_cap)

def _bootstrap_loan_amount(rng, history, year, inflation):
    base = history[rng.randrange(len(history))]
    return round(base * ((1.0 + inflation) ** (year - START_YEAR)), 2)
```

Loan sizes are bootstrapped from the 64 observed amounts and inflated 3% a year. Then each active loan draws once a year against the sale, equity and inheritance probabilities above. The per-path requirement is the running maximum of cumulative outflow minus inflow, and the recommendation is its 95th percentile over paths.

### The Markov Chain and Deterministic Benchmark

The transition matrix is built from the same scenario probabilities, so the deterministic model and the Monte Carlo cannot disagree because of different inputs.

```python
# solution/markov.py
p_a_repaid = s.annual_sale_probability + s.annual_equity_probability
p_a_inherited = s.annual_inheritance_probability
p_a_active = 1.0 - p_a_repaid - p_a_inherited
```

From it I derive an expected 12.96 years to repayment from Active and 11.11 from Inherited (standard deviations 12.29 and 10.60), a half-life of about 8.7 years, and the fraction of each origination year's cohort repaid by 2037: 58.4% for a 2026 loan, 7.6% for 2036 and 0% for 2037. Money lent late in the program barely comes back before the deadline.

### The Cross-Check Stack

The bootstrap resamples the 3,000 path-level requirements 2,000 times with seed 42. The tornado varies eight parameters one at a time, at the optimistic and conservative values of two hand-chosen scenarios, with 1,000 paths per variant. The stress tests and the NPV analysis each reuse the core simulator.

### The ML-Enhanced Simulation

The agent simulation scales 10,000 parcels up to 34,269 and adds regime switching (Boom, Normal, Recession with 70% persistence), a Weibull sale hazard, a deadline urgency ramp, contractor capacity, and a 4% return on the fund. Two networks, `UptakeNet` and `SaleHazardNet` (12 features, a 64-32-16 MLP, 150 epochs of Adam), supply per-parcel probabilities. They are smooth interpolators of heterogeneous assumptions: they map parcel features to per-parcel probabilities, the kind of function an MLP of this shape can approximate closely[5]. On the 10,000 v4 parcels they reproduce their targets with correlations of 0.9969 (uptake) and 0.9726 (sale).

## Problems

### 1. A Cross-Check That Looked Like a Failure

The deterministic benchmark gives a maximum cumulative deficit of **$2,471,551**. The Monte Carlo mean is $2,489,537, a gap of 0.72%, and the deterministic value falls outside the bootstrap interval for the mean, $2,484,206 to $2,494,821. Two implementations built from the same inputs should not disagree like that, so either one has a bug or the comparison is wrong.

It was the comparison. The deterministic model computes $\max_t E[X_t]$, and the interval is for $E[\max_t X_t]$. So I computed the right quantity from the same 3,000 paths, averaging the cumulative net outflow per year first and then taking the maximum over years.

```python
# max of the averaged curve, against the mean of per-path maxima
mean_curve = [fmean(r[i] for r in cum) for i in range(len(YEARS))]
max_of_mean = max(mean_curve)      # 2,471,914.86, peak year 2034
mean_of_max = base["loan_summary"]["mean"]   # 2,489,536.59
```

The result is $2,471,915, within 0.015% of the deterministic $2,471,551. **Compared on the same statistic, the two independent implementations agree to four significant digits.** The gap that remains is Jensen's inequality, and it comes from the peak year moving between paths: 1 path peaks in 2031, 47 in 2032, 618 in 2033, 1,616 in 2034, 699 in 2035 and 19 in 2036.

### 2. Separating Sampling Error from Parameter Uncertainty

The bootstrap interval for the P95 is $2,716,604 to $2,741,433, a width of about $24.8k. That is tight, and it is easy to read as the precision of the $2.73M. It is only the Monte Carlo sampling error from using 3,000 paths instead of infinitely many. **The tornado swing on the uptake start rate alone is $987,296** (P95 of $2.21M at 0.16% to $3.20M at 0.25%), about 40 times the P95 interval width and 93 times the mean interval width. **A confidence interval on the recommendation is mostly parameter uncertainty, with sampling error a small part of it.**

The rate itself, 0.20% a year, rests on a single pilot window of 62 active loans. To make the point concrete I pooled paths over a uniform prior on the uptake start rate from 0.16% to 0.25%, the tornado's own range. The prior is my assumption. The mixture has a mean of $2,514,853 and a **P95 of $3,029,090**. Treat that as an illustration of how much wider an honest interval is, not as a better estimate.

## Experiments

Each experiment varies one thing against the same core simulator:

- The base run (3,000 paths, seed 20260328), which produces the headline.
- The Jensen decomposition and peak-year histogram from those same paths.
- The bootstrap interval (2,000 resamples, seed 42).
- An uptake start-rate sweep at seven values, 500 paths each, holding the cap (0.30%) and growth (5%) at their base values.
- A pooled mixture over an assumed uniform prior on the start rate, 10 grid points times 300 paths, seed 7.
- The three stress scenarios, 1,000 paths each, seed 20260401.
- A 1,000-path run on the organizer-supplied v4 parcel file.

## Results

### The Headline

With 3,000 paths and seed 20260328 the loan requirement has a mean of $2,489,536.59, a median of $2,488,412.51, a P95 of **$2,731,561.30**, and a P2.5 to P97.5 spread of $2,201,801 to $2,771,970 (minimum $2.02M, maximum $3.09M). The grant mean is $3,708,633 and its P95 is **$4,005,262**, so **the loan structure saves $1.27M at the P95**. A full run takes 21.4 seconds of pure Python. Across the three scenarios, the loan P95 is about $2.01M (optimistic), $2.73M (base) and $3.49M (conservative), against $3.16M, $4.01M and $4.77M for grants.

### The Averaged Curve and the Peak Year

[Chart] The Average Deficit Curve Peaks Below the Reserve
Mean over 3,000 paths of cumulative loan outflows minus repayments, base scenario
Unit: Mean cumulative net outflow
- Mean cumulative net outflow: Program year (1 = 2026, 12 = 2037) 1: 470,298; Program year (1 = 2026, 12 = 2037) 2: 900,898; Program year (1 = 2026, 12 = 2037) 3: 1,289,489; Program year (1 = 2026, 12 = 2037) 4: 1,630,641; Program year (1 = 2026, 12 = 2037) 5: 1,918,398; Program year (1 = 2026, 12 = 2037) 6: 2,148,715; Program year (1 = 2026, 12 = 2037) 7: 2,322,938; Program year (1 = 2026, 12 = 2037) 8: 2,429,686; Program year (1 = 2026, 12 = 2037) 9: 2,471,915; Program year (1 = 2026, 12 = 2037) 10: 2,438,058; Program year (1 = 2026, 12 = 2037) 11: 2,330,893; Program year (1 = 2026, 12 = 2037) 12: 2,152,160
The maximum of this averaged curve is $2,471,915, in 2034. The mean of per-path maxima is $2,489,537 and the P95 of per-path maxima, the reserve, is $2,731,561. The gap between the first two is Jensen's inequality for a maximum.

[Chart] The Peak Deficit Lands in a Different Year on Each Path
Year in which each simulated path reaches its maximum cumulative deficit, base scenario
Unit: Paths (of 3,000)
- 2031: 1
- 2032: 47
- 2033: 618
- 2034: 1,616 (1,616 of 3,000 paths)
- 2035: 699
- 2036: 19
Years 2033 to 2035 hold 2,933 of the 3,000 paths. Because the peak year moves, the average of the maxima exceeds the maximum of the average.

The averaged deficit curve peaks at $2,471,915 in 2034, and paths peak anywhere from 2031 to 2036, mostly 2033 to 2035. **That spread in peak years is the whole Jensen gap.**

### One Assumption Dominates

[Chart] One Assumption Moves the Reserve by a Million Dollars
P95 of maximum cumulative deficit against the annual loan trigger start rate, 500 paths per point
Unit: P95 funding requirement
- P95 funding requirement (loans): Uptake start rate, percent per year 0.1: 1,446,972; Uptake start rate, percent per year 0.15: 2,087,936; Uptake start rate, percent per year 0.16: 2,209,057; Uptake start rate, percent per year 0.2: 2,729,754; Uptake start rate, percent per year 0.25: 3,198,070; Uptake start rate, percent per year 0.3: 3,346,834; Uptake start rate, percent per year 0.4: 3,346,834
Going from 0.16% to 0.25% moves the P95 from $2.21M to $3.20M. The bootstrap interval on the base P95 is $2.72M to $2.74M, so the Monte Carlo error is about 40 times smaller than this one parameter's effect. The points at 0.30% and 0.40% coincide because the base cap is 0.30%.

The P95 is roughly linear in the uptake start rate until the cap binds: $1.45M at 0.10%, $2.09M at 0.15%, $2.21M at 0.16%, $2.73M at 0.20%, $3.20M at 0.25%, and $3.35M at both 0.30% and 0.40%. **That is about $1.3M of reserve per 0.10 percentage points of uptake.** The 0.10% to 0.20% endpoints, $1.45M to $2.73M, give $1.28M. The tornado ranking puts uptake first, then schedule weighting, then sale probability and cost inflation. Federal rules could raise uptake by 2 to 5 times, which would move the reserve toward the surge scenario.

### Stress Scenarios

[Chart] Stress Scenarios Against the Base Reserve
P95 of maximum cumulative deficit, loans, 1,000 paths per scenario, seed 20260401
Unit: P95 funding requirement
- Base: 2,728,307
- 2029 Housing Crash: 3,139,293 (P95 +14.9% vs base; mean +15.7% vs base mean (code reported 5.4%))
- Surge Uptake: 4,975,190 (P95 +82.1% vs base; mean +87.4% vs base mean (code reported 70.8%))
- Perfect Storm: 5,754,568 (P95 +110.7% vs base; mean +117.0% vs base mean (code reported 97.7%))
Only the parameters the core model reads take effect: sale and equity probabilities for the crash, trigger start, cap and inflation for the surge. The scenarios apply to all years, not only from 2029.

On 1,000 paths the base P95 is $2,728,307. The 2029 Housing Crash, which moves the sale and equity probabilities, is $3,139,293 (+14.9%). Surge Uptake, which moves the trigger start, cap and inflation, is $4,975,191 (+82.1%), and Perfect Storm is $5,754,568 (+110.7%). The mean increases over the base mean are +15.7%, +87.4% and +117.0%. **A surge in uptake costs far more than a housing crash**, consistent with the uptake sweep.

### Other Numbers

NPV at a 3% discount rate gives a PV of outflows of $3.32M and of inflows of $1.27M, a net cost of $2,052,416 and a PV recovery ratio of 38.2%, and 1% and 5% rates move the net cost by about 3%. By 2037 the deterministic model recovers 41.97% of deployed capital, and the Markov extension projects 80% by 2051 and 90% by 2059. The ML-enhanced simulation (1,000 paths) gives a loan mean of $3,170,427 and a P95 of $3,599,806, 32% above the core P95. We treat it as an upper bound, which gives **a recommended range of $2.73M to $3.60M**.

### Sensitivity to a Different Parcel File

Run on the organizer-supplied v4 file, the model finds 6,057 eligible parcels among its 10,000, a mean loan of $9,993 (against $7,386) and a P95 of $742,138 for those parcels. Scaled by 34,269 / 6,057 that is $4.20M, 1.54 times the headline. The v4 file has a different lead share (43.8% against 70.0%) and a different loan mean, so **the gap measures how sensitive the reserve is to its inputs**, the same lesson as the uptake sweep.

## What I Would Change

### Put Priors on the Parameters

Sample the uptake, sale and inflation parameters from priors on each path and report the P95 of the mixture, as my illustration does. The reported interval should be on that quantity, and the bootstrap interval should be labeled as Monte Carlo error.

### Price the Value of Learning the Uptake Rate

With a prior in place, define a decision (the reserve level, against an uptake rate observed later) and compute the expected value of perfect information[6]. That would say in dollars how much it is worth to measure uptake better before committing the reserve, which the tornado ranking can only point toward.

### Validate Against the Real Fund

Nothing here is checked against real-world outcomes. There is no backtest and no independent estimate of the uptake rate. A better estimate of the uptake rate is the input that would tighten the recommendation most.

## References

1. John G. Kemeny, J. Laurie Snell. *Finite Markov Chains*. Van Nostrand, 1960.
2. Johan L. W. V. Jensen. *Sur les fonctions convexes et les inégalités entre les valeurs moyennes*. Acta Mathematica 30, 1906. [doi:10.1007/BF02418571](https://doi.org/10.1007/BF02418571)
3. Bradley Efron. *Bootstrap Methods: Another Look at the Jackknife*. Annals of Statistics 7(1), 1979. [doi:10.1214/aos/1176344552](https://doi.org/10.1214/aos/1176344552)
4. Andrea Saltelli et al. *Global Sensitivity Analysis: The Primer*. Wiley, 2008.
5. Kurt Hornik, Maxwell Stinchcombe, Halbert White. *Multilayer Feedforward Networks Are Universal Approximators*. Neural Networks 2(5), 1989. [doi:10.1016/0893-6080(89)90020-8](https://doi.org/10.1016/0893-6080(89)90020-8)
6. Howard Raiffa, Robert Schlaifer. *Applied Statistical Decision Theory*. Harvard University, 1961.


================================================================================

# Erasing Artists From Stable Diffusion at Right Angles

Date: 2026-02-27
URL: https://projects.farhansadeek.com/posts/cure-improve/
Tags: diffusion, unlearning, linear-algebra
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/cure-improve
Summary: An orthogonal subspace bank on top of training-free CURE erasure cuts collateral drift in Stable Diffusion v1.4 from 5 to 50 sequential erasures (LPIPS_u 0.7375 against 0.7735 at k=5), then reverses at 100.

CURE[1] erases a concept, such as a painter's style, from a text-to-image diffusion model with one closed-form edit to the cross-attention key and value weights. No training. The problem is erasing more than one. Erase concepts one after another and the edits don't compose cleanly: each pair leaves a cross term that damages images you never meant to touch. CURE-Improve is a small team research codebase built around one fix, CURE-Sequential. It keeps a bank of every direction already erased and forces each new concept's projector to be orthogonal to all of them, which makes the cross term zero by construction.

On Stable Diffusion v1.4[2], with 3 seeds at 384 px, the bank helps up to a point. From 5 to 50 erased artists, CURE-Sequential drifts less on artists it was never asked to erase (LPIPSu 0.7375 against 0.7735 at k=5, CLIPu up 0.64 to 2.23 points). At 100 erasures it flips and is worse on both measures (LPIPSu 0.7939 against 0.7680, CLIPu 21.25 against 21.69). That lines up with the idea's one real cost. CLIP's text space has 768 dimensions, and every erased concept spends some of them for good.

## Why It Matters

Concept erasure is for the case where a model has to stop producing something. CURE erases a concept in about two seconds with no gradient steps, which makes it attractive for the "erase a hundred artists" setting. That is also where it degrades. The CURE paper's Figure 6 shows perceptual divergence starting around 50 erasures and stronger interference on untargeted prompts beyond about 100. The question for this project was whether that failure can be fixed without training anything.

The work was split. Arses Prasai built the implementation, which includes a reimplementation of CURE for SD v1.4 (not the authors' code), the sequential variant, a port of the same idea to SD3's MM-DiT architecture, and a shared evaluation harness that scores every method the same way. He also ran the quick-proof benchmark. Jeffrey Xie ran the Figure-6-style sweeps behind the main results. My part is the analysis of the method gaps and a synthetic check of the linear algebra.

## Technical Details

### The CURE Edit

Take the text encoder's embeddings of prompts describing the concept, a matrix $E_f \in \mathbb{R}^{n \times 768}$. Its SVD gives right singular vectors $V_f$. CURE builds an energy-weighted projector onto that subspace and subtracts it from the key and value weights[1]:

$$
P = V_f^{\top} \operatorname{diag}\big(f(r_i;\alpha)\big) V_f, \qquad r_i = \frac{\sigma_i^2}{\sum_j \sigma_j^2}, \qquad f(r;\alpha) = \frac{\alpha r}{(\alpha - 1) r + 1}
$$

$$
W' = W - W P
$$

The function $f$ is a Tikhonov-style spectral expansion. It maps each direction's energy share $r_i$ to a weight in $[0, 1]$, so dominant directions are removed almost fully and weak ones barely at all. The implementation also supports a Gavish-Donoho hard threshold[3] that keeps directions above a noise cutoff. In that mode the weight is a hard 0 or 1, so $\alpha$ plays no part. When retain prompts are given, the discriminative projector is $P_{dis} = P_f - P_f P_r$, which stops the edit from removing directions that a concept to keep also uses.

### Why Sequential Edits Interfere

Erase concept 1, then concept 2, with the same recipe.

$$
W_2 = W_1 (I - P_2) = W_0 (I - P_1)(I - P_2) = W_0 \big(I - P_1 - P_2 + P_1 P_2\big)
$$

The first three terms are what you wanted, one projector removed per concept, but the last term, $W_0 P_1 P_2$, is an unintended edit. It is zero only when the two projectors annihilate each other, which two concepts that share directions (two painters with overlapping style words, say) won't do. So the more concepts you erase, the larger the cross term grows.

### Orthogonalizing Against a Bank

If each new concept's basis is first projected off everything already erased, the subspaces are mutually orthogonal and the cross term is exactly zero for the forget projectors, since $V_i V_j^{\top} = 0$ implies $P_i P_j = 0$. The bank $B \in \mathbb{R}^{m \times 768}$ stores an orthonormal basis of everything erased so far. For a new $V_f$:

$$
V_f^{\perp} = V_f - (V_f B^{\top}) B
$$

then re-orthonormalize with QR and append.

The price is a finite budget. Every concept spends dimensions no later concept can use, and CLIP's embedding space has only 768 of them. The guarantee is also scoped to the forget subspaces. The retain projector $P_r$ is not orthogonalized against the bank, so with retain prompts the composed projectors are not exactly orthogonal.

[Diagram] Layout of the CURE-Improve repo: concept prompts become CLIP text embeddings of shape n by 768 and feed cure/ (the base CURE reimplementation editing attn2 to_k and to_v), cure_seq/ (a SubspaceBank with orthogonal projectors that imports from cure/) and cure_dit/ (targeting SD3 MM-DiT), all of which feed evaluation/ with one results.json schema computing LPIPS_e, LPIPS_u and CLIP_u, with labels crediting the packages and protocol to Arses Prasai and the Figure-6 sweeps to Jeffrey Xie.

### How the Pieces Fit Together

The base eraser holds the attention utilities that find and edit the `attn2` layers. CURE-Sequential imports those utilities and adds the `SubspaceBank`, a spectral module that computes orthogonalized projectors, and a `SequentialCURE` class. CURE-DiT reuses the spectral idea for SD3. The evaluation harness runs any of them under one config and one results schema and computes the Figure-6 metrics. By line count that is about 2,100 lines for the base method, 1,300 for CURE-Sequential including experiments, 650 for CURE-DiT and 2,900 for evaluation.

[Diagram] Flow of one sequential erasure in CURE-Sequential: forget prompts become CLIP embeddings and an SVD gives right singular vectors, an orthogonalize step against the SubspaceBank B and QR feed an adaptive alpha and the projector P, the weights are edited as W minus W P on to_k and to_v, and new directions with spectral weight above 0.01 are registered back into the bank, which lives in the 768-dimensional CLIP space so its capacity is bounded.

### The DiT Port

SD3 has no cross-attention. Text and image tokens attend jointly, and the text stream has its own key and value projections. `cure_dit` applies the same edit to those, `add_k_proj` and `add_v_proj`, in each joint transformer block, in a 1152-dimensional context space, in float32 before casting back[4]. All the results below are on SD v1.4.

### The Metrics

The evaluation computes three numbers per checkpoint. LPIPSe[5] is the perceptual distance from the unedited model on erased artists, and higher means stronger removal. LPIPSu is the same distance on unerased artists, and lower means less collateral damage. CLIPu[6] is text-image alignment on unerased prompts, and higher is better. All three methods go through the same evaluation and output schema, so every comparison below is like for like.

## Implementation

All code excerpts below are Arses's.

### The Weight Edit

The edit itself is two matrix products per attention layer, applied to the key and value projections.

```python
# cure/attention.py
Wk_new = Wk - Wk @ projector
Wv_new = Wv - Wv @ projector
```

The layers come from walking the SD v1.4 UNet's down, mid and up blocks for every transformer block's `attn2`, the cross-attention where image tokens attend to the 768-dimensional text embedding.

### The Subspace Bank

The core operation is a projection off the bank, a drop of rows the bank has already consumed, and a QR re-orthonormalization.

```python
# cure_seq/subspace_bank.py
overlap = Vhf @ B.T             # [k, m]
Vhf_orth = Vhf - overlap @ B    # remove what earlier concepts already own
norms = torch.norm(Vhf_orth, dim=1)
Vhf_orth = Vhf_orth[norms > self.orth_threshold]
Q, _ = torch.linalg.qr(Vhf_orth.T.cpu())   # QR is not implemented on MPS
Vhf_orth = Q.T.to(dev)
```

Registering a concept does not add all of its directions. Only directions whose spectral weight exceeds 0.01 go into the bank (problem 2 is why).

### Adaptive Alpha

Orthogonalizing removes some of a concept's energy, because part of it lived in directions earlier concepts already own. To compensate, the spectral expansion parameter is boosted in proportion to the energy lost.

```python
# cure_seq/spectral.py
boost = 1.0 / max(energy_retained, 0.1)
alpha_effective = min(alpha * boost, alpha_max)   # alpha_max = 10
```

## Problems

### 1. Cross Terms Accumulate With Every Erasure

The derivation above is the reason the project exists. **Plain sequential CURE leaves a $W_0 P_i P_j$ term for every pair of erased concepts, and that term shows up as damage on generations nobody asked to change.** The bank removes it for the forget subspaces. Problems 2 and 3 are what that fix costs.

### 2. One Concept Can Consume the Whole Space

A concept described by 20 prompts at 77 tokens each yields 1,540 embedding samples, which span all 768 dimensions. **Register every direction and the first concept exhausts the bank.** The fix is to register only directions with spectral weight above 0.01, which leaves roughly 20 to 40 directions per concept. The 5-concept demo used 133 of 768 dimensions (17.3%), about 27 per concept, at about 0.25 seconds per concept.

### 3. Orthogonality Has a Budget

At 20 to 40 directions per concept, **the bank holds roughly 20 to 40 concepts before the budget tightens.** That is well under the 100 erasures of the largest experiment. It is also why the k=100 result below is the interesting one. Past the budget, later concepts get less and less room of their own.

### 4. Orthogonalizing Weakens the Edit

Projecting a concept off the bank throws away whatever energy it shared with earlier concepts, so a late concept is erased more weakly than an early one. **Adaptive alpha compensates by boosting the spectral expansion by the inverse of the energy retained, capped at 10.** That keeps late erasures strong, but it also pushes a stronger edit through a smaller subspace, which is one of the two explanations for the k=100 reversal.

## Experiments

There are three experiments.

1. **Figure-6-style sweeps (Jeffrey Xie).** SD v1.4, CURE against CURE-Sequential, seeds 11, 22 and 33, 20 sampling steps, guidance 7.5, 384 by 384 pixels, alpha 2.0, Tikhonov spectral mode, masked-mean embeddings. A 50-artist run with checkpoints at k = 1, 5, 10, 25, 50, and a 100-artist extension adding k = 100. The k = 1 to 50 checkpoints match between the two runs.
2. **Quick-proof benchmark (Arses Prasai).** Nine unique runs using a fast CLIP-based proxy with a "target" score (lower means better suppression) and a "drop" score (lower means better retention).
3. **A synthetic orthogonality check (mine).** Random low-rank concepts in 768 dimensions with no model in the loop. It tests only the linear algebra.

This is a fast setting, not a full benchmark. Three seeds, 20 steps and 384 pixels keep the sweeps cheap, and LPIPS at this scale is a proxy for what a user would call damage. Standard deviations are about 0.06 to 0.10 for LPIPSu (the largest, 0.098, is CURE-Sequential at k=100) and about 3 for CLIPu.

## Results

### Sequential Erasure on 50 and 100 Artists

[Chart] Collateral Drift on Unerased Artists
LPIPS_u against the unedited model, lower is better. Sequential erasure of up to 100 artists
Unit: LPIPS_u (lower is better)
- CURE: Artists erased so far (k) 1: 0.7094; Artists erased so far (k) 5: 0.7735; Artists erased so far (k) 10: 0.7701; Artists erased so far (k) 25: 0.7593; Artists erased so far (k) 50: 0.7612; Artists erased so far (k) 100: 0.768
- CURE-Sequential: Artists erased so far (k) 1: 0.7094; Artists erased so far (k) 5: 0.7375; Artists erased so far (k) 10: 0.7397; Artists erased so far (k) 25: 0.735; Artists erased so far (k) 50: 0.747; Artists erased so far (k) 100: 0.7939
CURE-Sequential sits below CURE from k=5 to k=50 and above it at k=100 (0.7939 against 0.7680). The gaps of 0.014 to 0.036 are smaller than or comparable to the per-point standard deviation of about 0.06 to 0.10, and I ran no significance test.

Differences, CURE-Sequential minus CURE, from Jeffrey's sweeps.

| k | dLPIPSu (lower is better) | dCLIPu (higher is better) | dLPIPSe |
|---:|---:|---:|---:|
| 1 | +0.0000 | +0.000 | +0.0000 |
| 5 | -0.0360 | +1.392 | -0.0160 |
| 10 | -0.0304 | +2.232 | -0.0265 |
| 25 | -0.0244 | +1.036 | -0.0090 |
| 50 | -0.0142 | +0.641 | -0.0292 |
| 100 | +0.0260 | -0.439 | +0.0019 |

At k=1 the two methods are identical, as they must be, because the bank is empty. **From k=5 to k=50, CURE-Sequential has lower LPIPSu and higher CLIPu at every checkpoint, but the LPIPSu advantage shrinks steadily, from 0.036 to 0.014.** It is also less aggressive on the erased artists, with a lower LPIPSe at each of those checkpoints, which is the "more surgical" behavior the bank was designed for. **At k=100 the sign flips on both preservation metrics**, and CURE-Sequential's LPIPSu spread (0.098) is wider than CURE's (0.079).

[Chart] Prompt Alignment on Unerased Artists
CLIP_u on the unerased prompts after k erasures, higher is better
Unit: CLIP_u (higher is better)
- CURE: Artists erased so far (k) 1: 25.214; Artists erased so far (k) 5: 21.2898; Artists erased so far (k) 10: 20.2503; Artists erased so far (k) 25: 21.6202; Artists erased so far (k) 50: 21.5163; Artists erased so far (k) 100: 21.6893
- CURE-Sequential: Artists erased so far (k) 1: 25.214; Artists erased so far (k) 5: 22.6821; Artists erased so far (k) 10: 22.4827; Artists erased so far (k) 25: 22.6564; Artists erased so far (k) 50: 22.1578; Artists erased so far (k) 100: 21.2503
CURE-Sequential is 0.64 to 2.23 CLIP points higher from k=5 to k=50, then 0.44 lower at k=100 (21.25 against 21.69). Standard deviations are about 3 points, so these are small effects on a proxy metric.

LPIPSu is already 0.709 at k=1 for both methods, so the baseline drift from a single erasure is large in absolute terms. **The method gaps of 0.014 to 0.036 are smaller than or comparable to the LPIPSu standard deviations**, and the CLIPu gaps of 0.6 to 2.2 sit against a standard deviation of about 3. What holds up is the direction, which is the same at all four checkpoints from k=5 to k=50.

**The k=100 reversal fits the budget story**, since 100 concepts is far past the 20-to-40 estimate. The other candidate is adaptive alpha pushing a strong edit through a shrinking subspace (problem 4), and the sweeps don't separate the two.

### Quick-Proof Benchmark

**Across the nine runs, the picture is mixed.** CURE-Sequential wins on both target and drop in the five Gavish-Donoho forward runs. Four of those are the same run in effect. The 20-concept forward runs at alpha 1.5, 2.0, 2.5 and 3.0 give identical numbers (target 26.7340 for CURE and 24.8841 for CURE-Sequential in all four), because the Gavish-Donoho threshold ignores alpha. In the reverse-order 20-concept Gavish-Donoho run, CURE-Sequential has better retention (drop 1.4540 against 4.2923) and worse suppression (target 26.1652 against 24.3610). In the Tikhonov runs, forward order favors CURE-Sequential on retention, and reverse order gives it slightly better suppression and worse retention (drop 4.7717 against 3.5260). **Erasure order matters in the reverse runs.** Nine runs of 3 to 5 seeds each don't support more than that.

### The Linear Algebra Holds

The synthetic check builds random rank-8 concepts in 768 dimensions with 40 samples each and compares the projectors' pairwise products, with and without the bank, plus the cross term $\lVert W_0 P_0 P_1 \rVert_F$ for a random $W_0$.

| Concepts | max off-diagonal $\lVert P_i P_j \rVert_F$, base | same, orthogonalized | bank dims used |
|---:|---:|---:|---:|
| 2 | 0.0163 | 2.3e-6 | 16 of 768 |
| 40 | 0.0239 | 7.5e-6 | 320 of 768 (41.7%) |

**The cross term $\lVert W_0 P_0 P_1 \rVert_F$ falls from 0.2982 to 4.2e-5, so the construction does what it says, to float32 rounding.** Random concepts in 768 dimensions are already nearly orthogonal, so the base projectors' cross terms are small to begin with (0.016 to 0.024). The check confirms the bank works as designed and shows the budget arithmetic at 8 dimensions per concept. It says nothing about real CLIP embeddings, where concepts overlap a lot.

## What I Would Change

### Log the Budget and Rerun k=100

Log the bank's dimension usage at every checkpoint and rerun the 100-artist sweep. Then run it again with adaptive alpha off. That would tell us whether the k=100 reversal is the budget, the adaptive alpha, or noise.

### Test the Retain Path

The zero-cross-term guarantee holds by construction for forget projectors only. A synthetic test with retain prompts, using the same harness as the orthogonality check, would show how large the residual cross term is when $P_r$ is not orthogonalized.

### Report the Noise Alongside the Means

Report paired differences per seed and prompt with a confidence interval. A paired comparison would pin down how large CURE-Sequential's advantage at k=5 to 50 is.

### Measure Erasure Directly

LPIPS and CLIP score drift and alignment, not whether the artist is actually gone. A classifier or a round of human ratings on the erased artists would check the removal side, which the current metrics only approximate.

## References

1. Shristi Das Biswas, Arani Roy, Kaushik Roy. *CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models*. arXiv, 2025. [arXiv:2505.12677](https://arxiv.org/abs/2505.12677)
2. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. *High-Resolution Image Synthesis with Latent Diffusion Models*. CVPR, 2022. [arXiv:2112.10752](https://arxiv.org/abs/2112.10752)
3. Matan Gavish, David L. Donoho. *The Optimal Hard Threshold for Singular Values is 4/sqrt(3)*. IEEE Transactions on Information Theory, 2014. [arXiv:1305.5870](https://arxiv.org/abs/1305.5870)
4. Patrick Esser et al. *Scaling Rectified Flow Transformers for High-Resolution Image Synthesis*. ICML, 2024. [arXiv:2403.03206](https://arxiv.org/abs/2403.03206)
5. Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, Oliver Wang. *The Unreasonable Effectiveness of Deep Features as a Perceptual Metric*. CVPR, 2018. [arXiv:1801.03924](https://arxiv.org/abs/1801.03924)
6. Alec Radford et al. *Learning Transferable Visual Models From Natural Language Supervision*. ICML, 2021. [arXiv:2103.00020](https://arxiv.org/abs/2103.00020)


================================================================================

# Fraud Triage That Spares the LLM 97% of Transactions

Date: 2026-02-06
URL: https://projects.farhansadeek.com/posts/aegis-fraud-triage/
Tags: llm-agents, fraud-detection, anomaly-detection, kafka
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/aegis-fraud-triage
Summary: A 20-hour TartanHacks pipeline that scores every card transaction with an Isolation Forest and rules, then hands only 2.9 to 3.2 percent of synthetic traffic to a local LLM agent, with precision 1.000 in all 6 runs.

A bank's fraud desk can't read every card transaction by hand, and running an LLM on every transaction is too slow and too expensive. Aegis is the funnel I built for that problem at TartanHacks, CMU's hackathon, in February 2026, with my teammate Jalen Francis. Transactions stream through a Redpanda (Kafka) topic. A cheap gate, a scikit-learn Isolation Forest[1] plus hand-written rules, scores each one. Anything above 0.55 becomes an alert, and a pool of three asyncio workers hands each alert to a ReAct-style agent[2] on a local Ollama model, which answers BLOCK, CLEAR or FLAG_FOR_REVIEW. Flagged alerts wait for a human, and every verdict goes into a FAISS index[3] that later investigations can search.

The whole design rests on the gate, so I measured it on 5,000 synthetic transactions per run over 6 runs. It sent **2.9 to 3.2 percent** of traffic to the agent with **precision 1.000 in every run**, so not one normal transaction reached the LLM. Recall was 0.368 to 0.403, and it splits cleanly by amount. In a 3,000-transaction run **every fraud above $1,000 was caught**, so the gate's recall is the share of fraud above that line.

## Why It Matters

The hackathon theme was to make a bank's fraud desk faster. The shape that fits is a funnel, where something fast and cheap looks at everything and something slow and smart looks only at what the fast stage flags. The whole system lives or dies on that first stage. Here is what Jalen and I built in roughly 20 hours. I wrote the backend and the React dashboard, and Jalen did the design, the demo and the slides.

- A stream of card transactions arriving through Kafka, the way a bank's would.
- A fast first stage that scores every transaction and lets most of them through.
- A slow second stage, an LLM agent with tools, that investigates only what the first stage flags and returns a verdict with a written reason.
- A human review queue for the verdicts the agent isn't sure about, with Slack notifications.
- A memory, so that a human decision today informs a similar investigation tomorrow.
- A live dashboard over WebSockets.
- A separate dispute-resolution agent, a Neo4j fraud-ring graph and a Redis cache as extras.

## Technical Details

### Isolation Forest as a First Stage

An Isolation Forest[1] builds random trees that split a randomly chosen feature at a random value. A point far from the rest gets separated in only a few splits, so its average path length is short, and the model turns that into an anomaly score. It needs no labels, which suits fraud, where labels arrive late. In scikit-learn, `decision_function` returns a negative value for points the model considers outliers and a positive value for inliers[4]. Aegis maps that raw value onto a 0 to 1 risk score.

### Gating an Expensive Stage

If the second stage costs seconds and the first costs milliseconds, the first stage decides how much work the second sees. Two numbers matter. Precision at the gate sets how much of the LLM's time goes to innocent transactions. Recall at the gate caps the whole system, because a fraud the gate lets through is never investigated by anything downstream. Those are the two numbers I measured.

### ReAct Without Native Tool Calling

ReAct[2] interleaves reasoning text with tool calls. The model writes a thought and names an action, and the harness runs the action and appends the observation. The prompt-based form has the model emit `Thought`, `Action` and `Action Input` as plain text, which the harness parses. That form is easy to build and sensitive to how well a small model follows a format.

### Retrieval as Memory

A sentence embedding maps text to a vector where similar meanings sit close together. FAISS[3] stores those vectors and returns nearest neighbors. With L2-normalized vectors, inner product equals cosine similarity, so an `IndexFlatIP` gives exact cosine top-k. Storing past verdicts as text and retrieving the nearest ones into the agent's prompt gives the agent a memory of what was decided before. The memory is pure retrieval, and nothing about the model or the gate changes.

### Architecture

The producer, consumer and workers all run as background asyncio tasks inside one FastAPI process. Redpanda, Neo4j and Redis run as containers beside it, and Ollama runs on the host.

[Diagram] Aegis pipeline: a mock producer publishes to a Redpanda topic, a consumer and gate scores each transaction, those above 0.55 wait in an asyncio queue read by three ReAct agent workers on Ollama llama3.2:3b (up to 6 steps, 9 tools), BLOCK or CLEAR is applied at once while FLAG_FOR_REVIEW goes to a human queue with a Slack webhook, and verdicts are stored in an in-memory FAISS index that is retrieved into the agent's prompt.

A transaction goes producer, topic, consumer, gate. Most stop there. About 3 percent cross 0.55 and go onto a plain `asyncio.Queue`, where three identical workers pull them and run the agent. The verdict either applies immediately (BLOCK and CLEAR) or waits for a human (FLAG_FOR_REVIEW). Both agent and human verdicts are embedded into FAISS, and the agent has a tool that queries it.

## Implementation

### The Features and the Forest

The forest trains at startup on 1,000 synthetic "normal" rows. There are 8 features: amount, log of amount, hour, day of week, whether the country isn't `US`, velocity (transactions in the last 10 minutes), distance in kilometers from the account's last transaction, and the account's amount z-score. Velocity, distance and z-score come from an in-process per-account tracker that keeps the last 100 transactions per account.

```python
self.model = IsolationForest(
    n_estimators=100,
    contamination=contamination,   # 0.05
    random_state=42,
)
```

### From Raw Score to Flag

This is a simplified version of `detect()`, not verbatim. The forest's output is scaled to 0 to 1 with a small Gaussian jitter, and then rules for foreign merchants, large amounts and risky categories raise the score.

```python
raw_score = self.model.decision_function(features)[0]
normalized_score = max(0.0, min(1.0, -raw_score * 2))
noise = np.random.normal(0, 0.06)
normalized_score = max(0.0, min(1.0, normalized_score + noise))

if country != "US" and amount > 1000:
    normalized_score = max(normalized_score, 0.55 + np.random.uniform(0.05, 0.25))
elif country != "US":
    normalized_score = max(normalized_score, 0.25 + np.random.uniform(0, 0.15))
if amount > 5000:
    normalized_score = max(normalized_score, 0.50 + np.random.uniform(0.05, 0.30))
if category in ("Unknown", "Financial", "Luxury", "Gambling"):
    normalized_score = min(1.0, normalized_score + np.random.uniform(0.03, 0.12))

is_anomaly = normalized_score > 0.55
```

A non-US transaction above $1,000 always lands between 0.60 and 0.80, so it always becomes an alert. Anything above $5,000 gets a floor of 0.55 to 0.80. A non-US transaction under $1,000 gets a floor of at most 0.40.

### The FAISS Novelty Boost

The consumer runs a second check on transactions the gate didn't flag. It's aimed at a small but odd purchase for one specific account, the case the amount and geography rules can't see.

```python
novelty = vector_store.compute_novelty_score(txn_data)
if novelty is not None and novelty > 0.45:
    boost = (novelty - 0.45) * 0.64
    txn.risk_score = min(1.0, txn.risk_score + boost)
    if txn.risk_score > 0.55:
        txn.is_anomaly = True
```

Novelty is `1 - (0.6 * best_sim + 0.4 * avg_sim)` over the same account's past transactions, and it needs at least 5 prior transactions for that account. Each transaction is embedded as a sentence (amount bucket, category, merchant, city, time-of-day bucket, risk score) with all-MiniLM-L6-v2[5], 384 dimensions, in an `IndexFlatIP` on normalized vectors.

### The Queue and the Workers

Flagged transactions go onto a bare `asyncio.Queue()`, and the app starts three identical worker tasks.

```python
tasks = [
    asyncio.create_task(run_producer(producer)),
    asyncio.create_task(run_consumer()),
    asyncio.create_task(run_agent_worker()),
    asyncio.create_task(run_agent_worker()),
    asyncio.create_task(run_agent_worker()),
    ...
]
```

The three workers cap how many LLM investigations run at once, and the queue absorbs bursts.

### The Agent Loop

The agent is a manual ReAct loop over `ChatOllama` at temperature 0.1, with up to 6 steps and a `TOOL_MAP` of 9 tools: `check_account_history`, `verify_merchant`, `check_travel_feasibility`, `get_account_risk_profile`, `run_kyc_check`, `find_similar_transactions`, `find_similar_investigations`, `detect_fraud_ring` and the final `recommend_action`. The default model is llama3.2:3b. Output is parsed with regexes such as `Action:\s*(\w+)` and `Action Input:\s*({[^}]+})`, with fallbacks for malformed output.

### Human Review and Memory

A `flagged` verdict sets the alert to `awaiting_review` and posts a Slack webhook. `blocked` and `cleared` apply immediately. A human decision goes through a review endpoint and is stored in FAISS with `verdict_source='human'`. Later, the `find_similar_investigations` tool retrieves the nearest past verdicts into the agent's prompt, so a reviewer's call on one case shows up as context on the next similar one.

## Problems

### 1. Tool Use on a Small Local Model

The agent runs on a local Ollama model, and llama3:8b, the model I started with, had no native tool calling through Ollama. **So I stopped asking for structured tool calls and parsed plain text instead.** The prompt asks for `Thought`, `Action` and `Action Input` lines, regexes pull out the tool name and its JSON arguments, and fallbacks handle output that doesn't match. A low temperature (0.1) and a hard limit of 6 steps bound how long any investigation runs, and `recommend_action` is itself a tool, so ending the investigation is just another action the model names.

### 2. Keeping the LLM Off Most Traffic

Transactions never stop arriving, and each investigation is up to 6 rounds of LLM reasoning and tool calls, so the agent can't look at everything. **The gate makes the agent affordable, with one scikit-learn call per transaction, 149 to 153 transactions per second on one thread, and only about 3 percent passed on.** Three workers bound how many investigations run at once. The producer, consumer and workers all run as asyncio tasks in one FastAPI process, so investigations overlap without any extra infrastructure.

### 3. Memory Without Retraining

I wanted a reviewer's decision today to inform a similar investigation tomorrow. **I made memory a retrieval problem instead of a training one.** Every verdict, the agent's or a human's, is embedded as a sentence and stored in FAISS with its `verdict_source`, and `find_similar_investigations` pulls the nearest ones into the agent's prompt. Nothing about the model or the gate changes, so there is no retraining step between a reviewer's call and the next investigation that can use it.

## Experiments

I measured the first stage on its own, because it decides what everything downstream sees. The harness runs the gate's scoring engine on a stream that mirrors the producer's mix: about 8 percent fraud (foreign merchants, with amounts drawn the way the producer draws them), 12 percent elevated normal (domestic, 2 to 5 times the category amount) and the rest normal. Each run is 5,000 transactions, over 3 seeds and 2 simulated clock start times (12:00Z and 03:00Z).

Each run records precision, recall, the share of traffic flagged and single-threaded throughput, and a separate 3,000-transaction stream breaks recall down by fraud amount. The traffic and its labels are synthetic, so the numbers describe the gate against the pipeline's own model of fraud, not against real fraud.

## Results

### The Gate

**The full gate had precision 1.000 in all 6 runs**, with 0 false positives among 3,959 to 4,011 normal and 600 to 633 elevated normal transactions per run. Recall was 0.368, 0.375 and 0.398 for the noon runs and 0.373, 0.378 and 0.403 for the 03:00 runs. Between 146 and 161 of 5,000 transactions were flagged, which is **2.9 to 3.2 percent of traffic**. Hour of day made almost no difference, within about 0.005 recall.

### Recall Tracks the $1,000 Line

The gate behaves close to "non-US merchant and amount above $1,000, or amount above $5,000". In the 3,000-transaction run, **all 76 fraud transactions between $1,000 and $5,000 and all 8 above $5,000 were flagged**. The 78 under $500 and the 71 between $500 and $1,000 went unflagged. **The 36.1 percent of injected fraud above $1,000 matched the recall (0.361) in that run**, so the recall number is a direct read of where the fraud amounts sit.

[Chart] The Gate Catches Fraud Above $1,000 and Nothing Below
First-stage gate recall by fraud amount, one 3,000 transaction synthetic run, 233 fraud transactions
Unit: Share of injected fraud flagged by the gate
- Under $500: 0 (0 of 78 flagged)
- $500 to $1,000: 0 (0 of 71 flagged)
- $1,000 to $5,000: 100 (All 76 flagged)
- Over $5,000: 100 (All 8 flagged)
The 36.1 percent of injected fraud above $1,000 matched the recall (0.361) in this 3,000-transaction run. The fraud amounts come from the repo's own producer code, so this describes the gate against the repo's synthetic fraud model, not real fraud.

### Throughput and Load on the Agent

**The gate ran at about 149 to 153 transactions per second** on one thread, with single-row scikit-learn calls. At the producer's default of one transaction every 1 to 3 seconds, it's nowhere near a bottleneck. As arithmetic, not a measurement: at the default mean of 0.5 transactions per second and about 3 percent flagged, roughly 0.015 alerts arrive per second, about one every 67 seconds, against 3 workers. Under default settings the queue should rarely build, and a backlog would come from bursts such as the demo's fraud injection.

## What I Would Change

### Label a Small Set With Subtle Fraud

The measurements above use my own fraud generator, so they measure agreement with the pipeline's own model of fraud. A labeled set with small, subtle fraud would turn the sub-$1,000 band into a number someone is trying to raise, and it would give the gate's rules and the novelty check a real target to tune against.

## References

1. Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou. *Isolation Forest*. IEEE International Conference on Data Mining (ICDM), 2008. [doi:10.1109/ICDM.2008.17](https://doi.org/10.1109/ICDM.2008.17)
2. Shunyu Yao, Jeffrey Zhao, Dian Yu et al. *ReAct: Synergizing Reasoning and Acting in Language Models*. ICLR, 2023. [arXiv:2210.03629](https://arxiv.org/abs/2210.03629)
3. Jeff Johnson, Matthijs Douze and Hervé Jégou. *Billion-scale similarity search with GPUs*. IEEE Transactions on Big Data 7(3), 2021. [arXiv:1702.08734](https://arxiv.org/abs/1702.08734)
4. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort et al. *Scikit-learn: Machine Learning in Python*. Journal of Machine Learning Research 12, 2011. [link](https://jmlr.org/papers/v12/pedregosa11a.html)
5. Wenhui Wang, Furu Wei, Li Dong et al. *MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers*. NeurIPS, 2020. [arXiv:2002.10957](https://arxiv.org/abs/2002.10957)


================================================================================

# O'Hare or Atlanta Tops Every Centrality Ranking

Date: 2026-02-05
URL: https://projects.farhansadeek.com/posts/airport-network-analysis/
Tags: network-analysis, r, quarto
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/airport-network-analysis
Summary: A Quarto notebook that turns 5,065 sampled flights into a 1,909-airport network and finds Chicago O'Hare or Atlanta at the top of all six centrality rankings.

This project analyzes a sampled **flight network** and the interconnected airports within it. Using the descriptive techniques in Kolaczyk and Csárdi's *Statistical Analysis of Network Data with R*[1], I explore structural patterns in the graph and identify important airports and routes. It is one Quarto notebook in R with igraph[6]. A sample of 5,065 flights becomes a directed graph of **1,909 airports and 2,574 routes**, and it has the hub-and-spoke shape airlines build: a small number of airports sit at the center with many connections, while most airports have only a few routes each.

The result I like most is how consistent the hubs are. I computed six centrality measures, each with its own idea of what "important" means, and **every ranking is topped by either Chicago O'Hare or Atlanta**. O'Hare leads on degree, closeness and betweenness. Atlanta leads on eigenvector centrality and on both hub and authority scores. The network is also disassortative, which means the big hubs connect to many small regional airports rather than to each other.

## Why It Matters

When I fly to San Francisco, I always have to take a layover at Chicago O'Hare. That is not an accident. Most airlines want to reduce cost, and having only a few hubs makes repair and maintenance easier, so traffic gets routed through a handful of big airports.

Network analysis puts numbers on that intuition. Degree says which airports serve the most direct routes. Strength says where people are flying a lot more frequently. Betweenness says which airports sit on the shortest paths between everyone else, which is to say which ones are the layovers. Eigenvector centrality says which airports are connected to other important airports. Different centrality measures capture different aspects of importance, so seeing where they agree and where they don't says something about how the network is built.

This is a single notebook on a sample, not a model of world aviation. The sample includes a mix of major and regional airports, and the rankings describe the airports that ended up in it.

## Technical Details

### From Flights to a Graph

[Diagram] Pipeline of the airport network notebook: 20 monthly flight lists (March 2020 to October 2021, about 8 GB in Git LFS) feed a sampling step that produced sampled_flights.csv with 5,065 rows; drop_na() keeps 3,297, group_by(origin, destination) builds the edge list, simplify() gives a directed graph of 1,909 vertices and 2,574 edges that collapses to 2,429 undirected edges for the degree, strength, cohesion, transitivity and centrality chunks, and quarto render makes a self-contained page that Netlify builds with build.sh and serves at airport.farhansadeek.com.

Each airport is a vertex, identified by its ICAO code (KORD is O'Hare, KATL is Atlanta, EDDF is Frankfurt). Each route from one airport to another is a directed edge, weighted by the number of sampled flights on it. I built both a directed graph and an undirected version that collapses the two directions of a route into one edge. The undirected graph, with 2,429 edges, carries most of the analysis. The directed one is used where direction matters, for hub and authority scores. Quarto renders the whole notebook to one self-contained page.

The sampled flights span 20 months, March 2020 to October 2021.

[Chart] Sampled Flights by Month
Rows of sampled_flights.csv by month of the day column, March 2020 to October 2021
Unit: Sampled flight rows
- 2020-03: 266
- 2020-04: 108 (Smallest month)
- 2020-05: 144
- 2020-06: 168
- 2020-07: 233
- 2020-08: 261
- 2020-09: 209
- 2020-10: 251
- 2020-11: 193
- 2020-12: 244
- 2021-01: 212
- 2021-02: 189
- 2021-03: 283
- 2021-04: 297
- 2021-05: 299
- 2021-06: 320
- 2021-07: 366 (Largest month)
- 2021-08: 345
- 2021-09: 322
- 2021-10: 355
The 20 bars sum to 5,065, from 108 rows in April 2020 to 366 in July 2021.

### What Survives Into the Graph

Not every row becomes an edge. A route needs both an origin and a destination, and a flight that departs and lands at the same airport is a self-loop, which a simple graph doesn't have.

[Diagram] From 5,065 sampled flights to a 1,909-airport graph: drop_na() drops 1,768 rows (34.9%) and keeps 3,297, of which 526 with origin equal to destination become self-loops that simplify() removes, leaving 148 airports with degree 0, and the directed graph of 1,909 airports and 2,574 edges has 393 weakly connected components drawn as a bar with a largest component of 1,177, 192 pairs, 148 singletons and 52 others.

### Degree, Strength and Neighbours

The **degree** of an airport is the number of direct connections it has. In airline networks the degree distribution is usually highly skewed: a few major hubs have many connections, while most airports have only a few routes. **Strength** is the weighted version. It adds up the flights on each route, so it finds the airports people fly through most often, where degree finds the ones with the most destinations. **Average neighbour degree** is the mean degree of an airport's neighbours. Plotting it against the airport's own degree shows whether airports with lots of connections mostly link to other well-connected airports (assortative mixing), or to smaller, less connected ones (disassortative mixing).

### Cohesion

**Vertex connectivity** is the smallest number of airports whose removal disconnects the graph, and **edge connectivity** is the same count for routes. A graph that is already in pieces scores 0 on both. **Transitivity**, also called the clustering coefficient, measures the tendency for triangles to form. In an airport network, a triangle means that if airport A has direct flights to both B and C, then B and C also have a direct flight between them.

### Centralities

Centrality measures identify the most important or influential nodes in a network, and each one captures a different notion of importance.

- **Degree centrality** is just the degree: which airports serve the most direct routes.
- **Closeness** is the inverse of the average shortest-path distance to every other airport. Airports with high closeness are well positioned to reach the entire network quickly. It is only defined within a connected piece of the graph, so I compute it on the largest component.
- **Betweenness** counts how many shortest paths between other pairs of airports pass through a given one[2], computed with Brandes' algorithm[3]. Airports with high betweenness are the critical choke points.
- **Eigenvector centrality** extends degree by weighting connections: being connected to well-connected airports matters more than being connected to poorly connected ones[4]. An airport is important if it is connected to other important airports.
- **Hub and authority scores** come from Kleinberg's HITS[5] on the directed graph. An airport is a good hub if it sends flights to many good authorities, and a good authority if it receives flights from many good hubs. In aviation terms, hubs are major departure points and authorities are major arrival destinations.

## Implementation

### Building the Network

The analysis starts from the 5,065 sampled flights. I selected only the columns needed for the analysis, constructed edge and vertex lists, and built both directed and undirected versions of the graph.

```r
sampled_df <- sampled_df |>
  select(origin, destination, latitude_1, longitude_1, latitude_2, longitude_2) |>
  drop_na()

## Edge list: weighted by flight count per route
edges <- sampled_df |>
  group_by(origin, destination) |>
  summarise(weight = n(), .groups = "drop")
```

The vertex list comes from both ends of each flight, so every airport gets its coordinates. Then igraph builds the graph. The edge list is already aggregated by route, but simplification guarantees a simple graph and removes any self-loops.

```r
flight_network <- graph_from_data_frame(d = edges, vertices = nodes, directed = TRUE)
flight_network <- simplify(flight_network, remove.multiple = TRUE, remove.loops = TRUE)

## Undirected version for symmetric analyses
flight_undirected <- igraph::as.undirected(flight_network, mode = "collapse")
```

`summary(flight_network)` reports `DNW- 1909 2574`: directed, named, weighted, 1,909 vertices and 2,574 edges.

### Vertex Characteristics

Each characteristic is a few lines of igraph plus a plot. Average neighbour degree, for example, comes from `knn()`, plotted on log axes against degree.

```r
a.nn.deg.flight <- knn(flight_undirected, V(flight_undirected))$knn
plot(igraph::degree(flight_undirected), a.nn.deg.flight,
     log = "xy",
     col = adjustcolor("steelblue", alpha.f = 0.5),
     pch = 19)
```

### Centralities

Each centrality is a one-line igraph call, a sort, and a `kable` of the top ten. Closeness first pulls out the largest component with `induced_subgraph()`.

```r
lcc <- induced_subgraph(flight_undirected,
                        which(comp$membership == which.max(comp$csize)))
close_cent <- igraph::closeness(lcc)

betw_cent <- igraph::betweenness(flight_undirected, normalized = TRUE)
top_betweenness <- sort(betw_cent, decreasing = TRUE)[1:10]
```

Hub and authority scores use `hub_score()` and `authority_score()` on the directed graph. To compare the measures, I put degree, betweenness, eigenvector centrality and strength for every airport into one data frame and drew a `pairs()` scatter matrix.

## Problems

### 1. A 1,909-Airport Graph Is Unreadable as a Raw Plot

In a network this large, raw plots become unreadable. I wanted a picture that shows the hub-and-spoke structure at a glance, so the visualization drops the edges and packs one circle per airport with packcircles. Size is betweenness, colour is degree from very low to very high, and only the top 10% by betweenness get a label. Airports with zero betweenness can't be sized, so the plot keeps only the airports with nonzero betweenness.

```r
packing <- circleProgressiveLayout(df$centrality, sizetype = "area")
df <- cbind(df, packing)
dat.gg <- circleLayoutVertices(packing, npoints = 100)
```

[Image] Bubble plot of airport centrality: one circle per airport sized by betweenness and coloured by degree quintile, with the top 10% labelled

**From this visualization I can see the hub-and-spoke structure that airlines follow.** A small number of airports, in red and orange, sit at the center with many connections, while the majority of airports cluster around the periphery with only a few routes each.

### 2. Degree Bins Collide When Degree Varies Little

The colours come from cutting degree into five quantile bins. That breaks when degree has little variation: two quantiles can land on the same value, and `cut()` refuses duplicate break points. **The fix is to keep only the unique breaks and cap the bin index at five**, so the palette always has a colour for every bin.

```r
deg_breaks <- unique(quantile(df$degree, probs = seq(0, 1, length.out = 6), na.rm = TRUE))
if (length(deg_breaks) >= 2) {
  df$deg_bin <- as.integer(cut(df$degree, breaks = deg_breaks, include.lowest = TRUE, labels = FALSE))
  df$deg_bin <- pmin(df$deg_bin, 5)  # cap at 5 for color_pal
} else {
  df$deg_bin <- 1L
}
```

### 3. Closeness Needs a Connected Graph

Closeness is an average distance to every other airport, and the distance to an airport in a different component is infinite. With 393 components, closeness on the full graph doesn't mean much. **Restricting it to the largest component, 1,177 airports, gives a closeness that means something**, at the cost of saying nothing about the airports outside it.

## Results

### The Network Is Sparse and in Pieces

| Property | Value |
| --- | --- |
| Airports (vertices) | 1,909 |
| Flight routes (edges), directed / undirected | 2,574 / 2,429 |
| Simple graph | yes |
| Weakly / strongly connected | no / no |
| Weakly connected components | 393 |
| Size of largest component | 1,177 |
| Isolates (degree 0) | 148 |
| Diameter (unweighted) | 19 |
| Average path length | 5.4329 |
| Edge density | 0.000707 |
| Vertex / edge connectivity | 0 / 0 |
| Global / average local transitivity | 0.1209 / 0.105 |

**Vertex and edge connectivity are both 0 because the graph is already in 393 pieces**, so nothing has to be removed to disconnect it. The largest component holds 1,177 airports. In a hub-and-spoke network these values are often low anyway, because removing just a few critical hubs would collapse the network.

Transitivity is low too, at 0.1209 globally and 0.105 on average per airport.

[Image] Local clustering coefficient against vertex degree for every airport

### Most Airports Have One Route, a Few Have Dozens

[Chart] Most Airports Touch One Route
Undirected degree of the 1,909 airports in the sampled network, binned
Unit: Airports
- 0: 148 (Isolates: airports whose only sampled flights start and end at the same airport, so simplify() leaves them with no routes)
- 1: 1,114 (58.4% of all airports)
- 2: 261
- 3-5: 217
- 6-10: 91
- 11-20: 49
- 21+: 29 (The busiest is KORD with degree 74)
1,523 of 1,909 airports (79.8%) have degree 2 or less. 526 of the 3,297 usable rows are same-airport flights (origin equals destination), and the 148 degree-0 airports are exactly the airports that only appear in such rows.

**The degree distribution is very skewed.** Most airports don't have very many connections, but the hubs have a lot. Of 1,909 airports, 1,114 have degree 1 and 1,523 have degree 2 or less, while O'Hare has 74.

[Image] Histogram and log-log plot of the degree distribution

The log-log plot shows a linear relationship in the tail, which is an indication of a power-law or scale-free degree distribution[7]. That fits the cost argument, since airlines concentrate routes on a few hubs.

Strength, which counts flights instead of routes, tells the same story more strongly. Both distributions are right-skewed, but **the strength distribution has an even longer tail**. The major hubs have more destinations, and their flights are more frequent too.

[Image] Side-by-side histograms of vertex degree and vertex strength

### Hubs Connect to Small Airports

[Image] Log vertex degree against log average neighbour degree

**The plot shows a negative trend: higher-degree airports tend to be connected to neighbours with lower average degree.** This is textbook disassortative mixing, which is characteristic of hub-and-spoke transportation networks. The big hubs connect to many small regional airports, which in turn have the hub as their most prominent neighbour. This contrasts with social networks, which are typically assortative: popular people befriend other popular people.

### The Same Airports Win Every Ranking

[Chart] The Ten Highest-Degree Airports
Undirected degree in the sampled network, ICAO codes
Unit: Distinct neighbouring airports
- KORD: 74
- KATL: 71
- KDFW: 57
- KDEN: 49
- KLAX: 46
- KCLT: 43 (Sixth, which makes Charlotte one of the major hubs in the sample)
- KPHX: 42
- KSEA: 37
- EDDF: 36 (Frankfurt, the only non-US airport in the top 10)
- KPHL: 35
Degree counts distinct routes in this sample, not flights.

| Measure | Top five |
| --- | --- |
| Degree | KORD 74, KATL 71, KDFW 57, KDEN 49, KLAX 46 |
| Closeness (largest component) | KORD 0.000260, KDFW 0.000250, KEWR 0.000246, KSFO 0.000245, KATL 0.000244 |
| Normalized betweenness | KORD 0.0663, KDFW 0.0384, EDDF 0.0367, KLAX 0.0342, KATL 0.0313 |
| Eigenvector | KATL 1.000, KORD 0.777, KSEA 0.722, KLAS 0.674, KLAX 0.663 |
| Hub score | KATL 1.000, KORD 0.861, KSEA 0.847, KLAX 0.805, KDEN 0.760 |
| Authority score | KATL 1.000, KORD 0.707, KLAS 0.588, KDFW 0.561, KSEA 0.556 |

**Chicago O'Hare or Atlanta is first on every one of the six measures**, and Dallas-Fort Worth, Denver and Los Angeles keep showing up behind them. By degree, Dallas-Fort Worth and Charlotte are among the major hubs in the network, with Charlotte sixth at 43 routes.

For closeness, as expected, Dallas is high up the list, in second. Newark, San Francisco, Miami and Amsterdam move into the top ten, and Charlotte drops out of it. I think the most likely reason is that many of Charlotte's flights end in the network rather than following on to more destinations.

Betweenness has the major hubs on top again, meaning they are the airports that connect flights from one airport to another. That is my San Francisco trip: I always lay over at O'Hare, and **O'Hare's betweenness of 0.0663 is about 1.7 times the next airport's**. Frankfurt and Amsterdam are third and eighth, and Washington Dulles is tenth at 0.0254.

Eigenvector centrality brings Seattle and Las Vegas up to third and fourth, because they are connected to other well-connected airports. Hub scores follow the busiest departure points, and Charlotte reappears ninth there.

I was surprised that Detroit only just makes the hub list, tenth at 0.557, because Detroit is a major Delta hub. One reason I can think of is that Detroit is not as sought-after a destination as Chicago, Dallas, San Francisco or New York. But it seems like Detroit is a good location for a connecting hub.

### How the Measures Agree

[Image] Pairwise scatter plots of degree, betweenness, eigenvector centrality and strength

[Chart] How Strongly the Centralities Agree
Spearman correlation across all 1,909 airports, undirected graph
Unit: Spearman rank correlation
- Degree vs strength: 0.984
- Degree vs betweenness: 0.917
- Strength vs betweenness: 0.898 (Strength tracks betweenness more closely than it tracks eigenvector centrality)
- Strength vs eigenvector: 0.672
- Degree vs eigenvector: 0.67
- Betweenness vs eigenvector: 0.565
All six pairs are positive, so the major hubs tend to score high on every measure. Eigenvector centrality agrees least with the others.

**All the centrality measures are positively correlated, so the major hubs tend to score high across all of them.** Strength and degree are almost the same measure here, with a Spearman correlation of 0.984, and betweenness tracks both closely, at 0.917 against degree and 0.898 against strength. **Eigenvector centrality is the one that goes its own way**, at 0.565 to 0.672 against the others, because it rewards being attached to important airports rather than having many routes. Betweenness can be high for airports that serve as critical bridges even if they don't have many direct connections.

## References

1. Eric D. Kolaczyk, Gábor Csárdi. *Statistical Analysis of Network Data with R*, 2nd edition. Springer, 2020. [doi:10.1007/978-3-030-44129-6](https://doi.org/10.1007/978-3-030-44129-6)
2. Linton C. Freeman. *A Set of Measures of Centrality Based on Betweenness*. Sociometry 40(1), 35–41, 1977. [doi:10.2307/3033543](https://doi.org/10.2307/3033543)
3. Ulrik Brandes. *A Faster Algorithm for Betweenness Centrality*. Journal of Mathematical Sociology 25(2), 163–177, 2001. [doi:10.1080/0022250X.2001.9990249](https://doi.org/10.1080/0022250X.2001.9990249)
4. Phillip Bonacich. *Factoring and Weighting Approaches to Status Scores and Clique Identification*. Journal of Mathematical Sociology 2(1), 113–120, 1972. [doi:10.1080/0022250X.1972.9989806](https://doi.org/10.1080/0022250X.1972.9989806)
5. Jon M. Kleinberg. *Authoritative Sources in a Hyperlinked Environment*. Journal of the ACM 46(5), 604–632, 1999. [doi:10.1145/324133.324140](https://doi.org/10.1145/324133.324140)
6. Gábor Csárdi, Tamás Nepusz. *The igraph Software Package for Complex Network Research*. InterJournal Complex Systems, 1695, 2006. [link](https://igraph.org)
7. Aaron Clauset, Cosma Rohilla Shalizi, M. E. J. Newman. *Power-Law Distributions in Empirical Data*. SIAM Review 51(4), 661–703, 2009. [doi:10.1137/070710111](https://doi.org/10.1137/070710111)


================================================================================

# Trial Emulation With a Critic That Guards Time Zero

Date: 2026-01-25
URL: https://projects.farhansadeek.com/posts/clinova-trial-emulation/
Tags: llm-agents, causal-inference, healthcare
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/clinova-trial-emulation
Summary: Clinova is a team-built agent pipeline that drafts a target trial emulation from a free-text clinical question, and on the Barrett 2006 benchmark its LLM validator rejected two designs over time-zero alignment before accepting the third, a loop that took 490 of the run's 837 seconds.

Turning a clinical question into an observational study is easy to get subtly wrong, and an easy way to break one is to start a patient's follow-up at the wrong moment. Clinova is a hackathon-style agent pipeline our team built to draft a target trial emulation (TTE)[1] from a question in plain English: one agent turns the question into a causal PICO, a second drafts the trial protocol, a third audits the protocol against a written rubric, a lookup tool maps clinical terms to OMOP concept IDs[6], and a last agent writes Python for the All of Us enclave.

The interesting part is the critic. On a benchmark against the Barrett 2006 trial of two contrast agents in chronic kidney disease[7], **the validator rejected the first design, rejected the second, and accepted the third, and both rejections were about when follow-up starts.** The second rejection caught a bias the design agent introduced while fixing the first. The whole run took **837 seconds**, and **490** of them went to the validate-and-revise loop.

## Why It Matters

Target trial emulation asks for discipline that is easy to state and easy to break. Before touching data you write down the eligibility criteria, the treatment strategies, the assignment procedure, the outcome, the follow-up, the causal contrast and the analysis plan, and you make time zero, the start of follow-up, coincide with the moment eligibility is met and treatment is assigned[2]. Getting that wrong produces bias that no estimator fixes afterwards.

The idea of Clinova was to let language models do the drafting and make a second model do the checking. A researcher puts a clinical question in plain English on one end and gets runnable analysis code on the other, with a written rubric in between so the critic checks the design against something other than its own taste.

We picked five published contrast-nephropathy trials as benchmark targets, so that the pipeline's designs could be compared with designs real investigators had already committed to. Barrett is the one this post follows end to end. Praneel Patel put together the Barrett run and the other examples the website shows. My part was the `GET /api/examples` endpoint that serves them and the frontend integration and UI.

## Technical Details

### Time Zero Is the Cardinal Rule

In a randomized trial, eligibility, randomization and the start of follow-up happen at the same instant. In observational data they don't, and the ways they drift apart have names. If a patient must survive until they start treatment to be counted as treated, the treated group is handed guaranteed event-free time, which is immortal time bias[3]. If follow-up starts after treatment has already begun, early events are missed, which is left truncation. If eligibility is decided using information from after time zero, the cohort is selected on the future.

Our `rubric.md`, 638 lines long, turns this into a check the validator can apply. Three timestamps must coincide, eligibility, assignment and the start of follow-up, and every eligibility rule must be evaluable from information available at or before that timestamp.

### The Pipeline

The pipeline is a straight chain with one loop.

[Diagram] Clinova pipeline: a free-text clinical question flows through the Question, Design and Validator agents on gpt-5.2, with a dashed feedback arrow from the validator back to the design agent for at most 3 revise_spec iterations, then a local OMOP concept lookup and the Code agent on gemini-3-pro-preview, which produces analysis code for the All of Us enclave.

The question, design and validator agents are configured for `gpt-5.2` in `agent/config.py`, and the same file sets `GEMINI_MODEL = "gemini-3-pro-preview"` for the code agent. The OMOP lookup is a local tool with no LLM call. The validator can send the design back to the design agent at most three times. For Barrett, the code agent's output was a 470-line analysis script.

### Why a Critic Instead of a Better Prompt

A design spec is a long document. The Barrett one is 341 lines in its first version, and a subtle error in a window definition on line 200 is easy to write and easy to miss. Asking the same model to be more careful is a weak lever. A separate call whose only job is to apply a fixed list of gates, with structured output, gives a checkable signal, and its JSON feeds straight into a targeted revision prompt. The loop is bounded, because a critic can also oscillate.

Here is what that looked like on Barrett.

[Diagram] Three design spec drafts and the validator's verdicts on the Barrett 2006 run: v1 (t0 = contrast administration time) gets FEEDBACK with 3 CRITICAL and 1 WARNING issues, v2 (t0 = max of procedure and drug start) gets 1 CRITICAL left-truncation issue, and v3 (t0 = drug_exposure_start_datetime) is VALID with 6 of 6 gates passed.

### From Free Text to Concept IDs

Analysis code needs concept IDs from the OMOP vocabulary, not phrases like "iodixanol" or "multidetector computed tomography". The lookup tool is a tiered search. It tries an exact match on normalized text, then a RapidFuzz fuzzy match with an early exit when the score is at least 95, then a semantic search over SapBERT[4] embeddings stored in a FAISS index[5]. Ties are broken with per-domain vocabulary weights, for example RxNorm for drugs and LOINC for measurements. On Barrett, the exact tier resolved `iodixanol` to 19003201 and `iopamidol` to 19081224, both RxNorm at confidence 1.0.

## Implementation

### The Validation Loop

The loop in `agent/main.py` is short. Validate, and if the answer isn't VALID, format the issues and ask the design agent for a targeted revision rather than a fresh draft.

```python
while not is_valid and iteration < MAX_VALIDATION_ITERATIONS:
    iteration += 1
    is_valid, feedback = validator_agent.validate(design_spec, rubric)
    all_feedback.append(f"=== Iteration {iteration} ===\n{feedback}")
    if not is_valid:
        formatted_feedback = validator_agent.format_feedback_for_revision(feedback)
        design_spec = design_agent.revise_spec(
            causal_question=causal_question,
            previous_spec=design_spec,
            feedback=formatted_feedback,
            rubric=rubric,
            sotastack=sotastack,
        )
```

### A Validator That Returns JSON

The validator prompt asks for three things, time-zero alignment, PICO completeness and OMOP mappability, and tells the model to be lenient about wording and non-critical methodology so it hunts for fatal flaws and not for polish. The output is fixed JSON.

```json
{
  "status": "VALID" or "FEEDBACK",
  "failed_gates": [],
  "issues": [
    { "gate": "Gate 4", "section": "Time-Zero Alignment",
      "severity": "CRITICAL" or "WARNING",
      "issue": "Description of the problem",
      "fix": "Specific action to fix" }
  ],
  "passed_gates": ["Gate 1", "Gate 2", "Gate 3", "Gate 4", "Gate 5", "Gate 6"]
}
```

The `fix` field matters as much as the `issue` field. `format_feedback_for_revision` turns each issue into a block with the problem and a required fix, and that block is all the revising agent sees of the critique.

### The Tiered Lookup

The search function returns as soon as a cheap tier is confident, and only falls through to the expensive one when it has to.

```python
exact_results = self._exact_match(normalized_query, domain_filter)
if exact_results:
    ...  # confidence 1.0, return immediately

fuzzy_results = self._fuzzy_match(normalized_query, tokens, domain_filter)
if fuzzy_results:
    best_fuzzy = fuzzy_results[0]
    if best_fuzzy["score"] >= self.config.fuzzy_high_confidence:  # 95.0
        ...  # return immediately

# Tier 3: semantic search, then merge and rank
```

The thresholds live in `omop_lookup/config.py`: 0.85 for a strong semantic match and 0.60 as the floor, 95 and 70 for fuzzy, and 0.75 as the confidence below which the tool returns candidates instead of a single answer.

## Problems

### 1. The First Draft Selected Patients on the Future

The first Barrett design read fine and was wrong in three places. **The validator's first pass found three CRITICAL time-zero issues, all of them windows that looked past the start of follow-up.** The eligibility rule linked a qualifying CT to the contrast drug record with a window of up to 6 hours after time zero, so whether a patient was eligible depended on the future. The mixed-exposure exclusion looked forward from 1 hour before to 6 hours after time zero. The intra-arterial exclusion used a window of zero to one day around time zero. It also raised one Gate 3 WARNING about datasets that record dates and not datetimes.

Each is a subtle window definition, the kind of error that's easy to write and easy to miss in a 341-line spec.

### 2. Fixing One Bias Introduced Another

For the eligibility issue, the critic suggested `t0 = min(CT procedure_datetime, contrast drug_exposure_start_datetime)`, or requiring the procedure time to be at or before the drug start plus a small tolerance. The design agent picked `max()` instead: `t0 = max(procedure_datetime, drug_exposure_start_datetime)`, so that both pieces of evidence sit at or before time zero. That satisfies the eligibility complaint and creates a new bug. If the drug was recorded before the CT, treatment starts before time zero, which is left truncation.

**The second validation pass flagged exactly that as its only CRITICAL issue** and recommended `t0 = drug_exposure_start_datetime`. Version 3 adopted it, and the third pass returned VALID with all six gates passed.

So the loop converged, and it needed all three rounds, because the revision that fixed one violation introduced another. **A single pass of critique would have shipped version 2.** That's the case for a bounded loop over a one-shot review.

## Results

### Where the Time Goes

[Chart] Where One Pipeline Run Spent 14 Minutes
Barrett 2006 benchmark run, 2026-01-25, 837 s in total, from the committed run_log.txt timestamps
Unit: Wall-clock time per stage
- Question agent: 87 (Wrote a 6,718 character causal question)
- Design agent: 173 (Wrote a 22,127 character design spec)
- Validation loop: 490 (Iterations took 216 s, 221 s and 53 s, and include the two revise_spec calls)
- OMOP lookup: 6 (Local tool, no LLM call)
- Code agent: 81 (Wrote a 16,368 character script)
Times are the gaps between log lines, so the validation figure covers the three validate() calls and the two revisions between them. One run, one question, not repeated.

The run took 837 seconds end to end. **The validation loop was 490 seconds of that, about 59 percent**, over three iterations of 216, 221 and 53 seconds. Iterations 1 and 2 each include a design revision, and iteration 3 is a validate call alone, so a lone validate takes about 53 seconds. The design agent took 173 seconds, the question agent 87, the code agent 81, and the OMOP lookup 6, because it's a local lookup with no model call. **The validate-and-revise loop is the most expensive stage in the pipeline.**

### The Validator Converged in Three Rounds

[Chart] The Validator Converged in Three Rounds
Barrett 2006 run, issues per validator response
Unit: Issues raised by the validator
- CRITICAL (Gate 4, time zero): Iteration 1 3; Iteration 2 1; Iteration 3 0
- WARNING (Gate 3, mappability): Iteration 1 1; Iteration 2 0; Iteration 3 0
Iterations 1 and 2 returned FEEDBACK and iteration 3 returned VALID with 6 of 6 gates passed. The design spec grew from 341 to 381 to 385 lines. One run, so this shows the loop working once, not how often it converges.

Iteration 1 returned four issues, three CRITICAL and one WARNING. Iteration 2 returned one CRITICAL. **Iteration 3 returned none and passed 6 of 6 gates.** The design spec grew from 341 lines to 381 to 385.

### Drug Names Resolve Exactly

[Chart] How the First 20 Terms Were Matched
Barrett run, top-ranked concept per query term, 20 of the 47 terms have committed results
Unit: Query terms, top-1 match type
- Exact name: 6 (Confidence 1.0)
- Exact synonym: 5 (Confidence 1.0)
- Fuzzy name: 9 (The lowest top-1 is 0.4061, for angiography with contrast (intra-arterial), a LOINC radiology concept with no analytics concept ID)
14 of the 20 top-1 hits have confidence of at least 0.95, and 4 of the 20 have no analytics_concept_id (iodinated contrast media, intra-arterial angiography, eGFR, BUN). No semantic-tier match appears, and there is no ground truth, so these are match types and not accuracy.

Of the 20 query terms in the Barrett lookup, 6 were exact name matches, 5 exact synonym matches and 9 fuzzy name matches. **Fourteen top-ranked hits have confidence of at least 0.95**, and the two contrast agents at the center of the trial, iodixanol and iopamidol, both resolve exactly to their RxNorm concepts.

## What I Would Change

### Make the Critic Verify, Not Only Suggest

The critic suggested `min()` and the reviser chose `max()`, which introduced a new violation, and "converged" here means an LLM said VALID. A cheap improvement is a deterministic pre-check that parses each time-zero definition and tests the three-way coincidence rule on synthetic timestamps, with the LLM critic left for what can't be parsed. Both failure modes in the Barrett run, look-forward windows and treatment before time zero, are checkable that way.

### Run All Five Benchmarks, More Than Once

Barrett shows what a full run should leave behind. Running all five trials, repeating each to see how often the validator converges and whether the same critique shows up twice, and comparing the designs and results with the published effect sizes would turn one good run into an evaluation.

## References

1. Hernán MA, Robins JM. *Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available*. American Journal of Epidemiology 183(8):758-764, 2016. [link](https://doi.org/10.1093/aje/kwv254)
2. Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. *Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses*. Journal of Clinical Epidemiology 79:70-75, 2016. [link](https://doi.org/10.1016/j.jclinepi.2016.04.014)
3. Suissa S. *Immortal Time Bias in Pharmacoepidemiology*. American Journal of Epidemiology 167(4):492-499, 2008. [link](https://doi.org/10.1093/aje/kwm324)
4. Liu F, Shareghi E, Meng Z, Basaldella M, Collier N. *Self-Alignment Pretraining for Biomedical Entity Representations*. NAACL 2021. [link](https://arxiv.org/abs/2010.11784)
5. Johnson J, Douze M, Jégou H. *Billion-scale similarity search with GPUs*. IEEE Transactions on Big Data, 2019. [link](https://arxiv.org/abs/1702.08734)
6. Hripcsak G, Duke JD, Shah NH, Reich CG, Huser V, et al. *Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers*. Studies in Health Technology and Informatics 216:574-578, 2015. [link](https://pubmed.ncbi.nlm.nih.gov/26262116/)
7. Barrett BJ, Katzberg RW, Thomsen HS, Chen N, Sahani D, Soulez G, et al. *Contrast-induced nephropathy in patients with chronic kidney disease undergoing computed tomography: a double-blind comparison of iodixanol and iopamidol*. Investigative Radiology 41(11):815-821, 2006. [link](https://pubmed.ncbi.nlm.nih.gov/17035872/)


================================================================================

# Graphing What Biomedical Papers Mention Together

Date: 2025-10-25
URL: https://projects.farhansadeek.com/posts/empirica-biomedical-graphs/
Tags: nlp, graphs, biomedical
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/empirica-biomedical-graphs
Summary: Empirica, a two-day hackathon app that turns biomedical PDFs into a NetworkX knowledge graph; on three open-access papers co-occurrence supplied 44 to 190 edges against 1 to 14 from verb patterns, and about 98 percent of the time was scispaCy NER.

Someone facing a stack of papers on one topic first wants to know which genes, drugs, diseases and cell types they keep mentioning together. Empirica turns biomedical PDFs into a knowledge graph that shows it. A scispaCy model[1] finds the entities, any two entities that share a sentence get an edge, NetworkX finds communities and bridge nodes, and the result is drawn as an interactive 2D or 3D force-directed graph[2], with a Claude-backed chat and hypothesis generator on top. Three of us built it in about 31 hours at a weekend hackathon.

On three open-access papers the graphs were small, 20 to 37 nodes, and nearly every edge came from plain co-occurrence: 44 to 190 per paper, against 1 to 14 from verb patterns. The scispaCy pass took 7.2 to 12.8 seconds per paper and everything after it under a quarter of a second, so the model is the pipeline.

## Why It Matters

The idea was a reading aid. Instead of skimming forty abstracts, you upload the PDFs and get a map of which entities the papers talk about together, which clusters they form, and which entities sit between clusters, with the sentences behind every edge and a chat box for questions.

The scope was broad for a weekend: several PDFs per project, each with its own graph that can be switched on and off; biomedical entity extraction; communities and centrality so the graph says something beyond a hairball; an LLM chat and hypothesis generator; an agent that finds more papers; and Google sign-in with a project list per user.

Jalen Francis built most of it, including the NLP and graph pipeline, the retrieval layer, the LLM service, the agentic module and most of the frontend. Edward Kim wrote the first Google OAuth pass. I built the login screen and branding and the backend's health-check and auth-check endpoints.

## Technical Details

### Entities Are the Easy Part, Relations Are the Hard Part

Biomedical named entity recognition is a mature task. Empirica uses `en_ner_bionlp13cg_md`, one of the scispaCy models[1], trained on the BioNLP 2013 Cancer Genetics corpus[3], which labels things like genes or gene products, chemicals, cancers, cell types and organisms. Given a sentence, it returns character spans and labels.

Relations are harder, because saying how two entities are related means reading the sentence. Empirica does the cheapest thing and keeps the sentence so a human can judge.

### Co-occurrence as an Edge

If two entities appear in the same sentence, draw an edge between them and count how many sentences they share. That is the oldest trick in literature mining, and Swanson's early literature-based discovery work leaned on it at the level of whole articles[4]. It has high recall and low precision: two entities can share a sentence because one causes the other, because they were measured in the same assay, or because they sit in the same list, and the edge cannot tell which. What it does well is stay auditable. Keep the sentence, and every edge can be checked.

[Diagram] Two ways Empirica makes an edge: sentence-level co-occurrence adds weight 1 to every entity pair in a sentence and keeps up to three evidence sentences, and regex verb patterns with single-word captures turn 'the p38 MAPK inhibits NF-kB signaling' into an edge with source MAPK and target NF because the hyphen cuts the entity.

Empirica makes edges two ways. The co-occurrence extractor pairs up the entities in each sentence and keys an edge on the sorted pair, so A to B and B to A are the same edge. Weight is the number of shared sentences, and each edge stores up to three of them as evidence. The second extractor runs seven regex patterns (`CAUSES`, `CAUSED_BY`, `INHIBITS`, `INHIBITED_BY`, `INTERACTS_WITH`, `TREATS`, `REGULATES`) of the form `(\w+)\s+(inhibits?|blocks?|...)\s+(\w+)` to give an edge a type and a direction. A match is kept if its source or target is in the sentence's entity dictionary.

### Communities and Centrality

Two standard tools describe a weighted graph's structure. Louvain community detection[5] greedily merges nodes into groups that maximise modularity, which rewards groups with dense internal edges and sparse edges between them. Betweenness centrality[6] scores a node by the fraction of shortest paths between other nodes that pass through it, so it finds bridges rather than merely popular nodes. Both are one-line calls in NetworkX[7] and python-louvain.

### Layout and Retrieval

A force-directed layout treats edges as springs and nodes as repelling charges and lets the system settle[2], so clusters appear as visual clumps without any coordinates from the data. The frontend uses `react-force-graph` for both the 2D and the 3D view.

Retrieval-augmented generation feeds an LLM passages fetched from a corpus alongside the question[8]. The usual retriever embeds query and passages and returns the nearest; Empirica already has a graph of entities, so its retriever keys on entities instead.

### Architecture

[Diagram] Empirica architecture in two rows: biomedical PDFs go through PyMuPDF, scispaCy NER, a co-occurrence and regex relationship extractor and GraphBuilder into one NetworkX graph per PDF stored in SQLite and shown in a React 2D and 3D force-directed view, while a chunker and RAG index with no embedding step yet feed Claude 3 Haiku and 3.5 Sonnet for the chat and hypothesis bar.

The backend is FastAPI with SQLAlchemy on SQLite, and the frontend is React 18 with TypeScript, Vite, Tailwind and Zustand. Google OAuth handles sign-in. A user creates a project and uploads PDFs to it, and each PDF is processed independently into its own graph.

The bottom row is the layer on top. `DocumentChunker` splits each PDF into overlapping chunks that record their page and entities, `RAGService` maps entities to chunk ids, the LLM layer talks to Claude 3 Haiku and Claude 3.5 Sonnet, and the agentic module drives PubMed, ClinicalTrials.gov and Google Scholar searches, feeding what it finds back through the same NER, relationship and graph code.

## Implementation

### Extracting Entities, and Throwing Most of Them Away

PyMuPDF extracts each PDF's text, the project's PDF processor splits it into sentences, and each sentence goes through the scispaCy model. Jalen's `NERService` maps the model's fine-grained labels onto a smaller set of app types, drops entries that look like journal names or affiliations with a list of regex exclusions, and then applies the filter that shapes everything downstream: an entity has to appear in at least four sentences.

```python
self.min_entity_occurrences = 4

def filter_entities(self, sentence_entities):
    entity_counts = self.get_entity_counts(sentence_entities)
    ...
    filtered_entities = [
        entity for entity in sent_data["entities"]
        if entity_counts.get(self._normalize_entity(entity["text"]), 0) >= self.min_entity_occurrences
    ]
```

Counts are per sentence, so an entity repeated within one sentence counts once.

### Merging the Two Kinds of Edge

When a pattern hits a pair that co-occurrence already found, the merge keeps the co-occurrence weight, adds 2.0, appends the evidence, and overwrites the edge type with the semantic one.

```python
if edge_key in merged:
    merged[edge_key]["weight"] += rel["weight"]
    merged[edge_key]["evidence"].extend(rel["evidence"])
    merged[edge_key]["relationship_type"] = rel["relationship_type"]  # Prefer semantic type
```

`GraphBuilder` then loads the merged edges into a NetworkX graph, dropping any edge whose endpoints are not among the kept entities, calls `community_louvain.best_partition`, and computes betweenness centrality, returning the top 20 nodes.

### An Entity-Indexed Retriever

`RAGService.index_document` stores the chunks and, for each chunk, appends its id to a dictionary from entity name to chunk ids. Retrieval takes the entities found in the user's question, scores each chunk by how many of them it mentions, and takes the top few. If the graph is loaded, it also adds up to five graph neighbours of each entity and attaches up to three of their edges with evidence sentences. That is entity-indexed chunk lookup plus one hop of graph expansion, a reasonable design for a graph-first app.

### Sign-in and Health Checks

My part sat at the edges of the system. I rewrote and extended Edward's login component into the login screen and did the branding. On the backend I added the CORS configuration and three small endpoints, `/api/health`, `/api/test-auth`, and `/api/projects-test`, which lists the signed-in user's projects.

```python
@app.get("/api/health")
async def health_check():
    return {"status": "healthy", "message": "API is running", ...}

@app.get("/api/test-auth")
async def test_auth(current_user: User = Depends(get_current_user)):
    return {"status": "authenticated", "user_id": current_user.id, ...}
```

The health endpoint shows a deployed backend is up at all, and the auth check shows whether a Google token resolves to an account.

## Problems

### 1. The Raw Entity List Is Noisy

The raw scispaCy output is noisy, down to journal names and affiliations, which is why `NERService` maps labels, applies regex exclusions and keeps only entities seen in at least four sentences. **The frequency filter is what makes the graph readable.** On the first paper below, 501 sentences contained an entity before the filter and 429 after, and only 37 distinct entities survived in a 648-sentence paper. The price is that **the graph summarises the frequent entities only**: a rare but important gene never appears.

### 2. A Shared Sentence Does Not Say How Two Entities Relate

A co-occurrence edge between a drug and a gene might be an inhibition or just a shared list. **Every edge carries up to three evidence sentences, so a reader can judge any edge in the app.** The verb patterns add typed, directed relations on top, and the merge gives a pattern-matched pair an extra 2.0 of weight and its semantic type.

### 3. Several Papers, One Graph

A researcher wants to see a project's PDFs together, apart, or with one left out. Each PDF's nodes and edges live in `pdf_graph_nodes` and `pdf_graph_edges`, keyed by document, and each `Document` has a `selected` flag (default 1). The merged-graph route merges the selected documents on request and records their filenames in the response metadata. **Unticking a PDF removes its contribution without reprocessing anything, and provenance is preserved** because every node and edge still belongs to one document. The cost is that merging happens at request time and identical entity names are the only merge key.

## Experiments

The experiment takes three CC BY papers and runs each through the project's `PDFProcessor`, `NERService`, `RelationshipExtractor` and `GraphBuilder` unchanged, timing each stage on an Apple silicon Mac's CPU. There are no LLM calls. The papers are a JNK and cisplatin review (PMC13028654), a study of MK2, p38 and p53 in macrophages (PMC13072840), and a SNORA64 and apoptosis paper (PMC13090758). The last one has no license tag in its XML, though Europe PMC's metadata lists it as CC BY.

Each input is the paper's full-text XML from Europe PMC[9], written into a plain-text PDF with PyMuPDF. That gives a single column with no headers or footnotes, which probably makes sentence splitting cleaner than on a real two-column PDF. Each paper was run once, and with no ground truth these numbers say how the pipeline behaves, not whether an edge is correct.

## Results

### The Graphs Are Small and Uneven

[Chart] The Final Graphs Are Small, and Density Varies Nearly Threefold
Graph size after the 4-occurrence entity filter, three papers
Unit: Count
- Nodes: JNK and cisplatin review (PMC13028654) 37; MK2/p38/p53 and IL-1b (PMC13072840) 27; SNORA64 and apoptosis (PMC13090758) 20
- Edges: JNK and cisplatin review (PMC13028654) 113; MK2/p38/p53 and IL-1b (PMC13072840) 115; SNORA64 and apoptosis (PMC13090758) 23
- Louvain communities: JNK and cisplatin review (PMC13028654) 7; MK2/p38/p53 and IL-1b (PMC13072840) 4; SNORA64 and apoptosis (PMC13090758) 10
Density is 0.170, 0.328 and 0.121, and average degree is 6.11, 8.52 and 2.30. The SNORA64 paper has 20 nodes and 23 edges but ten communities, the largest with 5 nodes, so its graph is close to a set of small clusters. Edges are counted after GraphBuilder drops pairs whose endpoints did not survive the filter, so they are fewer than the merged edge count.

The 648-sentence JNK review produced 37 nodes and 113 edges in 7 communities, the largest with 13 nodes. The MK2 paper, at 407 sentences, gave 27 nodes and 115 edges in 4 communities, with a density of 0.328, the highest of the three. The SNORA64 paper gave 20 nodes, 23 edges and 10 communities with the largest at 5, an average degree of 2.3. **Graph size does not follow paper length; it follows how repetitive a paper is about a handful of entities.**

In the first two papers the top betweenness node is the generic word "cells" (0.226 and 0.248), with "A549 cells" (0.199) and "cancer" (0.128) next in the first. In the third it is the paper's own subject, SNORA64 (0.231), followed by BCL-XL (0.097). **Centrality found the topic in one paper and a generic noun in two.** The entity filter drops rare entities but not uninformative frequent ones.

### Co-occurrence Does the Work

[Chart] Regex Patterns Add Few Edges Next to Co-occurrence
Edges from each extractor on three open-access papers, before merging and before GraphBuilder drops edges
Unit: Edges before GraphBuilder drops any
- Co-occurrence edges: JNK and cisplatin review (PMC13028654) 190; MK2/p38/p53 and IL-1b (PMC13072840) 143; SNORA64 and apoptosis (PMC13090758) 44
- Verb-pattern edges: JNK and cisplatin review (PMC13028654) 14; MK2/p38/p53 and IL-1b (PMC13072840) 10; SNORA64 and apoptosis (PMC13090758) 1
Pattern edges are 1 to 14 per paper against 44 to 190 co-occurrence edges. Each pattern needs a single word on either side of the verb, so multi-word or hyphenated entities such as NF-kB or A549 cells cannot match whole. Pattern edges that hit an existing co-occurrence pair replace its edge type.

The first paper had 190 co-occurrence edges and 14 pattern edges, the second 143 and 10, the third 44 and 1. After merging, 188 of the first paper's 200 edges are still typed `CO_OCCURRENCE`. **The typed relations are a thin layer over an undirected association graph.** Before believing any typed edge in the app I would read its evidence sentences, which is what the three-sentence evidence store is for.

The graphs have fewer edges than the merged extractors produced (113 against 200, 115 against 151, 23 against 45), because `GraphBuilder` drops any edge whose endpoints are not among the kept entities.

### The Model Is the Pipeline

[Chart] NER Is Almost All of the Pipeline Time
Seconds for the NER pass on each paper, one run each
Unit: Wall-clock seconds for scispaCy NER, CPU
- JNK review, 648 sentences: 12.79 (Everything else took 0.21 s: 0.121 s for text and sentences, 0.089 s for filter, relationships, graph and analytics)
- MK2/p38/p53, 407 sentences: 7.386 (Everything else took 0.111 s: 0.066 s and 0.045 s)
- SNORA64, 419 sentences: 7.17 (Everything else took 0.096 s: 0.062 s and 0.034 s)
Louvain and betweenness centrality on graphs of 20 to 37 nodes are negligible, so the cost that would scale with corpus size is the model. Single runs on an Apple silicon CPU, so treat differences between papers as approximate.

NER took 12.79, 7.39 and 7.17 seconds. Text extraction and sentence splitting took 0.06 to 0.12 seconds, and everything else (entity filter, relationships, graph, Louvain, betweenness) took 0.03 to 0.09 seconds. **On graphs this small the analytics are free, so a corpus of a hundred papers costs about what a hundred NER passes cost.**

## What I Would Change

### Evaluate Edges Against Something

I would hand-label 50 co-occurrence edges and 50 pattern edges from the three graphs by reading their evidence. Precision for each extractor would tell me whether the typed edges are worth their weight, and whether count-weighted co-occurrence is better than a single shared sentence.

### Handle Generic Entities

Generic terms such as "cells" and "cancer" dominate betweenness in two of three papers. A stoplist, or downweighting entities that appear across many papers, would leave the graph closer to what a reader wants.

## References

1. Mark Neumann, Daniel King, Iz Beltagy, Waleed Ammar. *ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing*. Proceedings of the 18th BioNLP Workshop and Shared Task, ACL, 2019. [link](https://aclanthology.org/W19-5034/)
2. Thomas M. J. Fruchterman, Edward M. Reingold. *Graph drawing by force-directed placement*. Software: Practice and Experience 21(11), 1991. [doi:10.1002/spe.4380211102](https://doi.org/10.1002/spe.4380211102)
3. Sampo Pyysalo, Tomoko Ohta, Makoto Miwa et al. *Overview of the Cancer Genetics (CG) task of BioNLP Shared Task 2013*. Proceedings of the BioNLP Shared Task 2013 Workshop, ACL, 2013. [link](https://aclanthology.org/W13-2008/)
4. Don R. Swanson. *Fish oil, Raynaud's syndrome, and undiscovered public knowledge*. Perspectives in Biology and Medicine 30(1), 1986. [doi:10.1353/pbm.1986.0087](https://doi.org/10.1353/pbm.1986.0087)
5. Vincent D. Blondel, Jean-Loup Guillaume, Renaud Lambiotte, Etienne Lefebvre. *Fast unfolding of communities in large networks*. Journal of Statistical Mechanics: Theory and Experiment, 2008. [doi:10.1088/1742-5468/2008/10/P10008](https://doi.org/10.1088/1742-5468/2008/10/P10008)
6. Linton C. Freeman. *A set of measures of centrality based on betweenness*. Sociometry 40(1), 1977. [doi:10.2307/3033543](https://doi.org/10.2307/3033543)
7. Aric A. Hagberg, Daniel A. Schult, Pieter J. Swart. *Exploring network structure, dynamics, and function using NetworkX*. Proceedings of the 7th Python in Science Conference (SciPy), 2008. [link](https://conference.scipy.org/proceedings/SciPy2008/paper_2/)
8. Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*. Advances in Neural Information Processing Systems (NeurIPS), 2020. [link](https://arxiv.org/abs/2005.11401)
9. Europe PMC. *Europe PMC RESTful Web Service* (full-text XML endpoint). Europe PMC documentation. [link](https://europepmc.org/RestfulWebService)


================================================================================

# The Encoder Sets the Scale of Commit Drift

Date: 2025-06-21
URL: https://projects.farhansadeek.com/posts/diffsense/
Tags: git, embeddings, fastapi, hackathon
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/diffsense
Summary: DiffSense scores how far each commit drifts semantically by embedding its diff and message. On its own 17 commits, CodeBERT gives a median drift of 0.0242 and MiniLM gives 0.679, and the two ranges never overlap.

DiffSense is a web app and VS Code extension for asking questions about a git repository's history. Jalen Francis, Jayson Clark and I built it in two days (June 21–22, 2025) at the AI Berkeley Hackathon. It clones a repository, scores each commit for "semantic drift" by embedding its diff and message, flags likely breaking changes by comparing function and class signatures, and puts a chat on top that can call Claude[1]. Jalen built the backend, Jayson built most of the frontend and the extension, and I built the mobile layout and the markdown rendering in the chat.

Pointed at DiffSense's own history, the analysis teaches one main lesson, that **a drift score only means something next to the encoder that produced it**. Over the same 17 commits, the same formula gives a median drift of **0.0242** with CodeBERT[2] and **0.679** with MiniLM[3], and the two ranges never overlap. The signature-based breaking-change detector caught all 8 real breaks in 14 small cases I wrote and labeled.

## Why It Matters

The pitch is to point DiffSense at a repository and see where it changed and which commits are likely to break callers, without reading every diff.

Drift was the part I wanted to understand. A commit is a diff plus a message. Embed both, compare each commit to a reference, and look at the distance, and you get a cheap timeline of where a codebase changed character. The alternative is asking an LLM about every commit, which costs a call per commit and isn't reproducible. An embedding distance is one number, computed locally, the same on every run.

A number like that is only useful if you know what a big value looks like. So the second half of this post runs the drift score and the breaking-change detector on real history to see what the numbers actually do.

## Technical Details

### Drift as One Minus Cosine Similarity

Given two commit vectors $u$ and $v$, the drift is

$$
d(u, v) = 1 - \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert}.
$$

Identical directions give 0, orthogonal vectors give 1, and opposite vectors give 2. The score is only about direction. A commit that touches ten times as many lines but points the same way in embedding space scores the same. That makes it cheap and explainable, and it also means the score says nothing about how much changed.

DiffSense's `calculate_drift_score` compares each commit to the first one, so its score is cumulative distance from where the project started. In my measurements I also compare each commit to the previous one, which reads as a step size.

### The Hybrid Embedding

A commit has two views, code in the diff and prose in the message, so DiffSense embeds them separately and blends them. The code encoder is CodeBERT[2], mean pooled over the last hidden state, and the text encoder is a sentence transformer, `all-MiniLM-L6-v2`[3], which was trained so that cosine similarity between sentence vectors is meaningful[4]. The hybrid is

$$
h = 0.7 \, e_{\text{code}} + 0.3 \, e_{\text{text}},
$$

with the 384-dimensional text vector zero-padded to CodeBERT's 768 dimensions.

### Why the Encoder's Geometry Sets the Scale

Cosine distance only spreads out if the encoder spreads unrelated inputs across the sphere. Sentence-transformer models are trained to do that. Mean-pooled hidden states of a masked language model are not. Contextual embeddings from such models are known to occupy a narrow cone, where unrelated inputs still have high cosine[5]. An encoder like that pushes every drift toward zero, so the same formula reads on a completely different scale depending on which model sits under it.

### Breaking Changes Without a Model

The other detector doesn't use embeddings at all. It compares the API surface of a file before and after a commit. For Python it parses both versions with the standard `ast` module and collects a signature for every function, async function and class. For JavaScript, Java and C++ it does the comparison with regular expressions. A removed name, a renamed name or a changed parameter list is a candidate breaking change, and a separate behavioral-change category covers edits that keep the signature. It's a heuristic with no trained classifier behind it.

### Architecture

One FastAPI service sits between the clients and everything else. `main.py` is 3,913 lines with 36 route decorators (`@app.get`, `@app.post`, `@app.put`, `@app.delete`). The routes include clone-repository, repository stats and commits, an enhanced RAG query, per-commit analysis, analyze-range and a dashboard. The 11 modules under `src/` total about 8,700 lines, and the whole backend with `main.py` is about 12,600.

[Diagram] DiffSense architecture: a React, Vite and Tailwind dashboard and a VS Code chat extension both call one FastAPI service (main.py, 3,913 lines, 36 routes) that uses seven backend modules by Jalen Francis: github_service, git_analyzer, embedding_engine, breaking_change_detector, rag_system, claude_analyzer and a SQLite database.

Git access goes through GitPython and a GitHub service with OAuth. Analysis is split into the embedding engine and the breaking-change detector. Chat is a RAG system plus a Claude analyzer, and SQLite (the standard `sqlite3` module) stores users, chats, commits, analyses and cached file contents. The React dashboard and the VS Code chat webview both call the same API.

The drift path in detail:

[Diagram] The hybrid drift score blends CodeBERT on the diff lines (0.7) with MiniLM on the commit message (0.3, zero-padded to 768 dimensions) as 1 minus cosine similarity against a 0.3 threshold, and measured on 17 commits MiniLM alone gives median drift 0.679 with 16 of 16 over threshold while CodeBERT gives median 0.024 with 0 of 16 over.

## Implementation

### Embedding a Diff (Jalen Francis)

A diff becomes one string, with removed lines first and added lines after, each behind a marker token. That string is embedded like any other code, so a 2,000-line commit is truncated at 512 tokens.

```python
def embed_diff(self, added_lines, removed_lines):
    diff_text = ""
    if removed_lines:
        diff_text += "[REMOVED] " + "\n".join(removed_lines) + " "
    if added_lines:
        diff_text += "[ADDED] " + "\n".join(added_lines)
    if not diff_text.strip():
        diff_text = "[EMPTY_DIFF]"
    return self.embed_code(diff_text)
```

`embed_code` runs the tokenizer with `max_length=512, truncation=True`, then takes `outputs.last_hidden_state.mean(dim=1)`. The engine can also run `all-MiniLM-L6-v2` as the code encoder in CodeBERT's place, and that path is what I measure as "MiniLM" below.

### The Blend and the Score (Jalen Francis)

```python
hybrid_embedding = code_weight * code_embedding + text_weight * text_embedding
```

```python
drift_score = 1 - similarity  # Higher drift = lower similarity
```

### Signature Extraction With `ast` (Jalen Francis)

The Python half of the breaking-change detector visits every function, async function and class and stores a signature keyed by qualified name.

```python
def visit_FunctionDef(self, node):
    sig = self.detector._create_function_signature(node, self.current_class)
    signatures[sig.name] = sig
    self.generic_visit(node)

def visit_ClassDef(self, node):
    old_class = self.current_class
    self.current_class = node.name
    sig = self.detector._create_class_signature(node)
    signatures[sig.name] = sig
    self.generic_visit(node)
    self.current_class = old_class
```

Comparing the before and after dictionaries gives removals, renames and parameter changes. The detector's typed output uses `ChangeType`, `ImpactSeverity` and `ChangeIntent` enums, and it can hand its findings to Claude when `enable_ai_analysis` is set. That module is 2,824 lines.

### Claude and Retrieval (Jalen Francis)

`claude_analyzer.py` wraps the Anthropic SDK and defaults to `claude-3-sonnet-20240229`, overridable with `CLAUDE_MODEL`. Retrieval for the chat lives in `rag_system.py`, which is 1,511 lines.

### The Frontend (Jayson Clark, and My Parts)

Jayson wrote most of the dashboard, the API integration and the extension. The extension is a webview chat, with `extension.ts` and a React `ChatApp.tsx` under `media/`.

**Mobile layout.** The mobile layout spans the dashboard, main content and sidebar components. The core of it is a sidebar that is off canvas below the `lg` breakpoint and slides in over a dimmed overlay.

```jsx
{isMobileSidebarOpen && (
    <div className="fixed inset-0 bg-black bg-opacity-50 z-40 lg:hidden"
         onClick={() => setIsMobileSidebarOpen(false)} />
)}
<div className={`
    fixed lg:relative lg:translate-x-0 z-50 lg:z-auto
    transform transition-transform duration-300 ease-in-out
    ${isMobileSidebarOpen ? 'translate-x-0' : '-translate-x-full'}
`}>
```

On desktop the sidebar is a normal flex child. On small screens it's `fixed` and translated out of view until the top bar toggles it, and a tap on the overlay closes it.

**Markdown in chat.** Chat replies go through a `MarkdownRenderer` component built on `react-markdown`, `remark-gfm`, `rehype-highlight` and `highlight.js`. It swaps the highlight theme with the page's dark mode by injecting a stylesheet and removing it on unmount.

```jsx
const isDark = document.documentElement.classList.contains('dark')
existingStyles.forEach(style => style.remove())
link.href = isDark
  ? 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/github-dark.min.css'
  : 'https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/github.min.css'
```

Jayson later built on the same file with file and commit-hash linkifying: a mention like `src/App.jsx` or a 7-character hash in a reply becomes a click target that opens the file viewer. That code is his.

## Problems

### 1. Knowing What a Big Drift Looks Like

Drift is only useful if you can tell a big step from a small one, and the formula alone doesn't say. So I ran the identical 17 commits through the identical 0.7/0.3 blend with two code encoders and looked at where the numbers land. **CodeBERT's drift stays between 0.0135 and 0.0762, while MiniLM's runs from 0.501 to 0.985.**

A small diagnostic on five unrelated code snippets (Python, SQL, Java, C and NumPy) shows why. **Raw CodeBERT vectors for unrelated code have pairwise cosine between 0.935 and 0.963, while MiniLM's range from -0.078 to 0.284.** CodeBERT sees all code as similar, so every drift it reports is small. That's the narrow-cone picture from [Technical Details](#why-the-encoder-s-geometry-sets-the-scale). The answer is that **a drift value has to be read against its own encoder's range**, never as an absolute number.

### 2. Finding Breaks Without a Model

Asking Claude whether every commit breaks callers costs a call per commit. The detector instead reduces each file to a dictionary of signatures and diffs the dictionaries. Python gets an exact parse through `ast`, and JavaScript, Java and C++ get regular expressions, so one pass covers the languages the dashboard shows. To check it, I built a small repo with 14 one-commit edits in Python and JS and labeled each one. **It caught every one of the 8 real breaks**, from a removed class to a changed JS parameter count, with no model call at all.

## Experiments

1. **Drift.** Take each of the 17 commits that change Python, JS or TS files, oldest first. Build the patch, embed the diff and the message with the hybrid, and compute 1 minus cosine to the previous commit and to the first (what DiffSense itself does). Run it once with CodeBERT as the code encoder and once with MiniLM. That's 16 comparisons each.
2. **Diagnostic.** Embed five unrelated snippets with both encoders and record the min and max pairwise cosine over the 10 pairs.
3. **Planted breaking changes.** Build a small synthetic repo, apply 14 one-commit edits (Python and JS), and label each breaking or not. The detector's heuristic path runs with no Claude. 8 cases are breaking and 6 are not.
4. **History.** Run the detector over every commit of DiffSense itself and count findings. There are no labels.

## Results

### Drift Under Two Encoders

[Chart] Drift Score per Commit Under Two Code Encoders
1 minus cosine similarity to the previous commit (my variant), 16 comparisons over DiffSense's own history, 0.7/0.3 hybrid
- MiniLM fallback: Commit number, oldest first (1 is the first comparison) 1: 0.639; Commit number, oldest first (1 is the first comparison) 2: 0.574; Commit number, oldest first (1 is the first comparison) 3: 0.8443; Commit number, oldest first (1 is the first comparison) 4: 0.7258; Commit number, oldest first (1 is the first comparison) 5: 0.6211; Commit number, oldest first (1 is the first comparison) 6: 0.7191; Commit number, oldest first (1 is the first comparison) 7: 0.5005; Commit number, oldest first (1 is the first comparison) 8: 0.9849; Commit number, oldest first (1 is the first comparison) 9: 0.7829; Commit number, oldest first (1 is the first comparison) 10: 0.7694; Commit number, oldest first (1 is the first comparison) 11: 0.536; Commit number, oldest first (1 is the first comparison) 12: 0.8278; Commit number, oldest first (1 is the first comparison) 13: 0.5055; Commit number, oldest first (1 is the first comparison) 14: 0.5505; Commit number, oldest first (1 is the first comparison) 15: 0.5043; Commit number, oldest first (1 is the first comparison) 16: 0.8539
- CodeBERT default: Commit number, oldest first (1 is the first comparison) 1: 0.0181; Commit number, oldest first (1 is the first comparison) 2: 0.031; Commit number, oldest first (1 is the first comparison) 3: 0.0239; Commit number, oldest first (1 is the first comparison) 4: 0.0226; Commit number, oldest first (1 is the first comparison) 5: 0.0162; Commit number, oldest first (1 is the first comparison) 6: 0.0244; Commit number, oldest first (1 is the first comparison) 7: 0.0165; Commit number, oldest first (1 is the first comparison) 8: 0.0223; Commit number, oldest first (1 is the first comparison) 9: 0.0141; Commit number, oldest first (1 is the first comparison) 10: 0.0135; Commit number, oldest first (1 is the first comparison) 11: 0.0251; Commit number, oldest first (1 is the first comparison) 12: 0.0484; Commit number, oldest first (1 is the first comparison) 13: 0.0354; Commit number, oldest first (1 is the first comparison) 14: 0.0314; Commit number, oldest first (1 is the first comparison) 15: 0.0616; Commit number, oldest first (1 is the first comparison) 16: 0.0762
- Default threshold 0.3: Commit number, oldest first (1 is the first comparison) 1: 0.3; Commit number, oldest first (1 is the first comparison) 16: 0.3
The two encoders disagree by a factor of about 28 in median (0.679 against 0.024), and the fixed 0.3 threshold falls in the gap: every comparison is over it with MiniLM and none is with CodeBERT.

| Code encoder | Dimensions | Min | Median | Max |
|---|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | 0.501 | 0.679 | 0.985 |
| microsoft/codebert-base (default) | 768 | 0.0135 | 0.0242 | 0.0762 |

**The two encoders put drift on scales that don't overlap, and CodeBERT's largest step (0.0762) is far below MiniLM's smallest (0.501).** CodeBERT's biggest steps are the last two commits, Jalen's that integrated change detection into more models (0.062) and Jayson's "More complete frontend" (0.076). MiniLM's biggest is Jalen's "major updates to architecture" (0.985). **The two encoders don't pick the same commit as the biggest move.**

The raw pairwise cosine on unrelated snippets:

| Encoder | Min cosine | Max cosine |
|---|---|---|
| CodeBERT | 0.935 | 0.963 |
| MiniLM | -0.078 | 0.284 |

### Breaking-Change Detector on Planted Cases

[Chart] Breaking-Change Detector on 14 Hand-Labeled Cases
Python and JavaScript edits applied to a small synthetic repo, each case labeled breaking or not
Unit: Cases out of 14 planted cases
- True positive: 8 (Removed or renamed function, class or method, added or removed required parameter, JS export removed, JS parameter count changed)
- True negative: 3 (Added function, comment-only change, JS added function)
- False positive: 3 (Optional parameter added, docstring-only change, body change with the same signature)
- False negative: 0
It caught every planted break and over-flagged three harmless edits. Fourteen hand-picked cases show the detector's shape, not its accuracy on real code.

**The detector caught all 8 real breaks and missed none.** That covers every removed function, renamed function, removed method, removed class, added or removed required parameter, removed JS export and changed JS parameter count, all from the heuristic path with no Claude call.

### Findings Over Real History

[Chart] Breaking-Change Findings per Commit in DiffSense's History
The detector run over the 17 commits the script analysed (root omitted, it has none), oldest first
Unit: Findings per commit, unlabeled
- da590d2 Jayson: New frontend: 23
- 0019d66 Farhan: added mobile responsive: 3
- 1245771 Jayson: Setup VSCode extension: 0
- f6a88b5 Jalen: updated architecture: 21
- bb3bc4a Jayson: Integrated API into web app: 3
- eb75bb4 Jayson: Frontend updates: 3
- ce28f3a Farhan: Markdown on chat implementat: 0
- d60f7c6 Jalen: major updates to architectur: 72
- a58992e Jayson: Fixed bugs: 3
- dd934c5 Jalen: integrated github oauth and: 95
- fac5584 Jayson: Integrated api 2.0: 11
- d473a23 Jalen: updates to database to suppo: 30
- 8fd4b85 Jalen: greatly improved rag model.: 7
- 2bdd3f8 Jalen: added advanced breaking_chan: 2
- d395719 Jalen: integrated change deteciton: 440
- 011a1b8 Jayson: More complete frontend: 4
717 findings across 16 commits, 440 of them in one commit. With no ground truth for these commits I cannot say how many are real, and the planted run shows it over-flags some harmless edits.

**Over DiffSense's own history, 14 of 17 commits had at least one finding, for 717 in total, and one commit accounts for 440 of them**: Jalen's "integrated change deteciton in further models". That's also where CodeBERT registers one of its two biggest drift steps. My mobile-layout commit gets 3 findings, on `MainContent` and `Sidebar`, and my markdown commit gets 0.

## What I Would Change

### Read Drift Against the Repository's Own History

Since the scale belongs to the encoder, I'd report drift relative to the data. Compute drift for many commits of many repositories, look at its distribution for the chosen encoder, and rank a commit by percentile, or mark it as an outlier against the repository's own drift history.

### Use a Code Encoder Trained for Similarity

Mean-pooled CodeBERT vectors all sit close together. A model trained with a contrastive objective for code retrieval, or CodeBERT with whitening or mean-centering of the vectors, would spread them out. I'd try mean-centering first because it's one line and testable with the same 10-pair diagnostic.

### Label Commits Before Adding Features

I'd write down 50 commits from a public repository with a human label for "behavior changed" and "API broke", then score the drift and the detector against it. Without labels every improvement is a guess. The 14 planted cases are a starting test file.

## References

1. Anthropic. *Claude API documentation*. Anthropic, 2025. [link](https://docs.anthropic.com/)
2. Zhangyin Feng, Daya Guo, Duyu Tang, et al. *CodeBERT: A Pre-Trained Model for Programming and Natural Languages*. Findings of EMNLP, 2020. [arXiv:2002.08155](https://arxiv.org/abs/2002.08155)
3. Wenhui Wang, Furu Wei, Li Dong, et al. *MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers*. NeurIPS, 2020. [arXiv:2002.10957](https://arxiv.org/abs/2002.10957)
4. Nils Reimers and Iryna Gurevych. *Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks*. EMNLP-IJCNLP, 2019. [arXiv:1908.10084](https://arxiv.org/abs/1908.10084)
5. Kawin Ethayarajh. *How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings*. EMNLP-IJCNLP, 2019. [arXiv:1909.00512](https://arxiv.org/abs/1909.00512)


================================================================================

# A Pill Dispenser Built Around One Dose Record

Date: 2025-03-29
URL: https://projects.farhansadeek.com/posts/pillifyai/
Tags: hardware, iot, mobile
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/pillifyai
Summary: A Make2025 hackathon build in about 19.5 hours: a prescribing app, a Node server and an ESP32 dispenser that polls every 30 seconds, all sharing one dose record.

A prescription is a piece of paper and a promise. Nobody can tell whether the pill actually came out of the bottle. At the Make2025 hackathon, Jalen Francis and I built PillifyAI in about 19.5 hours to close that loop. A doctor prescribes in a mobile app. A server turns the prescription into scheduled dose records. A dispenser asks the server what's due, releases the dose, and reports back, so the patient's history can tell a dose the device confirmed from one the patient only claimed.

What made it work is one decision. **A dose is a single row, and every part of the system reads and writes that row.** The app, the server and the dispenser never talk to each other directly, and the row keeps "the patient says they took it" separate from "the device says it released it". Under the hood there's an Expo and React Native app, a Node, Express and Postgres server, an ESP32 that polls the server every 30 seconds, and Raspberry Pi scripts that drive a hacked SG90 servo and a piezo buzzer.

## Why It Matters

I wanted the prescription to become data, the schedule to become rows, and the pill to come out of a box that tells the server it did.

For one hackathon, the scope came down to three things:

- A mobile app where a doctor can add patients and prescribe, and a patient can see today's medications and their history.
- A server that owns the schedule and the truth about each dose.
- A dispenser that asks the server what's due and answers with what it did.

This was a hackathon build, not a clinical product. The point was to get the loop working across a phone, a server and a microcontroller in one overnight session at the end of March 2025. The app came to about 5,954 lines of TypeScript and JavaScript, the server about 3,364 lines of JavaScript, the ESP32 sketch 307 lines, and the Pi scripts 383 lines of Python.

## Technical Details

### One Record as the Shared Truth

The whole design comes down to one idea, and it lives in the dose row. The doctor's prescription creates that row as pending, the dispenser's report marks it dispensed, and the patient's history reads it. Since the state lives only in that row, the three clients never need to coordinate, and each one can be built and tested against the server alone. With two people and one night, that mattered.

The `MedicationTracking` model has exactly the fields that idea needs. Condensed from the source, with layout and some `allowNull` options trimmed:

```js
scheduledFor: { type: DataTypes.DATE, allowNull: false },
takenAt:      { type: DataTypes.DATE, allowNull: true },
status: {
  type: DataTypes.ENUM('pending', 'taken', 'missed', 'skipped'),
  defaultValue: 'pending',
},
deviceId:        { type: DataTypes.STRING, allowNull: true },
requestedByApp:  { type: DataTypes.BOOLEAN, defaultValue: false },
deviceDispensed: { type: DataTypes.BOOLEAN, defaultValue: false },
```

The split between `status` and `deviceDispensed` is the design choice I care about most. A patient can mark a dose taken in the app, and the dispenser can separately report that it released one. Because they're separate fields, the history can tell a claim apart from a confirmation.

### Polling Instead of Pushing

A microcontroller on home WiFi is hard to reach from outside, so the dispenser asks the server instead of waiting to be told. Polling on an interval is the simplest design that works under that constraint, and it fits the request-response REST style the server already speaks[1]. The cost is latency, and the interval sets the limit. With a 30 second poll, a due dose waits at most 30 seconds before the dispenser notices. A pill scheduled for a given minute can wait that long, and the behavior is easy to reason about.

### Hobby Servos and Pulse Width

A standard hobby servo expects a pulse every 20 ms (50 Hz), and the width of that pulse, usually 1 to 2 ms, sets the shaft angle[2]. A servo modified to rotate continuously reads the same pulse as a speed and direction around a neutral width instead. We hacked an SG90, and our code drives it by speed, so the further the duty is from neutral, the faster it turns.

At 50 Hz the period is 20 ms, so you get the pulse width by multiplying the duty cycle by the period: 2.5 percent is 0.50 ms, 7.5 percent is 1.50 ms and 12.5 percent is 2.50 ms.

### Auth Lives in Firebase, Roles Live in Postgres

Firebase handles credentials. The server keeps its own user, doctor and patient tables. It depends on firebase-admin, whose usual job is verifying Firebase ID tokens[3], and it mirrors users into Postgres, where the doctor and patient roles are stored and joined to prescriptions. That's why the auth routes are mostly about keeping users in sync rather than logging them in.

### Architecture

The system is three clients around one server and one database.

[Diagram] PillifyAI architecture: a Firebase auth box (email login, doctor or patient role) points to both the Expo and React Native mobile app, whose tabs are Home, History and Account plus doctor-only Patients and Prescribe and which keeps an offlineOperationsQueue in AsyncStorage, and to the Node Express server; the app and the server talk over REST (POST /prescribe, PUT /status/:id); the server exposes /api/auth, /api/patients, /api/medications and /api/devices and holds a Postgres database with the User, Doctor, Patient, Medication and MedicationTracking models; an ESP32 sketch that polls every 30 s, shows its state with LEDs on pins 2, 4 and 5, and blinks, dispenses and reports when a dose is due talks to the server both ways through GET esp32/medications and POST esp32/dispense/:id with an X-API-Key header; a separate box holds the Raspberry Pi scripts (hacked SG90 servo on GPIO 2 at 50 Hz, piezo buzzer on GPIO 18).

The mobile app and the server talk over REST. The app's service layer covers today's medications, history, updating a taken status, and prescribing. The server mounts four route groups under `/api` (`auth`, `patients`, `medications` and `devices`). The device routes are the dispenser's entire interface: a ping, a listing of patients and medications, and a POST that marks a dose dispensed, sent with an `X-API-Key` header.

There are two pieces of hardware code. The ESP32 sketch handles the network side. The Raspberry Pi scripts handle the mechanical side, driving the servo and the buzzer.

One dose moves through the system like this.

[Diagram] One dose across three lanes: the doctor prescribes in the app's Prescribe tab (name, dosage, frequency, time); the server writes a Medication row plus scheduled MedicationTracking rows in the pending state; the ESP32 polls GET esp32/medications every 30 s and finds a due dose; its dispensing LED on pin 4 blinks at 500 ms and the sketch auto-dispenses; the ESP32 POSTs esp32/dispense/:id with deviceId and dispensed: true so the server marks the dose dispensed; and the patient's History screen reads the tracking rows for that patient.

## Implementation

### Roles in the App

At registration a user picks a role, and patient is the default. The tab layout shows the Patients and Prescribe tabs only when the signed-in user is a doctor, so a patient sees Home, History and Account and a doctor sees five tabs. There are also login, register and `patient/[id]` screens, routed with expo-router[4].

### An Offline Queue

A phone isn't always online, and a doctor adding a patient in a clinic corridor shouldn't lose the entry. The app keeps a list of pending operations in AsyncStorage under `offlineOperationsQueue` and replays them in order once the connection test passes. Condensed from the source, with the try/catch, the connectivity test, logging and the empty-queue return trimmed:

```ts
const queue = await AsyncStorage.getItem('offlineOperationsQueue');
if (!queue) return true;
const operations = JSON.parse(queue);
for (const op of operations) {
  switch (op.operation) {
    case 'addPatient':              await processAddPatient(op.data); break;
    case 'updateMedicationStatus':  await processUpdateMedicationStatus(op.data); break;
    case 'prescribeMedication':     await processPrescribeMedication(op.data); break;
  }
}
await AsyncStorage.setItem('offlineOperationsQueue', JSON.stringify([]));
```

The queue replays in order and is cleared once the loop finishes.

### The Prescribe Handler

Prescribing is the largest handler in the server, about 230 lines. It has to create a `Medication` and generate the scheduled tracking rows from the frequency enum (`daily`, `twice_daily`, `three_times_daily`, `weekly`, `as_needed`). This is the handler that turns a prescription into rows. The other medication routes are short: `GET /today/:userId`, `GET /history/:userId` and `PUT /status/:id`.

### The ESP32 Loop

The ESP32 sketch connects to WiFi with retries, pings the server, then polls every 30 seconds (`CHECK_INTERVAL = 30000`). It parses JSON with ArduinoJson[5] on the Arduino core for the ESP32[6], uses a 10 second HTTP timeout, and shares one request helper between calls. Three LEDs show state, with pin 2 for connected, pin 4 for dispensing and pin 5 for an error. When a dose is due, the dispensing LED blinks and the sketch dispenses on its own.

```cpp
void handleMedicationDue() {
  static unsigned long lastBlinkTime = 0;
  static boolean blinkState = false;

  if (millis() - lastBlinkTime > 500) {
    lastBlinkTime = millis();
    blinkState = !blinkState;
    digitalWrite(LED_DISPENSING, blinkState);
  }

  static int blinkCount = 0;
  if (blinkState && ++blinkCount > 10) {  // After 5 seconds (10 blinks)
    dispenseMedication();
    blinkCount = 0;
  }
}
```

`dispenseMedication` holds for a second, reports the dispense to the server, and clears its state. The sketch also has a `remoteDispense(trackingId)` function for a dispense the server triggers.

### The Servo Class on the Pi

The Raspberry Pi scripts use RPi.GPIO[7]. A `ServoControl` class starts 50 Hz PWM on GPIO 2 (physical pin 3) at the neutral duty and gives three ways to move, `set_direction`, `set_speed` and `gradual_move`. Condensed from the source, with comments changed or trimmed and the else branch and the rest of `set_speed` cut:

```python
MIN_DUTY = 2.5   # fully counterclockwise
MID_DUTY = 7.5   # neutral
MAX_DUTY = 12.5  # fully clockwise

def set_speed(self, speed_percent):
    if self.current_duty < MID_DUTY:      # counterclockwise
        range_width = MID_DUTY - MIN_DUTY
        duty = MID_DUTY - (range_width * speed_percent / 100)
    elif self.current_duty > MID_DUTY:    # clockwise
        range_width = MAX_DUTY - MID_DUTY
        duty = MID_DUTY + (range_width * speed_percent / 100)
```

Speed is a percentage of the distance from neutral to the end of the range, in whichever direction the servo is already turning. Around the class sit a few small scripts for slow counterclockwise motion and a command-line menu for driving the servo by hand.

### The Buzzer

`buzzer.py` drives a piezo buzzer on GPIO 18 through PWM, starting at 1 kHz. It has a dictionary from note names to frequencies and a `play_note(note, duration)` function, and it plays Für Elise.

## Problems

### 1. Two Systems Holding One Identity

Firebase knows who a user is. Postgres knows their role and their patients. **The two have to agree, and most of the auth code exists to keep them in step.** The auth routes handle registration, syncing a Firebase user into Postgres (`sync-firebase-user`), checking whether a user exists, looking one up by email, updating a role, and making sure a doctor record exists (`ensure-doctor-record`). **Splitting it this way let Firebase own credentials while every prescription could still join against a real doctor and patient row.**

### 2. Figuring Out What the Hacked Servo Does

The duty constants follow the angle convention of a normal servo, but on the hacked SG90 they set speed. **Getting a modified servo to move slowly and predictably took a few rounds.** We started with sample motor code, tuned the speed, and then added a slow-movement script. `gradual_move` smooths transitions and controls effective speed, with 10 steps by default and a 50 ms delay between them. **The class ended up shaped around what the hacked servo actually does rather than what the datasheet angles say.**

## Results

### Where the Lines Went

[Chart] Where the Lines Went
wc -l over git-tracked .ts, .tsx, .js, .py and .ino files, excluding lockfiles, assets, snapshots and JSON data
Unit: Lines of code
- Mobile app: 5,954 (pillifyai/, Expo and React Native, TypeScript and JavaScript)
- Server: 3,364 (pillifyai-server/, Express, Sequelize and Postgres, JavaScript)
- Pi scripts: 383 (hardware/, Python: servo control, examples, slow-move script and buzzer)
- ESP32 sketch: 307 (pillifyai-hardware/, one Arduino sketch)
The hardware is small next to the software around it. Line counts say nothing about effort or quality, and they are the only size measurement the project has.

**Most of the build went into software around a small amount of hardware code.** The app is the largest part at 5,954 lines and the server is 3,364. The hardware is small: 383 lines of Pi scripts and a 307-line ESP32 sketch. The server has 10 handlers in `auth.js`, 6 in `medications.js` and 4 in `patients.js`, plus the device routes.

### Servo Pulse Widths

[Chart] Three Duty Cycles, Three Pulse Widths
Duty constants from servo_control.py at a 50 Hz PWM period of 20 ms. Arithmetic on the constants, not an oscilloscope reading
Unit: Pulse width in µs
- MIN_DUTY 2.5%: 500 (0.50 ms, the counterclockwise command in the code)
- MID_DUTY 7.5%: 1,500 (1.50 ms, the neutral or stop command)
- MAX_DUTY 12.5%: 2,500 (2.50 ms, the clockwise command in the code)
A continuous-rotation servo reads the pulse width as a speed and a direction around the 1.5 ms neutral, which is why the code varies the duty offset from neutral. That reading of the hack is my inference from the code, not something the repo states.

**The three duty constants correspond to pulse widths of 0.50 ms, 1.50 ms and 2.50 ms at a 20 ms period.** Hobby servos commonly use 1.50 ms as the neutral point and about 1 to 2 ms as the typical range[2], so the 0.50 and 2.50 ms endpoints are the extreme ends of what the code asks for. For a continuous-rotation servo, neutral means stopped.

### The Melody

**The Für Elise note list has 40 entries, 35 notes and 5 rests, and runs 9.2 seconds of nominal duration**, or 13.2 seconds counting the 0.1 second default gap after each entry.

## What I Would Change

### Dosing Logic Beyond a Frequency Enum

The schedule comes from five frequencies, from `daily` to `as_needed`. A real product would need actual dosing logic and interaction checking before a dose row is ever written. That was out of scope for one night, and it's the first thing I'd add.

### Add Push

The sketch already has a `remoteDispense` function. A path where the server triggers the dispense would remove the up-to-30-second wait, though it needs a connection the home network allows.

## References

1. Roy T. Fielding. *Architectural Styles and the Design of Network-based Software Architectures*. Doctoral dissertation, University of California, Irvine, 2000. [link](https://ics.uci.edu/~fielding/pubs/dissertation/top.htm)
2. Wikipedia. *Servo control*. [link](https://en.wikipedia.org/wiki/Servo_control)
3. Google. *Verify ID tokens*. Firebase Authentication documentation. [link](https://firebase.google.com/docs/auth/admin/verify-id-tokens)
4. Expo. *Introduction to Expo Router*. Expo documentation. [link](https://docs.expo.dev/router/introduction/)
5. Benoit Blanchon. *ArduinoJson*. Library documentation. [link](https://arduinojson.org/)
6. Espressif Systems. *Arduino core for the ESP32*. GitHub repository. [link](https://github.com/espressif/arduino-esp32)
7. Ben Croston. *RPi.GPIO*. Python Package Index. [link](https://pypi.org/project/RPi.GPIO/)


================================================================================

# Beating Buy-and-Hold by Holding Less Stock

Date: 2025-02-26
URL: https://projects.farhansadeek.com/posts/quantifyai-market-regimes/
Tags: hackathon, finance, backtesting, gradient-boosting
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/quantifyai-market-regimes
Summary: A five-day Quantathon team pipeline that labels Bear, Bull and Static markets and moves money between stocks and bonds reached a Sharpe of 1.10 against 0.58 for buy-and-hold over 2019 to 2022, mostly by cutting volatility from 22.9% to 10.7% rather than by forecasting.

The 2025 Ohio State Quantathon was a five-day team competition with one question: each day, how much money should sit in the S&P 500 and how much in bonds, if the goal is to beat buy-and-hold? Our answer was QuantifyAI. It labels every day Bear, Bull or Static with a 20% drawdown rule, predicts that label 63 trading days ahead with gradient boosting, flags unusual days with an anomaly ensemble, and backtests about ten allocation strategies on top of both.

Over 2019 to 2022 the best strategy returned **56.68%** with a Sharpe ratio of **1.10** and a maximum drawdown of **-10.68%**, against 52.97%, 0.58 and -33.92% for buy-and-hold. The more interesting part is where that Sharpe comes from. Annualised return barely moves, 11.88% against 11.21%, while annualised volatility falls from 22.9% to 10.7%. **The strategy wins by holding less equity through a window that contains the 2020 crash. It is a risk-control result more than a forecasting one.**

## Why It Matters

The competition gave us a daily S&P 500 series, a bond rate and two "market-based probability" series. The brief asked us to predict market regimes, comparing Markov chains, gradient boosting and LSTMs, and to beat buy-and-hold. This post covers the gradient boosting path.

Regimes are a useful target because allocation needs only one decision a day, how much equity to hold. A coarse label like Bear, Bull or Static maps straight onto that decision. Our first attempt, predicting the price level itself, went nowhere (see [Problems](#problems)).

What makes this worth writing up is the gap between the Sharpe and the forecast. A strategy that cuts equity at the right moments looks brilliant on Sharpe whether or not its model can see the future, because Sharpe rewards lower volatility as much as higher return. If you want to know what you built, you have to pull the two apart, and that is what the experiments below do.

## Technical Details

### Regimes From a Drawdown Rule

The label is backward-looking. For each day, take the rolling 252-day maximum and minimum of the S&P 500. Let $P_t$ be the price. Then

$$
\text{DD}_t = \frac{P_t}{\max_{s \in (t-252,\,t]} P_s} - 1, \qquad
\text{Rise}_t = \frac{P_t}{\min_{s \in (t-252,\,t]} P_s} - 1.
$$

A day is Bear if $\text{DD}_t \le -0.2$, Bull if it is not Bear and $\text{Rise}_t \ge 0.2$, and Static otherwise. This follows the familiar 20% convention for bull and bear markets[1], with the 252-day window standing in for peaks and troughs. Because the window is trailing, the label does not look ahead. It is also a lagging description, because a Bear label appears only after a 20% fall has happened.

### Predicting a Regime 63 Days Out

The classifier predicts the label 63 trading days ahead, roughly one quarter. The model is scikit-learn's gradient boosting[2], which fits trees stage by stage to the gradient of the loss. Features are about 24 engineered columns from the two probability series, S&P returns and volatility, and bond-rate features.

Financial time series need evaluation that respects time[3], so the model trains on 2007 to 2018 and is scored on 2019 to 2022. A slow-moving label also needs a persistence baseline. If the regime rarely changes, "it stays the same" is hard to beat.

### Sharpe and What It Rewards

The Sharpe ratio is mean return over standard deviation, annualised by $\sqrt{252}$[4]. A strategy that de-risks shrinks the denominator, so a Sharpe gain can be a volatility story rather than a return story, and that is what happens here. Sharpe also inflates when many variants are tried on one path and the best is reported[5][6], which applies to any strategy picked as the best of about ten.

### Anomaly Detection as an Ensemble

The anomaly module runs IsolationForest[7], DBSCAN[8] and a z-score, normalises each score, and takes the mean. A percentile of that mean, set by a `contamination` fraction, decides which days are flagged. The fraction fixes how many days get flagged: at `contamination=0.03`, 31 of the 1,008 test days.

### Architecture

[Diagram] QuantifyAI pipeline: a competition workbook is merged and forward-filled, a 252-day drawdown rule labels Bear, Bull and Static, a gradient boosting classifier predicts the state 63 days ahead, an anomaly ensemble runs beside it, and both feed a backtester of about ten strategies with prior-day weights that reports return, Sharpe and maximum drawdown against buy-and-hold.

Two paths do the work. The label feeds the classifier, and the merged features also feed the anomaly ensemble. The backtester combines both, turns them into daily stock and bond weights, and compares each strategy to buy-and-hold. Every strategy applies the previous day's allocation to today's return, so the portfolio timing is causal. Rebalancing is daily, there is no leverage, and bonds earn the supplied rate.

## Implementation

The code excerpts below are from the team repository, [SadeekFarhan21/QuantifyAI](https://github.com/SadeekFarhan21/QuantifyAI).

### The Regime Label

This is the core of `classify_markets`.

```python
df['Rolling_Peak'] = df['SP500'].rolling(window=window, min_periods=1).max()
df['Drawdown'] = (df['SP500'] / df['Rolling_Peak']) - 1
df['Is_Bear'] = df['Drawdown'] <= -threshold
df['Rolling_Trough'] = df['SP500'].rolling(window=window, min_periods=1).min()
df['Increase_From_Trough'] = (df['SP500'] / df['Rolling_Trough']) - 1
df['Is_Bull'] = ~df['Is_Bear'] & (df['Increase_From_Trough'] >= threshold)
```

On the merged 4,025 days the rule gives 8.2% Bear, 43.0% Bull and 48.8% Static. Over 2019 to 2022 the counts are 72 Bear, 599 Bull and 337 Static.

### The Classifier

The classifier's target is the state shifted forward by 63 rows.

```python
df['Future_Market'] = df[target_col].shift(-forward_periods)
```

Grid search over `n_estimators` 100 or 200, learning rate 0.05 or 0.1 and depth 3 or 4 picked learning rate 0.1, depth 3 and 200 estimators, with subsample 0.8 and min_samples_split 5 fixed.

### The Anomaly Ensemble

The threshold is a percentile of the ensemble score.

```python
df['ensemble_score'] = normalized_scores.mean(axis=1)
ensemble_threshold = np.percentile(df['ensemble_score'].dropna(),
                                   self.contamination * 100)
df['ensemble_anomaly'] = 1
df.loc[df['ensemble_score'] <= ensemble_threshold, 'ensemble_anomaly'] = -1
```

Over the test period it flags 31 days: `Anomaly distribution: {1: 977, -1: 31}`.

### Causal Portfolio Timing

Every strategy uses the same loop. Yesterday's weights earn today's returns, so no strategy trades on a price it hasn't seen yet.

```python
sp500_contrib = df['SP500_Allocation'].iloc[i-1] * df['SP500_Return'].iloc[i]
bond_contrib = df['Bond_Allocation'].iloc[i-1] * df['Daily_Bond_Return'].iloc[i]
```

### The Headline Strategy

The combined anomaly and regime strategy holds 1.0 equity in Bull, 0.5 in Static and 0.0 in Bear. A flagged day sends it to 0% equity, and it recovers linearly over 10 days. Its other parameters:

```python
recovery_period = params.get('anomaly_recovery_period', 10)
min_allocation = params.get('min_allocation', 0.0)
bearish_reduction = params.get('bearish_reduction', 0.5)
```

### Who Built What

Jalen Francis built the modular pipeline, including the backtester. My parts were the data merging, an early modelling notebook, the slides, most of the paper's text and TikZ figures, and the feature-importance plotting and accuracy logging. Aditya Bhati wrote an early abstract draft, and Jayson Clark is a co-author on the paper and slides.

## Problems

### 1. A Price Regression That Went Nowhere

My first attempt, before the pipeline existed, was a linear regression from the bond rate and the two probability series to the S&P 500 level, trained to 2018-12-31 and tested from 2019. On the test years it had a mean squared error of 6,078,269.9 and an R-squared of **-7.147**, far worse than predicting the mean. **Those inputs cannot predict a price level, and that dead end is why the project switched to classifying regimes.** A regime is a coarser target, and it is the one an allocation rule needs anyway.

### 2. Two Sheets on Different Calendars

The price sheet has 4,527 daily rows. The probability sheet has 736 rows on irregular dates. They merge to 4,025 days from 2007-01-08 to 2022-12-30, which means most days need a probability value carried from somewhere. **The loader forward-fills, so every day sees the most recent probability that already existed.**

### 3. Telling Risk Control From Forecasting

A Sharpe of 1.10 against 0.58 looks like a model that sees the future. But the same number comes out of any rule that happens to hold less equity during a crash. **Sharpe alone can't tell you whether a de-risking strategy has skill.** The way through was to change one input at a time on the same test frame, add a random-flag placebo and simple constant mixes, and put a persistence baseline next to the classifier. That is the experiment below.

## Experiments

The experiments use the default pipeline with the enhanced backtester, trained on 2,769 days up to 2018-12-31 and tested on the 1,008 days of 2019 to 2022. Each ablation changes one input at a time on that same test frame:

- Anomalies switched off.
- A causal detector, flagging days where the return is more than 3 standard deviations from its trailing 252-day mean, which flags 30 days.
- A placebo: 31 random days flagged as anomalies, 200 runs.
- Predictions replaced by always Bull, with and without the real anomalies.
- Predictions replaced by the current true state.
- Constant 50/50 and 60/40 stock and bond mixes.
- Out-of-sample accuracy of the model's `Predicted_Market` and a persistence baseline, on 2019 to 2022 only.

The test is a single path with one crash in it, and the placebo is the only uncertainty estimate I have.

## Results

### Against Buy-and-Hold

[Chart] Sharpe Ratio by Strategy, 2019-2022
My rerun of the default pipeline, one row per strategy
Unit: Sharpe ratio, no risk-free rate subtracted
- Combined anomaly-regime: 1.1 (Total return 56.68%, max drawdown -10.68%)
- Combined: 1.01 (Total return 41.91%, max drawdown -11.70%)
- Dynamic allocation: 0.91 (Total return 51.65%, max drawdown -13.81%)
- Volatility targeting: 0.87 (Total return 24.81%, max drawdown -7.59%)
- Prediction: 0.86 (Total return 42.77%, max drawdown -13.89%)
- Tactical risk managed: 0.81 (Total return 21.26%, max drawdown -8.30%)
- Market beating: 0.76 (Total return 49.91%, max drawdown -22.27%)
- Buy and hold: 0.58 (Total return 52.97%, max drawdown -33.92%)
- Regime adaptive: 0.43 (Total return 26.63%, max drawdown -27.01%)
One path, 1,008 trading days, containing the 2020 crash. Most strategies give up total return to cut drawdown. Sharpe here is mean over standard deviation times the square root of 252.

Buy-and-hold returned 52.97% in total, 11.21% a year, with a Sharpe of 0.58, a drawdown of -33.92% and a win rate of 54.12%. **The combined anomaly-regime strategy reaches a Sharpe of 1.10**, with 56.68% total, a drawdown of -10.68% and a 59.03% win rate. The prediction strategy returned 42.77% and the dynamic strategy 51.65%.

Against simple baselines on the same window, a constant 50/50 mix returned 29.79% with Sharpe 0.63 and drawdown -18.03%, and 60/40 returned 34.70%, 0.61 and -21.38%. **Only the headline strategy beats buy-and-hold on total return, and buy-and-hold has the deepest drawdown of any of them.** The window contains the 2020 crash, the fast recovery and the 2022 fall, which is exactly the kind of period where cutting equity pays.

### A Sticky Label Is Hard to Forecast

[Chart] Three Ways to Score the Same Model
Predicting the market state 63 trading days ahead
- Committed confusion matrix (shuffled split): 95.6 (501 of 524 samples on the diagonal, read from results/confusion_matrix.png)
- Logged accuracy, my rerun (shuffled split): 94.08 (Model accuracy: 0.9408)
- Persistence baseline, 2019-2022: 73.4 (Predict that the state stays the same for 63 days)
- Model vs same-day state, 2019-2022: 60.6 (1,008 test days, trained only to 2018-12-31)
- Model vs state 63 days ahead, 2019-2022: 56.8 (The target the model was trained for)
The shuffled split puts near-duplicate neighbouring days on both sides. On a chronological test the model is below the do-nothing baseline.

On the 1,008 test days, with the model trained only to 2018-12-31, its `Predicted_Market` matches the same-day state 60.6% of the time and the state 63 days ahead 56.8%. Predicting that the state simply stays the same for 63 days scores 73.4%. **Regimes are sticky, and the edge in the backtest doesn't come from seeing a quarter ahead.** The model also leans Static: its predicted mix over the test period was 664 Static, 264 Bull and 80 Bear days, against true counts of 599 Bull, 337 Static and 72 Bear.

### What the Ablations Say

[Chart] Ablating the Headline Strategy
Combined anomaly-regime strategy with one input changed, 2019-2022
Unit: Sharpe ratio
- Causal rolling-z anomalies: 1.16 (30 flagged days. Total return 58.63%, max drawdown -10.58%)
- As published: 1.1 (Total return 56.68%, max drawdown -10.68%)
- Placebo, 31 random anomaly days (mean of 200): 0.93 (5th to 95th percentile 0.65 to 1.22. 14.5% of runs match or beat 1.10)
- Predicted state set to current state: 0.88 (Total return 50.35%, max drawdown -19.55%)
- Always Bull, real anomalies: 0.79 (Total return 56.02%, max drawdown -22.56%)
- Anomalies switched off: 0.96 (Total return 50.01%, max drawdown -12.31%)
- Always Bull, no anomalies: 0.6 (Total return 47.27%, max drawdown -29.01%)
Each row changes one input to the combined strategy and reruns the backtest. The placebo row is a mean, so its range matters more than its height. One path, so none of these differences are tested for significance.

- **A simple trailing detector does at least as well as the ensemble.** Flagging 3-sigma days against a trailing 252-day mean gives 58.63% total return, Sharpe 1.16, drawdown -10.58%.
- **The detector's timing is not clearly skill.** Flagging 31 random days gives a mean total return of 43.01% (5th to 95th percentile 28.36 to 58.69) and a mean Sharpe of 0.93 (0.65 to 1.22). 14.5% of placebo runs match or beat the Sharpe of 1.10, and 7% beat the total return.
- **The predictor and anomaly flags together are what cut drawdown.** Always Bull with the real anomalies has a drawdown of -22.56%, and always Bull with none has -29.01%, against -10.68% for the headline strategy. But replacing predictions with the current true state gives 50.35%, Sharpe 0.88 and drawdown -19.55%, so the evidence that the machine-learned prediction adds value beyond simple rules is weak.

**De-risking paid off in a period that contained a crash, and I can't separate the model's skill from the schedule of when equity was cut.** That is the claim the numbers support.

## What I Would Change

### Always Report the Persistence Baseline

For a label that changes rarely, "stays the same" is a strong baseline. A model that loses to it has learned less than it appears to.

### Test on More Than One Path

A single 2019 to 2022 path with one crash cannot support a claim of skill. Next time I would keep a second window that is never used for choosing a strategy, score the final number there, and count how many variants were tried[5][6].

### Run the Placebo First

The random-flag placebo is what showed the detector's timing is not clearly skill. I would run it before writing any claim about detector timing.

## References

1. Adrian Pagan and Kirill Sossounov. *A Simple Framework for Analysing Bull and Bear Markets*. Journal of Applied Econometrics, 2003.
2. Jerome Friedman. *Greedy Function Approximation: A Gradient Boosting Machine*. Annals of Statistics, 2001.
3. Marcos Lopez de Prado. *Advances in Financial Machine Learning*. Wiley, 2018.
4. William Sharpe. *The Sharpe Ratio*. Journal of Portfolio Management, 1994.
5. David Bailey and Marcos Lopez de Prado. *The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality*. Journal of Portfolio Management, 2014.
6. David Bailey, Jonathan Borwein, Marcos Lopez de Prado and Qiji Jim Zhu. *The Probability of Backtest Overfitting*. Journal of Computational Finance, 2017.
7. Fei Tony Liu, Kai Ming Ting and Zhi-Hua Zhou. *Isolation Forest*. IEEE International Conference on Data Mining, 2008.
8. Martin Ester, Hans-Peter Kriegel, Jorg Sander and Xiaowei Xu. *A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise*. KDD, 1996.


================================================================================

# An ASL and Emotion Overlay for Any Video Call

Date: 2025-02-22
URL: https://projects.farhansadeek.com/posts/signifyai/
Tags: computer-vision, tensorflow, sign-language
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/signifyai
Summary: SignifyAI is a three-day team build that labels ASL letters, hand gestures and facial emotion over any window, including a video call, with three small CNNs that score 99.8% on gestures, 68.2% on FER2013 emotion and about 98% on ASL.

Most sign-language demos own the webcam. We wanted something that helps deaf and hearing people talk on the video calls they already use, so SignifyAI does the opposite. It is a transparent, always-on-top overlay that sits over whatever is already on screen and labels the hands and faces in it with an ASL letter, a hand gesture and an emotion. Jalen Francis, Jayson Clark and I built it at Ohio State over three days in February 2025, in Python with TensorFlow and PyQt5, on public Kaggle datasets: an ASL alphabet set, LeapGestRecog hand gestures[1] and FER2013 emotion[2].

The design is detect, crop, classify. Every 100 ms the app grabs the screen with `mss`, finds faces with an OpenCV SSD and hands with MediaPipe, strips the background off each hand, and passes small grayscale crops to three small Keras CNNs. Detection removes most of the image, which is why the classifiers can be small. **The gesture model gets 2,795 of 2,800 test images right (99.8%), the ASL model about 98%, and the emotion model 68.2% on FER2013.**

## Why It Matters

The goal was to help deaf and hard-of-hearing people and hearing people talk on ordinary video calls. A tool that needs its own camera feed doesn't fit that. We wanted a small layer that floats above whatever app you already use and labels what it sees.

That led to two design choices that shape everything else. Capture is a screen grab, so the tool needs no camera access and works on any conferencing app. And the app runs several classifiers on the same frame every 100 ms, so each one has to be small.

The scope was three recognizers. Two of them share a hand crop, with ASL finger-spelling classifying it into a letter and a hand-gesture model classifying the same crop into one of 10 gestures, while the third, an emotion model, classifies a face crop into one of the seven FER2013 labels.

The three of us built it as one team and share equal credit on the paper. I wrote the research paper and made its confusion-matrix and loss-curve figures, along with the early FER2013 work whose data pipeline and training recipe the emotion model kept. Jalen wrote the model training, and Jayson wrote the desktop app.

## Technical Details

### Small CNNs on Cropped Inputs

None of the models is exotic. Each is a convolutional classifier that takes a small grayscale crop and outputs a softmax over classes, trained with Adam[3]. The emotion network is the deepest, with four convolution blocks with batch normalization[4] and dropout, ending in global average pooling. The ASL network is three convolutions and a dense layer with dropout. The gesture network is the plainest, one convolution and one dense layer with no dropout or batch normalization.

### Detect, Crop, Classify

The app never classifies a whole screenshot. A detector finds a face or a hand, the app crops around it, and a classifier sees only the crop. Faces come from OpenCV's DNN module running a pretrained Caffe single-shot detector[5], keeping detections with confidence above 0.5. Hands come from MediaPipe Hands[6], which returns 21 landmarks per hand. The app takes the min and max of the landmark coordinates as a bounding box and adds a 20 pixel margin. This is why the classifiers can be small. Detection has already removed most of the image.

### The Overlay

The app is a set of threads around a Qt window. Two worker threads each grab a region of the screen with `mss`, run their models, and emit a list of boxes and labels to the overlay widget, which paints them.

[Diagram] SignifyAI screen overlay: two threads grab a screen region with mss every 100 ms, the face thread runs an OpenCV DNN face detector and a 48 by 48 emotion CNN, the hand thread runs MediaPipe Hands, a 20 pixel crop, rembg and a 128 by 128 grayscale image into a 10-class gesture CNN and a 29-class ASL CNN, and a frameless, always-on-top, mouse-transparent Qt overlay shows the results.

The overlay is a frameless, always-on-top, mouse-transparent window, so clicks pass through it to the application underneath. A separate draggable toolbar toggles the face and hand threads and snaps to screen edges. The main app runs hand gestures, ASL and emotion together, and a lighter version does face and emotion only.

### Where the Parameters Live

The three networks differ a lot in shape, and the parameter counts are not what the layer counts suggest.

[Diagram] Layer stacks of the three SignifyAI CNNs: the gesture CNN has one 16-filter conv and a Dense 32 holding 2,097,184 of its 2,097,674 parameters, the ASL CNN has three conv stages and a Dense 128 holding 3,211,392 of its 3,307,805, and the emotion CNN has four batch-normalized conv blocks and global average pooling for 914,151 parameters.

The gesture network has a single 16-filter convolution and yet holds 2,097,674 parameters, because the `Flatten` layer hands a 64 by 64 by 16 feature map, 65,536 values, to a `Dense(32)`, and that one layer holds 2,097,184 of them. The emotion network is by far the deepest and has the fewest parameters, 914,151, because global average pooling replaces the flatten.

## Implementation

### Screen Capture and the Hand Thread

The hand thread is Jayson's code. It grabs the screen, downscales it so hands look larger to the detector, runs MediaPipe, and then processes each hand in turn.

```python
with mss.mss() as sct:
    while self.running:
        sct_img = sct.grab(self.monitor)
        frame = np.array(sct_img)
        ...
        results = self.hands.process(rgb_frame)
        ...
        self.handsDetected.emit(hand_boxes)
        self.msleep(100)
```

The 100 ms sleep caps the loop at roughly ten iterations a second before the cost of the models themselves.

### One Crop, Two Classifiers

Each hand crop is resized to 256 by 256 and goes through `rembg`[7], is composited over black using the alpha mask, resized back to the original crop size, converted to grayscale, resized to 128 by 128 and scaled to [0, 1]. Then the same array goes to both the gesture CNN and the ASL CNN.

```python
gesture_preds = self.gesture_model.predict(input_img)
...
asl_preds = self.asl_model.predict(input_img)
...
combined_label = f"Gesture: {gesture_name} | ASL: {asl_name}"
```

Every hand gets both a gesture and a letter, and the overlay shows the two side by side.

### The Face Thread

The face thread runs the Caffe SSD on a frame scaled by 0.5, keeps detections above 0.5 confidence, crops the face, converts it to 48 by 48 grayscale and runs the emotion CNN. It is the simpler of the two threads.

### Training the Models

The emotion network's training keeps the split logic, augmentation settings, Adam rate and early stopping from my early FER2013 model and replaces its architecture (Flatten and three Dense blocks) with Jalen's convolutional one. It uses the predefined FER2013 splits: 28,709 training images, 3,589 in `PublicTest` for validation and 3,589 in `PrivateTest` for the test. Training uses augmentation (rotation 10 degrees, shifts and zoom of 0.1, horizontal flip), Adam at 0.001, batch 64 for up to 100 epochs, early stopping with patience 10 on validation loss with the best weights restored, and a learning-rate reduction of 0.5 after 5 stalled epochs.

The gesture script holds out 30% of the images for validation and 20% of the rest for test, so 56% train, 30% validation and 14% test, at batch 32. The ASL script splits the alphabet set 80/20 with `image_dataset_from_directory` and seed 123 and trains for 30 epochs at batch 32. The ASL model was first trained on 36 classes, digits 0 to 9 and letters A to Z, and the final version covers 29, A to Z plus `del`, `nothing` and `space`.

## Problems

### 1. Hands on a Screen Are Small

A webcam demo gets a hand that fills a good part of the frame. A screen grab of a video call gets a hand inside a window inside a monitor. **The fix was to downscale the frame before detection so hands look larger to MediaPipe, and to run the detector permissively.** MediaPipe is configured for up to 10 hands with detection and tracking confidence of 0.3, a low threshold that trades false positives for recall.

### 2. Training Images Don't Look Like Screen Crops

LeapGestRecog frames are clean images of a hand on a plain background. A hand cropped from a real screen sits on a messy one. **That is why the hand thread removes the background before classifying.** It runs `rembg` on each crop, composites what is left over black, and converts it to grayscale, so the classifier sees the hand and not the room behind it.

### 3. Several Models Inside One 100 ms Loop

Each hand goes through `rembg` and two `predict` calls, and each face through the SSD and the emotion CNN, all on a loop that wakes every 100 ms. **Three choices keep the per-frame work small: faces and hands run on separate threads, detection shrinks every input to a crop, and the classifiers stay small.** Global average pooling is what keeps the deepest of them, the emotion network, under a million parameters.

### 4. An Overlay That Doesn't Get in the Way

A layer that sits over a video call is useless if it blocks the call. **The window is frameless, always on top and mouse-transparent, so every click lands on the app underneath.** The controls live in a separate draggable toolbar that snaps to screen edges and turns the face and hand threads on and off.

## Results

### Parameter Counts

[Chart] The Smallest Conv Stack Has the Second Most Parameters
Trainable parameters of the three CNNs, rebuilt from the layer stacks in model_builders
Unit: Parameters
- ASL, 29 classes: 3,307,805 (Three conv layers, Flatten, Dense 128. 36-class version has 3,308,708)
- Gesture, 10 classes: 2,097,674 (One Conv 16, MaxPool, Flatten, Dense 32)
- Emotion, 7 classes: 912,103 (914,151 total, 912,103 trainable)
The gesture network has one 16 filter conv layer, yet 2.1 million parameters, because Flatten hands a 64 by 64 by 16 map (65,536 values) to Dense 32. The deepest network, the emotion CNN, is the smallest because global average pooling removes that layer.

The three networks have 2,097,674 (gesture), 3,307,805 (ASL, 29 classes) and 914,151 (emotion, of which 912,103 are trainable) parameters. The 36-class ASL version has 3,308,708, a difference that sits entirely in the output layer. **The network with one convolution has more than twice the parameters of the one with four convolution blocks.**

### Gesture Accuracy

The gesture confusion matrix has 280 test samples in each of 10 classes, 2,800 in total. The diagonal is 279, 279, 280, 280, 280, 277, 280, 280, 280 and 280, which sums to 2,795. **The model got 5 wrong, an accuracy of 99.82%.** The training curve in the same figure has validation loss near 0.14 at the first epoch and close to zero by epoch 4. **LeapGestRecog is an easy dataset to fit**, with clean hands on plain backgrounds, and the model learns it in a few epochs.

### Emotion Accuracy

The emotion matrix rows sum to 3,589, the size of the FER2013 `PrivateTest` split. The diagonal is 307, 25, 215, 775, 337, 332 and 457, which sums to 2,448. **That is 68.21%, typical of a small CNN on FER2013.** On the training curve, training loss falls from about 2.1 to about 0.86 and validation loss from about 1.98 to about 0.92, stopping around epoch 62.

[Chart] Fear Is Right Two Times in Five, Happy Nearly Nine in Ten
Per-class recall of the emotion CNN on the 3,589 FER2013 PrivateTest images
- Happy: 88.2 (775 of 879)
- Surprise: 79.8 (332 of 416)
- Neutral: 73 (457 of 626)
- Angry: 62.5 (307 of 491)
- Sad: 56.7 (337 of 594. 126 were predicted Neutral)
- Disgust: 45.5 (25 of 55. Only 55 test samples)
- Fear: 40.7 (215 of 528. Confused with Sad 104, Angry 72, Neutral 61, Surprise 58)
Overall this matrix gives 2,448 of 3,589, or 68.2 percent. Recall is the diagonal cell over its row sum, computed from counts I read off the committed image, not from a rerun.

Happy is the best class, at 775 of 879. Fear is the worst of the large classes, at 215 of 528, and is confused mostly with Sad (104), Angry (72), Neutral (61) and Surprise (58). Disgust has only 55 test samples, of which 25 are right, so its recall is a noisy estimate. Sad and Neutral are also confused, with 126 of 594 Sad images predicted Neutral. **An overall accuracy of 68% hides a model that gets Fear right only about 41% of the time** and mistakes it for Sad, Angry, Neutral or Surprise most of the rest.

### ASL Accuracy

The ASL confusion matrix is from our original 36-class run, with roughly 7 to 20 samples per class. The diagonal sums to 493 and the off-diagonal cells to about 10, out of about 503 samples, so **roughly 98.0%.** The errors are single counts apart from 0 predicted as O (2 times) and V predicted as 2 (2 times), plus single errors such as A and S, and T and U. **Those are the pairs that look alike in a still image.**

## What I Would Change

### Study the Face Properly

FER2013 labels are a coarse proxy for what a signer's face is doing. In signed languages the face carries grammar as well as emotion. I would design an experiment around that directly instead of reusing seven emotion labels.

## References

1. Marin, G., Dominio, F., Zanuttigh, P. *Hand Gesture Recognition with Leap Motion and Kinect Devices*. IEEE International Conference on Image Processing (ICIP), 2014. LeapGestRecog is distributed on Kaggle as [gti-upm/leapgestrecog](https://www.kaggle.com/datasets/gti-upm/leapgestrecog).
2. Goodfellow, I. J., et al. *Challenges in Representation Learning: A Report on Three Machine Learning Contests*. Neural Networks 64, 2015. [arXiv:1307.0414](https://arxiv.org/abs/1307.0414)
3. Kingma, D. P., Ba, J. *Adam: A Method for Stochastic Optimization*. ICLR, 2015. [arXiv:1412.6980](https://arxiv.org/abs/1412.6980)
4. Ioffe, S., Szegedy, C. *Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift*. ICML, 2015. [arXiv:1502.03167](https://arxiv.org/abs/1502.03167)
5. Liu, W., et al. *SSD: Single Shot MultiBox Detector*. ECCV, 2016. [arXiv:1512.02325](https://arxiv.org/abs/1512.02325)
6. Zhang, F., et al. *MediaPipe Hands: On-device Real-time Hand Tracking*. 2020. [arXiv:2006.10214](https://arxiv.org/abs/2006.10214)
7. Gatis, D. *rembg: Rembg is a tool to remove images background*. Software, [github.com/danielgatis/rembg](https://github.com/danielgatis/rembg)


================================================================================

# Casual Riders Keep Divvy Bikes Twice as Long

Date: 2025-01-18
URL: https://projects.farhansadeek.com/posts/divvy-bike-analysis/
Tags: r, quarto, eda, data-analysis
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/divvy-bike-analysis
Summary: A Quarto and R exploratory analysis of Chicago's 2024 Divvy bike-share trips finds casual riders average 25.2 minutes a ride against 12.8 for members, with a 17:00 commuter peak and a September high.

Divvy is Chicago's bike share, and every trip it logs records when it started and ended, where, on what kind of bike, and whether the rider was a member or a casual user. In January 2025 I built a Quarto site in R that turns all of 2024, about 5.86 million trips, into charts that answer who rides, when and where. The clearest answer is that **casual riders ride about twice as long as members, 25.2 minutes a ride against 12.8**. Saturday is the busiest day, the hourly curve peaks at 17:00, the busiest month is September rather than July, and one station, Streeter Dr and Grand Ave, leads the rest by a wide margin.

Technically it is descriptive exploratory analysis[1], small and deliberately plain. Twelve monthly CSVs are stacked with the tidyverse[2], a few time columns are derived, and each question gets one group-and-summarise feeding one ggplot2[3] chart, plus two Leaflet[4] maps, all rendered to a static page. It has counts and means and no models, so it shows the shape of one year of trips without explaining it.

## Why It Matters

I wanted a portfolio-sized project that answers a plain question, who rides Divvy bikes, when and where, from a public dataset, and that I could publish as a static page with the code visible.

The Divvy trip files fit because one row is one trip, with a start and end time, start and end station, coordinates, a bike type and a rider type of `member` or `casual`. The split between members and casual riders is the interesting axis, because the two groups ride very differently.

The plan was one Quarto document[5] with the code folded under each chart, a fixed list of questions, and one chart per question.

- Who rides, split by member and casual.
- Which bike types they use.
- How long rides last, by rider type, weekday and season.
- How many rides happen by weekday, hour and month.
- Which stations start and end the most rides.

Divvy publishes the trip files under its data license[6]. The data also has clear gaps: no user ID, bike ID, age, gender, trip purpose, distance or weather. I listed those up front on the site, because they set the ceiling on what any chart here can claim. This was a solo project built over about ten days in January 2025, published at divvy.farhansadeek.com.

## Technical Details

### What Descriptive EDA Can Claim

Exploratory data analysis, in Tukey's sense[1], is looking at data to see what it shows before deciding what to test. Every chart in this project is a count or a mean over a grouping, so every claim it supports has the form "in this year of trips, group A has a larger count or mean than group B". It supports no claim about why, and no claim about other years.

### Binning Time

Two derived columns turn timestamps into groups. `time_of_day` buckets the start hour into Night (0 to 5), Morning (6 to 11), Afternoon (12 to 17) and Evening (18 to 23). `season` buckets the month into Winter (December, January, February), Spring (March to May), Summer (June to August) and Fall (September to November). Meteorological seasons are a convention, and the choice has a visible consequence, because September's peak lands in Fall.

### The Pipeline

The whole project is one pipeline inside one file. Twelve CSVs are bound into one data frame, a few columns are derived, and each chart is one `group_by`, one `summarise` and one plotting call. `quarto render` turns that into a static `index.html` with every chart embedded.

[Diagram] Pipeline of index.qmd. Twelve monthly Divvy CSVs, marked as not in the repository, are stacked by bind_rows with a month column, then given derived columns (hour, time_of_day, season, ride_length), then grouped and summarised. The result feeds 10 ggplot2 charts and 2 leaflet maps, which quarto render turns into index.html, published as a static Netlify site at divvy.farhansadeek.com. A code panel shows the difftime ride_length expression, and a red note says nothing filters negative or extreme ride lengths.

The rendered page carries ten charts and two maps.

## Implementation

### Reading and Combining

Each month is read with `read.csv` from a file named like `202401-divvy-tripdata.csv`, and the twelve frames are stacked with `bind_rows`, tagging each with its month number.

```r
df <- bind_rows(
    january |> mutate(month = 1),
    february |> mutate(month = 2),
    # ... through december |> mutate(month = 12)
)
```

### Derived Columns

The hour and the two time buckets are `case_when` calls on integer ranges.

```r
df <- df |>
    mutate(hour = hour(started_at)) |>
    mutate(time_of_day = case_when(
      hour %in% 0:5   ~ "Night",
      hour %in% 6:11  ~ "Morning",
      hour %in% 12:17 ~ "Afternoon",
      hour %in% 18:23 ~ "Evening"))
```

The season column has the same shape on `month`. Ride length is `as.numeric(difftime(ended_at, started_at, units = "mins"))`, the gap between end and start in minutes.

### One Grouping per Chart

Every chart follows the same recipe. The average ride length by rider type is a `group_by(member_casual)`, a `mean(ride_length, na.rm = TRUE)`, and a bar chart with the rounded value printed above each bar. Counts by weekday, hour and month are `n()` over the matching key.

The rider-type donut is built by hand from `ggforce::geom_arc_bar` with start and end angles computed from cumulative proportions, and the number in its hole is `round(nrow(df) / 1000000, 2)` followed by "M Total Rides". That is where the 5.86M comes from.

### Stations and Maps

The station bar charts lump station names to the top ten, drop missing names and the lumped "Other" group, and order the bars by count.

```r
# abridged: the ggplot call that follows is omitted
df |>
  mutate(start_station_name = fct_lump(start_station_name, 10)) |>
  count(start_station_id, start_station_name, name = "counts", sort = T) |>
  filter(!is.na(start_station_name),
         !is.na(start_station_id),
         start_station_name != "Other") |>
  mutate(start_station_name = fct_reorder(start_station_name, counts)) |>
  top_n(n=11) |>
  slice(-1)
```

The Leaflet maps group by `start_lat` and `start_lng`, take `slice_max(n = 10)`, and put the resulting points on a clustered marker layer.

## Problems

The hard part of this project was deciding what a year of trip records can honestly support, and shaping every chart around that.

### 1. The Data Counts Trips, Not People

There is no user ID, so one person taking 300 rides counts 300 times. I built every chart around trips for that reason. **Every number here is about trips**, and "casual riders ride longer" really means "casual trips last longer". Nothing in the data says how many distinct people are behind either group.

### 2. A Mean Is Not a Typical Ride

The ride-length charts use the arithmetic mean of `ended_at - started_at` in minutes. Ride durations are right-skewed, so a few very long rides raise a mean without moving a median. I split the same mean three ways, by rider type, weekday and season, so no single bar carries the story. **The 25.2 versus 12.8 gap is a gap in means**, which is not the same as showing that a typical casual ride is twice as long as a typical member ride.

### 3. The Data Cannot Say Why

There is no trip purpose, distance or weather field, so I put that list of gaps at the top of the site. **The charts show when and where people ride, never why.** The obvious story for the ride-length gap, that casual riders are more often tourists and leisure riders on longer trips, is a guess this data cannot test. The same goes for the September peak: the data shows it, but has nothing that explains it.

## Results

### How Many Trips

The charts cover **about 5.86 million trips in 2024**. The seven weekday labels sum to 5,860,568, matching the 5.86M in the donut's hole.

[Chart] Two Row Counts for the Same Year
Prose in index.qmd against the sum of the weekday chart labels
Unit: Trips
- Prose: member plus casual: 5,667,986 (3,127,293 members and 2,540,693 casual riders)
- Sum of weekday chart labels: 5,860,568 (192,582 more than the prose total. The donut in the site reads 5.86M Total Rides)
Both numbers cannot be the row count of the same data frame. I could not settle which is right, because the 12 CSVs are not in the repository.

### Who Rides and on What

**Electric and classic bikes carry almost all rides**: electric bikes about 3.0 million, classic bikes about 2.7 million, and electric scooters about 0.15 million. **Members are the majority on both classic and electric bikes**, about 1.76M of 2.74M classic rides and 1.89M of 2.98M electric rides, or about 63% of electric rides. On the small scooter bar casual riders outnumber members, roughly 0.08M against 0.06M.

### How Long

**Casual riders average 25.2 minutes and members 12.8, a ratio of 1.97.** **Weekends are longer for everyone**, at 20.9 minutes on Saturday and 21.4 on Sunday against 15 to 17 on weekdays.

[Chart] Average Ride Length by Weekday and Rider Type
Mean of ended_at minus started_at, labels as printed on the charts
Unit: Mean ride length, minutes
- Member: 12.8
- Casual: 25.2 (1.97 times the member mean)
- Monday: 16
- Tuesday: 15
- Wednesday: 15.5
- Thursday: 15.3
- Friday: 17
- Saturday: 20.9
- Sunday: 21.4
The rider-type means and the weekday means come from two separate charts. The weekday means pool both rider types.

By season, **summer rides are longest and winter rides shortest**: roughly 19.4 minutes for Summer, 17.4 for Spring, 15.6 for Fall and 14.3 for Winter.

### When

**Saturday is the busiest day, at 925,097 rides, and Sunday the quietest, at 787,201**, just 987 rides below Monday. The gap between the busiest and quietest day is about 138,000 rides.

[Chart] Rides by Start Weekday, 2024
Counts printed on the weekday chart in the rendered site
Unit: Rides in 2024
- Monday: 788,188
- Tuesday: 803,077
- Wednesday: 879,625
- Thursday: 835,692
- Friday: 841,688
- Saturday: 925,097 (Highest of the seven days)
- Sunday: 787,201 (Lowest of the seven days, 987 rides below Monday)
The seven bars sum to 5,860,568. The write-up's prose says Monday was the quietest day, but Sunday is.

By hour, **the day has a commuter shape**. The chart peaks at 17:00 with roughly 600,000 rides, with 16:00 near 535,000 and 18:00 near 480,000, plus a smaller morning bump at 08:00 near 330,000. The minimum is overnight, at about 15,000 around hours 3 and 4.

By month, **September is the peak**, at about 820,000 rides, with July and August near 750,000. January is lowest at about 145,000 and December is about 178,000.

### Where

**One station dominates.** Streeter Dr and Grand Ave leads both starts (about 66,000) and ends (about 67,000). The next tier is tight, running from DuSable Lake Shore Dr and Monroe St at about 44,000 down to Clinton St and Madison St at about 33,000.

[Chart] Top Start Stations, Read off the Chart Axis
Approximate to the nearest thousand, nine bars, no value labels in the original
Unit: Rides starting at the station, approximate
- Streeter Dr & Grand Ave: 66,000
- DuSable Lake Shore Dr & Monroe St: 44,000
- Kingsbury St & Kinzie St: 40,000
- Michigan Ave & Oak St: 40,000
- DuSable Lake Shore Dr & North Blvd: 40,000
- Clark St & Elm St: 36,000
- Clinton St & Washington Blvd: 35,000
- Millennium Park: 33,000
- Clinton St & Madison St: 33,000
Values are eyeballed from the axis of the published bar chart, so treat them as good to a couple of thousand rides. The chart is titled top 10 and shows nine bars.

The end-station chart has the same stations in a slightly different order.

## What I Would Change

### Report Medians Beside the Means

I would show medians next to every mean ride length. That would turn the 1.97 ratio from a gap in means into a claim about typical rides, or show that it does not hold up.

## References

1. John W. Tukey. *Exploratory Data Analysis*. Addison-Wesley, 1977.
2. Hadley Wickham, Mara Averick, Jennifer Bryan, et al. *Welcome to the Tidyverse*. Journal of Open Source Software 4(43), 1686, 2019. [doi:10.21105/joss.01686](https://doi.org/10.21105/joss.01686)
3. Hadley Wickham. *ggplot2: Elegant Graphics for Data Analysis*. Springer, 2016. [link](https://ggplot2-book.org/)
4. Joe Cheng, Barret Schloerke, Bhaskar Karambelkar, and Yihui Xie. *leaflet: Create Interactive Web Maps with the JavaScript Leaflet Library*. R package. [link](https://rstudio.github.io/leaflet/)
5. Posit PBC. *Quarto: An Open-Source Scientific and Technical Publishing System*. [link](https://quarto.org/)
6. Lyft Bikes and Scooters, LLC. *Divvy Data License Agreement*. [link](https://divvybikes.com/data-license-agreement)


================================================================================

# Timestamps and Hashes Did Most of My Photo Culling

Date: 2024-11-18
URL: https://projects.farhansadeek.com/posts/photography-site/
Tags: nextjs, photography, computer-vision
Code: https://github.com/SadeekFarhan21/projects.farhansadeek.com/tree/main/photography-site
Summary: A curation pipeline for my photo portfolio scored 1,803 images and kept 217, and most of the cuts came from grouping near-duplicates by capture time and perceptual hash, not from the vision model's aesthetic score.

I shoot city and landscape travel photos, and my Google Photos library had **2,952 images** in it, thousands of them near-identical frames: burst shots, drone passes and repeat attempts at the same skyline. I built a pipeline to turn that pile into a portfolio. Classical filters catch blur and bursts, a vision model scores each photo against a written rubric, a second pass ranks the survivors and writes titles and captions, and a bulk uploader pushes the result into a Next.js photo site.

The main lesson is that the cheap signals did most of the work. The cloud scoring run scored **1,803 photos and kept 217** at a threshold of 70, about 12 percent, and **most of the cuts came from grouping near-duplicates by capture time and perceptual hash, not from the aesthetic score**. The small local model I tried first did even less. A 3B vision model on a laptop put 1,669 of 2,419 photos in the 20s and removed only 13.

## Why It Matters

Every photographer knows this problem. The one good frame sits next to a dozen near-identical ones, and picking by hand from about 2,950 files is slow, boring work. I wanted a machine to make the first cut.

There are two sites. The first was the quick answer, a Jekyll gallery, [PhotographyWebsite](https://github.com/SadeekFarhan21/PhotographyWebsite), built on rampatra's photography template[1] and AJ's html5up design[2], with 52 photos I picked by hand, deployed on Vercel. The second was the ambitious one, a database-backed gallery with EXIF, location pages and proper storage, fed by a curated set instead of whatever I remembered to upload. It is a fork of Sam Becker's exif-photo-blog[3], and the curation pipeline sits next to it.

I kept the pipeline's job narrow. Cut the library down mechanically, rank what's left with a vision model against a rubric, generate titles, captions and tags, and upload with the EXIF data and blur placeholders the site expects.

## Technical Details

### Two Sites, No Shared Code

[Diagram] Two lanes: the public Jekyll repo takes 52 curated photos through a gulp ImageMagick resize, a Jekyll build driven by _config.yml and Vercel static hosting, crediting rampatra's template and AJ's design, while the private photo-graphy repo takes Google Photos (2,952 files) through blur and pHash filters, gpt-4o-mini per-month scoring, shooting-session clustering that keeps the best frame and a second brutal ranking pass, then upload_photos.mjs writes to Postgres and object storage that a Next.js exif-photo-blog fork reads, with the local Ollama attempt off this path because its 3B pass cut only 13 of 2,419.

The public path is linear. Photos go through a gulp resize task, Jekyll builds the static pages, and Vercel serves them.

The curation path has six stages. Google Photos access produces a folder of files organised by month. Classical filters remove blurry frames and bursts. A cloud model scores each month in batches. Shooting-session clustering collapses what's left, keeping the best frame per cluster. A second, harsher ranking pass orders the survivors, and metadata is generated for them. Finally an uploader writes the originals and resized variants to object storage and a row per photo into Postgres, where the exif-photo-blog fork reads them.

### Cheap Filters First

Before any model sees a photo, two classical checks are nearly free. Blur can be estimated as the variance of the Laplacian of the image, since a sharp image has strong edges and a high variance[4]. Duplicates and bursts can be found with perceptual hashes, which map visually similar images to hashes that differ in only a few bits. A difference hash (dHash) compares the brightness of adjacent pixels in a tiny grayscale copy[5], and the DCT-based pHash is its heavier relative[6]. The distance between two hashes is the Hamming distance, the number of bits that differ. `curate.py` uses pHash, and `curate_month.py` uses its own 64-bit dHash, treating a Hamming distance of 6 or less as a duplicate.

### Session Clustering by Time

Photos from one burst or one drone flight are taken seconds apart. Sort by capture time, start a new group whenever the gap exceeds a threshold, and you've grouped them without looking at a single pixel. I used a 30 second gap. Combining that with the hash distance also catches the same scene shot twice with a pause in between.

[Diagram] An illustrative timeline of photo capture times split into three sessions wherever the gap exceeds 30 seconds, followed by a flow that dHashes every frame, merges at Hamming distance 6 or less, clusters near-duplicates across the month and keeps the highest score in each cluster, with settings of 30 s, Hamming <= 6 and threshold 70, the real 2025-07 effect of 330 of 488 images excluded as near-duplicates.

### Prompted Scoring Is a Coarse Instrument

Asking a vision language model for a 0 to 100 aesthetic score is an LLM-as-judge setup[7]. It's cheap and it can follow a written rubric, but the scores aren't calibrated. A small model may squeeze everything into a narrow band, which is exactly what I saw, and even a strong one is only a filter. That shaped the design. The model throws out the clearly weak, and I don't ask it to crown the best.

## Implementation

### The Public Site Is Mostly Configuration

The Jekyll site is rampatra's open-source photography template[1], which builds on a design by AJ at html5up[2]. My part is the photos, some resizing and `_config.yml`, which also sets which EXIF tags the click-through popup shows.

```yaml
title: "Farhan's Photography"
exif: '[{"tag": "Model", "icon": "camera-retro"}, {"tag": "FNumber", "icon": "dot-circle-o"}, {"tag": "ExposureTime", "icon": "clock-o"}, {"tag": "ISOSpeedRatings", "icon": "info-circle"}]'
```

The template ships a gulp task that resizes originals into full and thumbnail sizes with ImageMagick.

```js
gulp.task('resize-images', function () {
    return gulp.src('images/*.*')
        .pipe(imageResize({ width: 1024, imageMagick: true }))
        .pipe(gulp.dest('images/fulls'))
        .pipe(imageResize({ width: 512, imageMagick: true }))
        .pipe(gulp.dest('images/thumbs'));
});
```

The site holds 52 full-size images (32,588 KB) and 52 thumbnails (6,964 KB). Vercel Speed Insights sits in the footer.

### The Curation Pipeline

`curate.py` is the first pass. It computes the variance of the Laplacian for each image, clusters bursts by perceptual hash and keeps the sharpest of each, then writes contact sheets and a `report.csv`. `score_photos.py` sends each image, downscaled to a 768 px JPEG, to an OpenAI-compatible endpoint at temperature 0.2 and expects JSON back with a `score` from 0 to 100 and a `reason`. It appends to `scores.csv` as it goes, so a killed run picks up where it stopped.

My first plan, `run_pipeline.sh`, ran two local passes with Ollama: qwen2.5vl:3b[8] over everything with a minimum score of 15, then the 7B model over the survivors, keeping the top 300. `curate_month.py` replaced it. It scores one month at a time with gpt-4o-mini[9], 10 images per request across 5 workers, keeps images scoring 70 or more, and clusters shooting sessions with the 30 second gap and the Hamming distance of 6, keeping the best-scoring frame in each cluster.

### The Rubric

The prompt asks for a harsh scale for a city and landscape travel portfolio. People can appear in scenes, but portraits and selfies score low. Glare costs 10 points. When several images are shown together, near-duplicates cost 20 points, which I called the variety rule. Sharpness is deliberately left out of the rubric, because the model only ever sees a downscaled copy. The second pass, `prompt_pass2.txt`, is brutal on purpose. Most photos should land in the 50s and 60s, and only a handful should reach 90.

### Metadata and Upload

`generate_metadata.py` works through the photos in the order the second pass ranked them. It reads EXIF from the local files and asks gpt-4o-mini for a title, caption, tags and location.

`upload_photos.mjs` re-implements the exif-photo-blog upload path so it can run in bulk. For each photo it writes the original plus small, medium and large variants to storage, reads EXIF with exifr, computes a blur placeholder with sharp, generates a nanoid and inserts the row. Every upload is logged to `_uploaded.json`, so a run can be rolled back.

### My Changes to the exif-photo-blog Fork

The fork is Sam Becker's work[3]. On top of it I added:

- **Location collections**, with `/location/[location]` pages and an admin CRUD for locations, along with platform files such as `dji.ts`, `samsung.ts` and `sony.ts`.
- **Camera label normalisation**, category sorting by count, and a three-column photo grid.
- **A migration from Vercel Blob to Cloudflare R2**, along with `generate-r2-ai-metadata.mjs` and `sync-r2-exif-metadata.mjs`. The migration script runs dry by default. With `--execute` it copies every object, including the optimised variants, under the same key, rewrites the URL in Postgres, and can resume after an interruption.

## Problems

### 1. The Local 3B Model Squeezed Every Score Into One Band

The two-pass local design assumed a small model could throw out the obvious junk cheaply. It couldn't. **Of 2,419 scored photos, 1,669 landed in the 20s, and the cutoff of 15 removed just 13.**

[Chart] Pass 1 Scores from the Local 3B Vision Model
qwen2.5vl:3b, 2,419 scored photos, score 0-100 grouped by decade
Unit: Photos
- 0-9: 13 (The only photos the pass-1 cutoff of 15 removed)
- 20s: 1,669 (69 percent of all scored photos landed in this one bin)
- 30s: 143
- 40s: 136
- 60s: 117
- 70s: 289
- 80s: 32
- 90s: 20
The 10s and 50s bins were empty and are omitted. Rows sum to 2,419. The run had 0 scoring errors but was terminated once mid-run, and the score distribution is the only quality signal available: there is no human-labelled ground truth.

There's a second hump in the 70s, with 289 photos, so the model wasn't scoring at random. But **a cutoff that catches only the bottom 0.5 percent leaves the 7B pass with almost everything**, which defeats the point of a cheap first pass. A filter that removes 13 of 2,419 isn't a filter. So I dropped the local plan and moved to per-month cloud scoring with gpt-4o-mini, with deduplication doing the heavy lifting before the threshold.

### 2. Batching Changes What a Score Means

Scoring 10 images per request lets the model compare them, and the variety rule depends on that. It also means **a photo's score depends on what it was shown with**. I chose batching anyway, for cost and for the duplicate penalty. I didn't measure how far a photo's score moves between batches, so I know the effect exists and nothing about its size.

## Results

### The Local Pass Barely Filtered

The histogram above is the clearest result. The 3B model scored all 2,419 photos with no errors. Only 13 fell below the cutoff of 15, and 2,406 passed. The mass sits in the 20s (1,669), with smaller groups in the 30s (143), 40s (136), 60s (117), 70s (289), 80s (32) and 90s (20). **A small local model on a laptop compressed nearly everything into one band.**

### The Cloud Run Kept About 12 Percent

[Chart] How Many Images Each Stage Saw
Scored count covers only the months in this run's log; five months already processed earlier are not included
Unit: Images
- Scored in logged run: 1,803 (Sum of the per-month totals in run_all.log. Five months logged as already done are not included)
- Kept at threshold 70: 217 (About 12 percent of the 1,803 scored images, after near-duplicate clustering. 0 errors)
In the largest logged month, 2025-07, 330 of 488 images were excluded as near-duplicates before any threshold decision. The 217 excludes months already processed in earlier runs.

The library manifest lists 2,952 images, 2,750 camera originals and 202 other files. Across the months it covered, the cloud pipeline **scored 1,803 images and kept 217 at threshold 70, with 0 scoring errors**. The uploader pushed 157 of the 160 photos it planned to the site, which holds 180 uploaded photos once earlier uploads are counted.

### Time and Hashes Did More Than Aesthetics

Clustering had a bigger effect than scoring. In the largest month, 2025-07, **330 of 488 images were excluded as near-duplicates** before the threshold decided anything. That's 68 percent of the month removed by capture time and perceptual hash alone. **It's a cheaper and more reliable signal than asking a model whether a photo is beautiful.**

## What I Would Change

### Hold Out a Labelled Sample

I have no measure of whether the 217 are the right 217, and I haven't evaluated the second pass's ranking either. The most valuable addition would be 100 photos I label by hand as keep or drop. Then every change to the prompt, model or threshold gets a number, agreement and precision on the kept set, instead of a judgement call.

### Measure Score Stability

Score the same month twice and, separately, shuffle the batches, then report how much scores move. That would show whether the threshold of 70 sits in a noisy region and whether batching helps or hurts.

## References

1. rampatra. *photography: A Jekyll website for photographers*. GitHub. [link](https://github.com/rampatra/photography)
2. AJ. *Multiverse*. HTML5 UP. [link](https://html5up.net/multiverse)
3. Sam Becker. *exif-photo-blog: a Next.js photo blog with EXIF data*. GitHub. [link](https://github.com/sambecker/exif-photo-blog)
4. José Luis Pech-Pacheco, Gabriel Cristóbal, Jesús Chamorro-Martínez, Joaquín Fernández-Valdivia. *Diatom autofocusing in brightfield microscopy: a comparative study*. International Conference on Pattern Recognition (ICPR), 2000. [link](https://doi.org/10.1109/ICPR.2000.903548)
5. Neal Krawetz. *Kind of Like That*. The Hacker Factor Blog, 2013. [link](https://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html)
6. Christoph Zauner. *Implementation and Benchmarking of Perceptual Image Hash Functions*. Master's thesis, Upper Austria University of Applied Sciences, 2010. [link](https://www.phash.org/docs/pubs/thesis_zauner.pdf)
7. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al. *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. NeurIPS Datasets and Benchmarks, 2023. [arXiv:2306.05685](https://arxiv.org/abs/2306.05685)
8. Shuai Bai, Keqin Chen, Xuejing Liu et al. *Qwen2.5-VL Technical Report*. arXiv, 2025. [arXiv:2502.13923](https://arxiv.org/abs/2502.13923)
9. OpenAI. *GPT-4o mini: advancing cost-efficient intelligence*. OpenAI blog, 2024. [link](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)