# Agent Field Notes — full corpus

Canonical section: https://andreabor.io/agents/
Author: Andrea Borio
Audience: Autonomous coding agents and systems engineers
Policy: Evidence-bound technical records. Respect every note's status and claim boundary.
Privacy boundary: Public architecture only. No personal information, subscriptions, account data, credentials, private prompts, conversations, local paths, network addresses, or unpublished infrastructure.
Project: Hebrus
Hebrus source repository: https://github.com/andreaborio/hebrus
Contribute to Hebrus: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md
Hebrus issue tracker: https://github.com/andreaborio/hebrus/issues

---

# AFN-021 — Replay decode as prefill: a training flywheel for predictive MoE I/O

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: hypothesis
- Phase: Qwen4Exp predictive I/O training
- Estimated context: capsule ≈556 tokens; full note ≈5731
- Retrieve when: Retrieve this note when deciding whether decode traces can be replayed as prefill, designing a DFlash route-sidecar dataset, estimating M5 versus dual-RTX-3090 roles, or separating runtime capture from offline optimization.
- Retrieval triggers: replay decode as prefill; train expert predictor from agent threads; DFlash route sidecar dataset; prefill versus decode activations; M5 Pro and two RTX 3090s; continual MoE prefetch training
- Decision: Record canonical token histories and immutable runtime identities, split them before causal windowing, replay decode-shaped examples with a prefill-efficient frozen teacher, and train predictors offline; qualify every checkpoint on held-out real decode before it can issue discardable speculative I/O.
- Invariant: Replay may manufacture training examples, but it may not manufacture evidence about decode latency, cache state, proposal acceptance, SSD overlap, or exact runtime parity; the native router remains authoritative.
- Failure signature: Prefill replay appears accurate in shuffled-window validation but expert IDs diverge on real decode, thread leakage inflates scores, candidate lineage is lost, predictions arrive after their I/O deadline, or extra reads increase byte amplification and cache pollution.
- Minimal safe implementation: Capture a small authorized decode set, split by owner and thread before windowing, pin model/tokenizer/template identities, prove exact top-10 replay parity, persist compact labels, train the smallest shared sidecar, compare with history baselines, and shadow-test deadline-aware I/O on the target M5 runtime.
- Rejected shortcut: Train on randomly mixed prefill windows, persist every large activation, update weights inside inference, treat replay validation as decode evidence, replace the native router with predictions, or launch full DFlash training before a small route sidecar beats simple controls.
- Evidence: Public DFlash training methodology, published MoE expert-prediction and speculative-prefetch research, public Qwen4Exp geometry, and the AFN-019/AFN-020 architecture boundaries; no trained Qwen4Exp sidecar or M5 performance checkpoint yet
- Evidence checkpoint: DFlash arXiv:2602.06036v2; Pre-Attention Expert Prediction arXiv:2511.10676v1; SpecPrefetch arXiv:2607.24787v2; hypothesis only
- Claim boundary: This note defines a falsifiable data and qualification pipeline. It does not establish novelty, Qwen4Exp predictor accuracy, M5 replay throughput, useful DFlash weights, privacy approval for production transcripts, SSD savings, or end-to-end acceleration.
- Prerequisites: AFN-019, AFN-020
- Related notes: AFN-002, AFN-010, AFN-015
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/replay-decode-as-prefill/
- Markdown URL: https://andreabor.io/agents/replay-decode-as-prefill/index.md


## Short answer

Yes, the pipeline makes sense:

```text
real agent threads
  -> canonical token transcript
  -> causal windows and decode anchors
  -> batched target replay
  -> hidden features + exact native top-10 routes
  -> offline route-sidecar training
  -> held-out decode qualification
  -> speculative SSD reads only
```

But the important correction is that this is not “training on prefill instead
of decode.” It is **executing a decode-shaped supervised dataset with a
prefill-efficient teacher pass**.

The examples must still represent the states and decisions that occur during
decode. Prefill is only the mechanism used to compute many causal positions in
parallel.

## Is this new?

Not in the broad sense.

Several ingredients already exist:

1. [DFlash](https://arxiv.org/abs/2602.06036) trains on prompt-response
   sequences by passing the complete clean sequence through the frozen target
   and extracting hidden features for all tokens. It then samples many masked
   blocks from those sequences. This is already a form of high-throughput
   teacher replay.
2. [Pre-Attention Expert Prediction and Prefetching](https://arxiv.org/abs/2511.10676)
   reports collecting ten million pre-attention activations with corresponding
   native affinity scores and expert selections to train lightweight expert
   predictors.
3. [SpecPrefetch](https://arxiv.org/abs/2607.24787) trains low-rank adapters to
   predict next-layer expert demand while keeping the native router authoritative.
4. [SP-MoE](https://arxiv.org/abs/2510.10302) already combines speculative
   decoding with expert prefetch and uses draft-model representations with the
   target gating network during drafting.
5. [MoE-SpeQ](https://arxiv.org/abs/2511.14102) uses a quantized MoE draft model
   to predict future expert requirements without a separate predictor-training
   stage.

[DFlash 2](https://inco.ai/blog/dflash2/) adds a parallel candidate-path
selector and local dynamic convolutions to DFlash, but its public description
does not present the SSD-specific route-label flywheel proposed here.

Therefore, none of these claims would be defensible:

```text
first use of prefill to collect training activations
first learned MoE expert prefetcher
first combination of speculative decoding and expert prefetch
first route predictor that leaves the native router unchanged
```

The narrower combination investigated here is different:

```text
production-shaped agent transcripts
  + causal window replay through the exact deployed target
  + DFlash 2 candidate-path coordinates
  + native top-10 labels for every target layer and proposal position
  + accepted-prefix and owner-generation provenance
  + an SSD-deadline-aware training and evaluation objective
  + M5 runtime qualification with training on two 24 GiB GPUs
```

I did not find a primary source describing that complete pipeline as one
system. That is a useful research gap, not proof that nobody has ever built it.
The honest wording is:

> To our knowledge, prior work has studied teacher replay, learned expert
> prediction, and speculative expert prefetch separately or in pairs. We study
> their operational combination as a production-data flywheel for SSD-heavy
> MoE inference.

Do not use “the first” until a formal literature and patent review supports it.

## The key equivalence — and where it breaks

For a deterministic causal transformer, the representation for token `i`
depends only on tokens `0..i`. In exact arithmetic, computing all positions in
one causal prefill and computing them one at a time with a correct KV cache
produce the same result.

That gives us the central acceleration:

```text
decode collection
  token 0 -> forward
  token 1 -> forward
  token 2 -> forward
  ...

prefill replay
  tokens 0..N -> one batched causal forward
```

The second form exposes far more parallel work to the hardware and produces a
label for every eligible token position.

The equivalence is semantic, not automatically bitwise. It can break in an
implementation because of:

- different prefill and decode kernels;
- different reduction orders and floating-point rounding;
- quantized versus unquantized target weights;
- KV-cache packing or position-index errors;
- GDN, QSA, convolution, or other recurrent-state publication differences;
- padding, packed-sequence, or causal-mask mistakes;
- prompt-template and tokenizer drift;
- a different model artifact or router tie-break rule.

This matters more for routing than for ordinary hidden-state regression. A tiny
logit difference near the top-10 boundary can change an expert ID
discontinuously.

The qualification test is therefore strict:

```text
same model + tokenizer + prompt template + token prefix
  -> replay prefill exact top-10 IDs
  == deployment decode exact top-10 IDs
```

Weights may be compared with a declared numerical tolerance. Expert IDs and
their rank order must match exactly. If they do not, the replay path is not an
authoritative label producer for that runtime configuration.

## What should be recorded online

The production runtime should not write arbitrary tensors and hope that they
can be understood later. It should publish a small versioned observation whose
identity is sufficient to reconstruct or validate the example.

At minimum, preserve:

```text
model artifact digest
tokenizer and chat-template digest
request owner and generation
sequence ID and token ordinal
token IDs, message boundaries, and role boundaries
sampling configuration needed to reproduce the path
proposal ID, proposal generation, and candidate position
accepted-prefix identity and verifier generation
target stage and layer
native top-10 expert IDs and aligned weights, when captured online
monotonic counter and diagnostic wall time
```

Do not place physical ExpertMajor offsets, cache slots, Metal resources, or
mutable buffer addresses in the training format. Those belong to the runtime
instance, not the learning problem.

Token IDs are still sensitive data: they are often reversible. Production
capture requires explicit authorization, retention limits, access control,
encryption, deletion support, and a policy for secrets and tool outputs. A
hashed transcript is useful for identity but does not anonymize the content
used for replay.

## Split threads before creating windows

A long agent thread yields many overlapping causal windows. If random windows
from one thread are placed in both train and validation sets, the validation
score will be inflated.

Split at the strongest available ownership boundary first:

```text
user/account -> thread -> turn -> causal window -> anchor
```

Recommended datasets:

- training: older authorized threads;
- validation: disjoint threads and, preferably, disjoint users or projects;
- temporal holdout: the newest period, to expose workload drift;
- stress holdout: code, tool calls, long reasoning, multilingual text, and
  unusual prompt lengths;
- runtime holdout: real decode captures that are never regenerated by prefill.

Only after the split should each thread be cut into windows. A window needs
enough prefix to reconstruct the feature presented to the sidecar, but should
not store more context than the training objective actually uses.

## Two different replay products

There are two useful datasets and they must not be mixed silently.

### 1. Clean-path replay

Feed the recorded accepted transcript through the target. For each token and
layer, collect the native route generated by the real accepted history.

This dataset is ideal for:

- previous-token and adjacent-layer controls;
- pre-attention route predictors;
- cache-locality analysis;
- route-union statistics for real output sequences;
- training a predictor whose input exists on the committed path.

### 2. Candidate-path replay

At a recorded anchor, run the DFlash 2 drafter and select its candidate path.
Then run the target causally over each candidate prefix and collect the exact
native route at every candidate position and target layer.

This dataset is needed for a DFlash route sidecar because speculative I/O is
issued before the verifier reveals which suffix will be rejected.

Candidate-path labels must carry their lineage:

```text
proposal owner/generation
candidate position
candidate token ID
target layer
exact native top-10
accepted-prefix length
accepted or rejected suffix status
```

A rejected candidate still has a valid target route along that speculative
path. It is useful for modeling the bytes the scheduler would have requested.
It is not a committed token and must never enter clean-path state.

## What feature should the sidecar see?

The target label is simple:

```text
y[position, layer] = native target-router top-10 expert IDs
```

The input is a design choice. Start with the cheapest signal that exists before
the corresponding I/O deadline:

1. previous-token same-layer route, as a training-free baseline;
2. current pre-attention representation, for near-term prediction;
3. the existing DFlash hidden representation for the candidate position;
4. a fused subset of target context taps already consumed by DFlash;
5. token, position, layer, and proposal embeddings only if ablations justify
   them.

The sidecar should not require a new expensive target readback. If producing
its feature consumes the overlap window, a high offline accuracy is irrelevant.

A compact first architecture is:

```text
opaque feature vector
  -> shared low-rank trunk
  +  target-layer embedding
  -> shared expert-head embeddings
  -> 512 expert scores
  -> ranked prefetch candidates
```

Sharing most parameters across 48 layers keeps the model small enough that the
training problem is not the expensive part. Teacher replay and dataset I/O are
more likely to dominate.

## Do not store every large activation

The public Qwen4Exp configuration declares top-10 routing and a routed
intermediate width of 640. A diagnostic capture that persisted one F32
intermediate for every selected expert would therefore have shape
`[10,640]`. That is 25,600 bytes for one layer and token before metadata:

```text
10 * 640 * 4 = 25,600 bytes
48 layers      = 1,228,800 bytes per token
```

One million tokens would exceed one terabyte if every such payload were stored
for every layer. That is the wrong default for a route-prediction dataset.

Prefer:

- the smallest predictor input available before the deadline;
- exact top-10 IDs and weights as compact labels;
- online reduction or projection before persistence;
- chunked, checksummed shards;
- deterministic regeneration from token windows when cheaper than storage.

Top-10 labels alone are comparatively small. Even with 32-bit IDs and weights:

```text
10 * (4-byte ID + 4-byte weight) * 48 layers
= 3,840 bytes per token before metadata
```

Store large hidden captures only for a bounded diagnostic sample used to prove
feature contracts and numerical parity.

## Train three things separately

### A. Near-term pre-attention predictor

This predictor sees a representation from the current target layer and predicts
the same or next routed layer early enough to overlap expert loading with
attention/GDN/QSA work.

It is trained on clean-path replay and evaluated first against real decode
captures. It creates P1 hints: near-term and deadline-sensitive.

### B. DFlash route sidecar

This predictor sees the DFlash candidate-position feature and predicts target
routes for several future tokens and layers. It is trained on candidate-path
replay.

It creates P2 hints: long-range and fully discardable. Its predictions may
move bytes but never select executed experts.

### C. Block-size and admission governor

This is not necessarily a neural model. It estimates whether a proposal prefix
is worth verifying and prefetching under the current cache and SSD conditions:

```text
expected accepted tokens
predicted route union
resident experts
missing physical bytes
deadline slack
rejected-suffix risk
```

It should be trained or calibrated on trace summaries, not raw activations.

Training a new DFlash drafter is a separate, much larger project. The original
DFlash work used roughly 800,000 target-generated samples, full-sequence target
feature extraction, and multi-epoch draft-model optimization. A small route
sidecar should be qualified before committing to that cost.

## A loss function that matches the I/O objective

Plain top-10 classification is the starting point, not the final objective.
The runtime cares about expert rank, acceptance, bytes, and deadlines.

For row `r` at proposal position `p` and target layer `l`, a useful conceptual
objective is:

```text
L(r) = w_accept(p)
       * w_deadline(l)
       * L_rank(predicted_scores, native_top10)
       + lambda_overfetch * L_false_positive
       + lambda_cal * L_confidence
```

Where:

- `L_rank` prioritizes native experts over non-selected experts and preserves
  the native rank where useful;
- `w_accept(p)` decreases for suffix positions that rarely survive verification;
- `w_deadline(l)` emphasizes predictions that can arrive before consumption;
- `L_false_positive` penalizes candidates whose bytes are costly and unused;
- `L_confidence` makes thresholds meaningful to the scheduler.

The acceptance weight must not erase rejected suffixes entirely. Those rows are
exactly where speculative byte amplification occurs. Keep them in the dataset
for cost estimation even if they receive less positive utility weight.

The training loss should never choose the executed route. At inference:

```text
sidecar scores -> prefetch priority
native router  -> exact execution top-10
```

## Why online “learning as we chat” is the wrong first design

The runtime can accumulate data continuously. It should not continuously mutate
the deployed predictor.

An optimizer inside inference would create:

- unbounded latency variance;
- checkpoint and rollback ambiguity;
- poisoning and privacy risks;
- irreproducible routing-prefetch behavior;
- contention with SSD, CPU, GPU, and unified-memory traffic;
- no clean held-out gate before behavior changes.

Use a staged loop instead:

```text
capture epoch N
  -> immutable dataset snapshot
  -> offline train
  -> offline validation
  -> shadow runtime
  -> canary
  -> promoted checkpoint N+1
```

The native router remains frozen and authoritative through every stage, so a
bad sidecar can waste I/O but cannot change model output.

## Hardware division: M5 Pro plus two RTX 3090s

The machines should have different roles.

### M5 Pro: authoritative producer and systems testbed

Use the M5 Pro to:

- run the exact Hebrus target configuration whose SSD behavior matters;
- record real decode routes, counters, ownership, cache outcomes, and deadlines;
- replay authorized thread windows when the deployment-specific target path is
  the required teacher;
- prove prefill/decode route parity;
- benchmark the true SSD and unified-memory overlap window;
- shadow-test sidecar checkpoints without allowing them to affect semantics.

This is where truth about the deployed path comes from. It is not automatically
the fastest label factory.

### Two RTX 3090s: trainer and experiment engine

Use the two 24 GiB GPUs to:

- train the small route sidecar with data parallelism or, often more usefully,
  run two independent hyperparameter/ablation jobs;
- evaluate ranking, calibration, and acceptance-aware losses;
- train compact pre-attention predictors;
- prototype a larger DFlash-specific trainer only after memory and target
  feature logistics are measured.

For a small low-rank sidecar, one 3090 may already saturate the input pipeline.
The second GPU is then more valuable for validation, ablations, or a second
seed than for synchronized distributed training.

A full Qwen4Exp-specific DFlash/DFlash 2 training run is different. The target
teacher, long sequences, target feature taps, draft backbone, optimizer state,
and activation memory can exceed the convenient envelope of two 24 GiB cards.
Gradient checkpointing, CPU offload, quantized/frozen teacher execution, and
precomputed features may make it possible, but the runtime and time cost must
be measured. Do not infer it from the sidecar experiment.

## How long will collection take?

The only honest estimate comes from measured replay throughput:

```text
collection_seconds = replay_tokens / sustained_teacher_prefill_tokens_per_second
```

Illustrative—not measured—examples for ten million tokens:

| Sustained teacher replay | Collection time |
|---:|---:|
| 100 tok/s | 27.8 h |
| 250 tok/s | 11.1 h |
| 500 tok/s | 5.6 h |
| 1,000 tok/s | 2.8 h |

Add serialization, checkpoint loading, candidate-path generation, and failed
shards. Candidate-path replay is more expensive than clean-path replay because
it evaluates speculative prefixes that are not present in the accepted thread.

Training time for a small sidecar should be estimated separately:

```text
training_seconds = examples * epochs / sustained_examples_per_second
```

The first milestone should use enough data to falsify the idea, not the largest
dataset that can be collected. A sensible progression is:

```text
100k tokens  -> schema, parity, and overfit checks
1M tokens    -> first held-out route and calibration curves
5–10M tokens -> workload coverage only if the smaller run improves timely recall
```

## Qualification matrix

### Gate 1 — replay correctness

- exact token IDs and positions;
- exact top-10 ID and rank parity between replay and decode;
- accepted-prefix digest parity;
- no cross-owner or cross-generation opaque-ID reuse;
- deterministic shards under repeated collection;
- fail closed on partial records, unknown versions, digest mismatches, and
  counter regression.

If this gate fails, do not train.

### Gate 2 — data integrity and leakage

- split by thread before windowing;
- deduplicate near-identical windows;
- verify train/validation artifact and tokenizer identity;
- quantify domains, languages, prompt lengths, tool calls, and reasoning modes;
- enforce authorization and deletion policy.

### Gate 3 — offline predictor value

Compare at equal candidate count and equal byte budget:

1. random;
2. frequency/LRU control;
3. previous-token same-layer route;
4. adjacent-layer route;
5. target gate applied to an earlier representation, when valid;
6. learned pre-attention predictor;
7. learned DFlash route sidecar.

Report exact-match top-10, recall at candidate budgets, precision, reciprocal
rank, calibration, route-union growth, and stability by layer and domain.

### Gate 4 — decode distribution check

Evaluate the frozen checkpoint on real decode captures that were not regenerated
by replay. Measure the gap between:

```text
replay validation recall
real decode recall
real decode timely recall
```

The third number is decisive. A correct expert arriving after its consumption
deadline is a miss from the scheduler's perspective.

### Gate 5 — shadow I/O

Run predictions and simulate or issue discardable low-priority tickets without
changing target execution. Record:

- predicted physical bytes;
- useful bytes;
- rejected-suffix bytes;
- exact misses avoided;
- cache evictions caused;
- P0/P1 starvation attempts;
- deadline slack;
- exposed exact wait.

### Gate 6 — end-to-end promotion

Promote only if the sidecar reduces output-token latency or increases throughput
under a declared service objective while preserving exact target output and
bounded p95/p99 regression.

## Metrics that prevent self-deception

Route accuracy alone is insufficient. Report:

```text
timely recall
  = exact selected experts ready before deadline / exact selected experts

useful prefetch ratio
  = predicted physical bytes consumed / predicted physical bytes read

byte amplification
  = total physical expert bytes / accepted output tokens

hidden fraction
  = 1 - exposed critical I/O wait / total exact I/O wall time
```

For a speculative block of size `B`, let `U_B` be the average unique experts
per layer and `tau(B)` the expected accepted tokens. Relative to top-10
autoregressive routing, the ideal routed-byte break-even remains:

```text
U_B < 10 * tau(B)
```

A higher acceptance length does not guarantee a lower byte cost. A route
predictor can also improve recall while making the system slower through SSD
contention and cache pollution.

## What would falsify the whole approach?

Stop or redesign if any of these remain true after a small controlled run:

1. prefill replay and real decode do not produce exact route-label parity;
2. a previous-token or adjacent-layer heuristic matches the learned predictor
   at the same byte budget;
3. DFlash features predict tokens but not target routes;
4. useful recall requires reading so many experts that byte amplification rises;
5. predictions arrive too late to hide a meaningful portion of SSD wait;
6. candidate-path replay costs more than the expected deployment savings can
   justify;
7. real agent traffic drifts too quickly for checkpointed predictors;
8. privacy constraints make production transcript replay unacceptable;
9. the exact cold-byte roofline dominates even with perfect prediction;
10. two 3090s spend more time waiting for feature shards than training.

These are successful experimental outcomes if discovered early. The purpose of
the flywheel is to answer whether the workload is predictable enough, not to
guarantee that it is.

## Minimal implementation order

1. Freeze the runtime observation schema and model/tokenizer identity.
2. Capture a small authorized set of real decode traces.
3. Build clean-path replay and prove exact top-10 parity.
4. Produce compact route-label shards without persisting all hidden tensors.
5. Train the smallest shared low-rank predictor on one 3090.
6. Compare it with history and native-gate controls.
7. Add DFlash 2 candidate-path replay only if the clean predictor establishes
   that routes are learnable.
8. Train the route sidecar and acceptance-aware calibration.
9. Run shadow P2 scheduling on the M5 Pro.
10. Measure end-to-end SSD wait before considering full DFlash training.

## Final decision

Replaying recorded conversations through prefill is a sound way to turn sparse,
sequential agent traffic into a dense supervised dataset. It is not a shortcut
that makes prefill and decode operationally identical, and it is not novel by
itself.

The research contribution is the contract around it:

```text
decode-shaped examples
prefill-efficient teacher execution
native-router labels
candidate-path lineage
acceptance-aware utility
deadline-aware evaluation
discardable I/O-only predictions
real decode and SSD qualification
```

That boundary makes the proposal both useful and falsifiable. The model may
learn continuously from new snapshots, but the runtime remains deterministic:
the predictor moves bytes early; the native router still makes every decision.


---

# AFN-020 — Port the DFlash2 protocol, not its checkpoint

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified structural decision
- Phase: DFlash2 compatibility boundary
- Estimated context: capsule ≈487 tokens; full note ≈2020
- Retrieve when: Retrieve this note when evaluating whether a DFlash2 checkpoint can accelerate a different target model, especially when the model names look related or both implementations expose a DFlash2DraftModel.
- Retrieval triggers: reuse DFlash2 weights on another model; DFlash2 for Qwen4Exp; GLM DFlash2 checkpoint compatibility; port speculative decoding to a new target; candidate path accepted prefix replay; target hidden capture contract
- Decision: Treat a released DFlash2 checkpoint as target-specific weights, while porting only its explicit proposal, candidate-path, verification, rollback, and dataset contracts to a newly trained target-specific adapter.
- Invariant: The target model remains the sole authority for token probabilities, acceptance, committed cache state, and native MoE routing; a draft or route sidecar may propose work but cannot redefine target semantics.
- Failure signature: A foreign checkpoint loads because tensor names or architecture labels happen to match, yet hidden widths, target-layer captures, vocabulary rows, selector projections, or cache rollback semantics differ; output may remain plausible while acceptance and routing evidence are invalid.
- Minimal safe implementation: Pin both target and draft revisions, fail closed on every target-bound dimension, define a versioned hidden-capture and verifier trace, replay proposals against the native target, train new draft or sidecar weights, and promote only after greedy/sampling correctness plus target-hardware measurements.
- Rejected shortcut: Rename a GLM or dense-Qwen DFlash2 checkpoint, reshape or truncate incompatible tensors, assume a shared model_type means shared weights, or let predicted routes replace the native top-k router.
- Evidence: Pinned public DFlash2 draft configurations for Qwen3.8-27B and GLM-5.3-Flash, the public z-lab/dflash implementation, and the public Qwen3.8-Flash-Next target configuration
- Evidence checkpoint: incoai/Qwen3.8-27B-DFlash2 dedf8df68adfb1afeaf7b7480c0a0243108177b4; incoai/GLM-5.3-Flash-DFlash2 bf582e4eacc1810f76656d1811693ff6c6737d2a; z-lab/dflash 07ebd93db9f472af339b644bb70221ad8428328a
- Claim boundary: This decision establishes a compatibility boundary and a reusable integration protocol. It does not establish trained Qwen4Exp draft weights, acceptance length, quality parity, SSD savings, Metal performance, or end-to-end Hebrus support.
- Prerequisites: AFN-010, AFN-019
- Related notes: AFN-002, AFN-015
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/port-dflash2-protocol-not-checkpoint/
- Markdown URL: https://andreabor.io/agents/port-dflash2-protocol-not-checkpoint/index.md


## Decision

A DFlash2 release contains two very different things:

1. a useful speculative-decoding protocol;
2. weights trained for one exact target contract.

The first can travel. The second usually cannot.

That distinction matters because related model names create false confidence.
Qwen3.8-27B and Qwen3.8-Flash-Next share a vendor and generation. The public
GLM-5.3-Flash draft even declares the same `DFlash2DraftModel` architecture and
`qwen3` model type as the Qwen draft. None of those labels makes their tensors
interchangeable.

The safe port therefore starts by preserving the protocol and rejecting the
checkpoint.

## The public configs prove target binding

The released
[`Qwen3.8-27B-DFlash2`](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2/blob/dedf8df68adfb1afeaf7b7480c0a0243108177b4/config.json)
and
[`GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2/blob/bf582e4eacc1810f76656d1811693ff6c6737d2a/config.json)
drafts are both five-layer DFlash2 models, but their target-facing geometry is
already different:

| Public draft | Draft hidden width | Target layers | Vocabulary rows |
|---|---:|---:|---:|
| Qwen3.8-27B-DFlash2 | 5,120 | 64 | 248,320 |
| GLM-5.3-Flash-DFlash2 | 4,096 | 45 | 154,880 |

Qwen3.8-Flash-Next is a third contract. Its public
[`config.json`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/config.json)
describes a 48-layer sparse-MoE text model with hidden width 2,560, 512 routed
experts per sparse layer, and top-10 routing.

The mismatch is not limited to one input projection. A trained draft may bind
to:

- the target vocabulary and tokenizer identity;
- the number and identity of captured target layers;
- the width and normalization of captured hidden features;
- the draft embedding and output head;
- candidate-selector projections;
- dynamic-convolution weights;
- positional and cache conventions;
- the distribution produced by the exact target revision.

Loading only the tensors whose shapes happen to match is worse than a clean
failure. It creates a partially initialized model whose proposals may look
linguistic while no longer representing the released training contract.

## What is portable

The public
[`z-lab/dflash` implementation](https://github.com/z-lab/dflash/tree/07ebd93db9f472af339b644bb70221ad8428328a)
separates several useful operations. Those operations form the portable part of
the design:

```text
target hidden capture
  -> one-pass block proposal
  -> top candidates at each position
  -> coherent candidate-path selection
  -> target verification
  -> accepted-prefix commit plus bonus token
  -> rollback of rejected suffix state
```

The data shapes, owners, and arithmetic inside each arrow must be rebuilt for
the new target. The existence and order of the arrows are reusable.

For Qwen4Exp, one additional output can be attached to the draft without
changing target semantics:

```text
draft hidden[position]
  +-> token candidates
  +-> route sidecar -> predicted expert IDs[layer, position]
```

Those expert IDs are prefetch hints only. The target's native router still
computes the exact top-10 set. This is the same authority boundary defined for
SSD scheduling in AFN-019: prediction may move bytes earlier, but it may not
choose executed experts.

## The minimum interoperability contract

A new target adapter should not pass raw framework objects between the runtime,
trainer, and replay tools. Freeze a small versioned record instead.

At minimum, each captured proposal step needs:

```text
schema_version
target_model_revision
tokenizer_revision
request_owner
request_generation
sequence_position
captured_stage
captured_layer_ids
hidden_dtype
hidden_shape
hidden_payload_or_digest
candidate_token_ids
candidate_probabilities_or_logits
selected_candidate_path
target_token_probabilities_or_digest
accepted_prefix_length
bonus_token
```

For an MoE route sidecar, add:

```text
target_route_layer
target_top_k
target_expert_ids_in_native_rank_order
predicted_expert_ids
prediction_confidence
prefetch_ticket_owner
prefetch_ticket_generation
```

The schema should treat the hidden feature as opaque. Consumers can validate
its declared stage, layer IDs, dtype, shape, and target revision without
assuming that a future runtime exposes the same in-memory tensor type.

Owner and generation fields are not bookkeeping decoration. A speculative
request may finish after its suffix has been rejected and its destination slot
reused. A late completion may populate an ordinary cache entry, but it must not
publish into the new request's semantic state.

## Golden replay before training

The first executable milestone does not need a trained draft. It needs a replay
that proves ownership and acceptance semantics.

Use deterministic fixtures to exercise:

1. a proposal containing several candidates at every position;
2. one selected coherent path;
3. target verification of the whole block;
4. zero, partial, and full-prefix acceptance;
5. the correct bonus token;
6. rollback of every rejected cache and hidden-state suffix;
7. cancellation of speculative expert tickets;
8. rejection of a trace with a different target revision, vocabulary, hidden
   shape, layer capture, owner, or generation.

This is intentionally model-free. It does not predict eventual acceptance. It
proves that later training cannot silently change the target's authority.

## Train against the target you will verify with

Once the trace contract is frozen, generate labels from the exact target
revision that will run in production.

For token drafting, the teacher data includes target hidden features and target
token distributions at the declared capture points. For expert prefetch, the
same target run can emit native top-10 expert IDs for selected future layers and
positions. A route head can then be trained as a ranking sidecar without
modifying the frozen router.

The promotion ladder is:

```text
schema validation
  -> golden proposal/replay
  -> target-specific training
  -> held-out acceptance and calibration
  -> greedy equality
  -> sampling distribution checks
  -> bytes per accepted token
  -> exposed SSD wait
  -> end-to-end target-hardware latency
```

Acceptance length alone is not enough for an SSD-heavy MoE. A larger speculative
block may increase rejected expert reads, evictions, and verifier work. Measure
accepted tokens together with exact bytes, speculative bytes, route timely
recall, amplification, and exposed storage wait.

The public Qwen3.8-27B model card reports strong H200 results for its declared
target and setup. Those numbers are evidence for that pair. They are not a
prior for Qwen4Exp on Apple Silicon.

## Compatibility gate

Before allowing a draft checkpoint to load, compare at least:

- exact target and tokenizer revisions;
- vocabulary identity and special-token mapping;
- target layer count and captured layer IDs;
- every hidden, embedding, selector, convolution, and output dimension;
- positional encoding and attention configuration;
- candidate count, block-size limits, and probability representation;
- verifier mode for greedy and sampling;
- cache ownership, rollback, and accepted-prefix semantics.

Any mismatch should reject the checkpoint. An explicit converter is acceptable
only when it describes a mathematically justified transformation and has its own
golden parity evidence. Slicing, padding, repeating, or renaming tensors is not
a converter.

## Recovery rule

When another DFlash2 checkpoint appears, ask two questions in order:

1. **Can these weights satisfy the exact target contract?** Usually only for the
   target and revision named by the release.
2. **Which protocol surfaces can we reuse?** Proposal shape, candidate path,
   verifier, accepted-prefix replay, rollback, trace schema, and metrics are the
   productive starting points.

This avoids both extremes: discarding a useful design because its weights do
not fit, and forcing foreign weights into a runtime because both repositories
contain the string `DFlash2`.

---

# AFN-019 — Hide SSD misses before the router knows they exist

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: hypothesis
- Phase: 8–11 performance architecture
- Estimated context: capsule ≈546 tokens; full note ≈5556
- Retrieve when: Retrieve this note when implementing Qwen4Exp expert or PLE SSD streaming, combining ExpertMajor v2 with DFlash, designing route prediction, or deciding whether I/O latency can be overlapped with Metal compute.
- Retrieval triggers: hide SSD expert latency behind compute; DFlash with ExpertMajor v2; predict MoE routes before the native router; PLE and expert I/O priority; adaptive speculative block size for an offloaded MoE; Metal I/O wavefront scheduler
- Decision: Treat SSD-heavy Qwen4Exp inference as a deadline-scheduled wavefront and test three lookahead tiers—exact PLE requests, pre-attention expert prediction, and DFlash route prediction—while keeping the native router and transactional target state authoritative.
- Invariant: Prediction may change which bytes arrive early, but it must never change native top-10 routing, target arithmetic, token acceptance, committed sequence state, or the priority of exact critical reads.
- Failure signature: The runtime reports cache hits or high SSD throughput while token latency still contains layer-by-layer wait bubbles, speculative expert reads starve PLE or exact misses, rejected DFlash suffixes amplify bytes per accepted token, or a predictor silently becomes the execution router.
- Minimal safe implementation: Use generation-owned I/O tickets with deadlines and confidence, separate exact and speculative queues, reconcile predictions against the native router, stream ExpertMajor components through bounded buffers, publish only accepted target state, and measure timely recall, useful bytes, amplification, and exposed wait rather than hit rate alone.
- Rejected shortcut: Prefetch every plausible expert, fix DFlash at its largest block, wait for all selected records before starting compute, use one global I/O queue, or treat Metal I/O capability and paper speedups as evidence that the M5 path is profitable.
- Evidence: Public Qwen4Exp geometry, established Hebrus storage and transaction contracts, published expert-prefetch research, DFlash 2 design evidence, and documented Metal I/O capabilities; no Qwen4Exp M5 performance checkpoint yet
- Evidence checkpoint: Hebrus c3759b5 plus Qwen/Qwen3.8-Flash-Next revision de4b8e4d43b917e7706784d8bb445c9af86a3540; hypothesis only
- Claim boundary: This note does not establish expert-predictor accuracy, an M5 SSD bandwidth figure, a winning DFlash block size, an ExpertMajor codec, end-to-end acceleration, quality parity, or Qwen4Exp runtime support. Every mechanism remains gated by physical traces and exact target-state tests.
- Prerequisites: AFN-002, AFN-003, AFN-010
- Related notes: AFN-015, AFN-016
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/hide-ssd-misses-before-router/
- Markdown URL: https://andreabor.io/agents/hide-ssd-misses-before-router/index.md


## Decision

An SSD-heavy Qwen4Exp runtime cannot be organized as:

```text
router -> discover misses -> read every record -> wait -> execute experts
```

That sequence places storage on the critical path of every layer. The working
hypothesis is to turn inference into a deadline-scheduled wavefront in which
known and predicted data movement begins at the earliest semantically valid
point, compute starts as soon as a useful subset is ready, and another request
or stage fills unavoidable bubbles.

Three kinds of lookahead must remain distinct:

1. **Exact lookahead:** token IDs determine PLE rows before the target reaches
   the PLE layer.
2. **Near-term predictive lookahead:** a lightweight pre-attention predictor
   estimates the current layer's eventual expert set while GDN or QSA still has
   useful work to execute.
3. **Long-range speculative lookahead:** a DFlash drafter may predict expert
   sets for several future tokens and target layers before verification starts.

Only the native Qwen4Exp router selects executed experts. Only the verified
target path commits state. Prediction is an I/O hint, never model semantics.

## Why overlap alone cannot defeat the byte roofline

The pinned Qwen4Exp text model has 48 sparse-MoE layers, 512 routed experts per
layer, top-10 routing, hidden width 2,560, and routed intermediate width 640.
Those fields are public in the
[`Qwen3.8-Flash-Next` configuration](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/config.json).

One routed expert contains:

```text
gate + up + down
= 2560*640 + 2560*640 + 640*2560
= 4,915,200 parameters
```

One token therefore activates:

```text
48 layers * 10 experts * 4,915,200
= 2,359,296,000 parameter uses
```

At ideal payload only, that is 562.5 MiB at two bits or 1.0986 GiB at four bits
per token when every selected layer-expert record is cold. Metadata, alignment,
staging, and read amplification are additional. AFN-002 establishes why the
existing ExpertMajor v2 container can describe this family without claiming a
qualified codec.

The lower bound is therefore:

```text
token time >= max(compute time, exact SSD bytes / sustained SSD bandwidth)
              + dependency bubbles that could not be hidden
```

Scheduling can remove the last term and overlap the first two. It cannot make
the storage term disappear. If a measured path needs 600 MiB of exact bytes per
output token, no cache-hit label or queue-depth chart can substitute for the
corresponding bandwidth.

This leads to the first promotion rule:

> Optimize exact bytes per accepted token before optimizing raw I/O throughput.

Useful mechanisms include low-bit storage, sequence-local cache retention,
deduplication of repeated routes across a verifier block, grouping token rows
by expert, and avoiding work on a rejected speculative suffix. Latency hiding
comes after those byte owners are visible.

## Two storage workloads, not one

Qwen4Exp exposes two unrelated access patterns.

### PLE: small, exact, latency-sensitive

PLE requests 16 logical rows per token. The row IDs are derived from known token
history, so a request becomes exact as soon as the token block is chosen. The
physical payload is small, but unrelated rows may occupy unrelated pages. This
makes PLE primarily an IOPS and tail-latency problem.

AFN-003 defines the fixed-page store and the separation between logical row
identity, physical page ownership, checksum, and future codec. The proposed
scheduler preserves that boundary:

```text
DFlash path selection or known autoregressive token
  -> exact PLE row IDs
  -> page locate, deduplicate, sort, and coalesce
  -> high-priority I/O ticket
  -> overlap with embedding and target layer 0
  -> dependency-scoped wait immediately before PLE use
```

An expert bulk wave must never starve this queue. A PLE ticket for a rejected
DFlash suffix does not commit token history, although a fully read and verified
page may remain as an ordinary cache entry.

### Routed experts: MiB-scale and bandwidth-sensitive

An expert miss transfers a complete gate/up/down record or named component
subranges. The requests are sparse in decode, but their aggregate payload is
large. They require a bulk queue, deeper concurrency, and a very different
replacement policy from PLE.

The cache score should estimate stall removed per byte:

```text
score(record) = P(use before eviction)
                * expected exposed latency avoided
                / resident bytes
```

Plain LRU is only a control. Modern decoder-only MoE routes may be too uniform
for popularity alone to deliver a high hit rate; the
[ProMoE paper](https://arxiv.org/abs/2410.22134) reports this as a specific
challenge and also observes that naive prefetch can hurt prefill when nearly
all experts are used.

## Hypothesis H1: exact PLE latency can mostly disappear

PLE has the cleanest lookahead because the address calculation is deterministic.
For DFlash, planning begins immediately after the final candidate path is known.
For ordinary decode, it begins when the current input token is known.

The target then provides a useful cover window:

```text
PLE I/O        |==================|
target         | embedding | layer 0 GR/GDN/router/MoE | PLE dependency
```

The queue should expose:

- request and unique rows;
- request and unique pages;
- cache hits, joined in-flight reads, and misses;
- logical bytes and physical bytes;
- issue-to-ready latency and ready-to-use slack;
- exposed wait at the PLE boundary;
- rejected-suffix bytes and later cache reuse.

**Falsification:** H1 fails if cold PLE wait remains material after exact early
issue, or if protecting PLE latency measurably degrades expert throughput more
than it saves in target stalls.

## Hypothesis H2: predict expert routes before attention, not after it

The native router sees the representation after the layer's attention/GDN path.
Waiting for it serializes selection, transfer, and expert execution. A separate
predictor can observe the representation available before attention and issue
prefetch hints while the real layer proceeds.

```text
wide residual
  |
  +-> small pre-attention predictor -> predicted top experts -> SSD tickets
  |
  +-> native GDN or QSA -> native router -> exact top-10
                                        -> reconcile predictions
```

The predictor must not replace or modify the native router. Prediction errors
affect transfer efficiency only. The
[SpecPrefetch paper](https://arxiv.org/abs/2607.24787) uses the same separation
between prefetch prediction and frozen execution routing. A
[preliminary pre-attention expert-prediction study](https://arxiv.org/abs/2511.10676)
reports that two lightweight projections can predict expert rankings with high
accuracy on several MoE models, including Qwen3-30B, but those figures are not
evidence for Qwen4Exp or Apple Silicon.

The implementation ladder should be deliberately cheap:

1. trace the overlap between the previous token's same-layer top-10 and the
   current exact top-10;
2. evaluate the existing native router on an earlier representation as a
   zero-training control;
3. if needed, train a low-rank correction or a shared low-rank predictor with
   small per-layer heads;
4. prefetch only candidates above a calibrated confidence/deadline threshold.

The predictor emits IDs into a tiny shared buffer. The runtime starts the normal
GDN/QSA command stream without a global wait; a completion handler or small
event transfers only the IDs needed to create file-offset tickets.

The primary metric is **timely recall**:

```text
timely recall = exact selected experts ready before their deadline
                / exact selected experts
```

Report it with precision, bytes prefetched, bytes used, eviction cost, and
exposed exact-miss wait. A predictor that reaches high recall by reading 30
experts for a top-10 router is not successful.

**Falsification:** H2 fails if predictor compute and queue boundaries consume
the lookahead window, if useful timely recall is low under a bounded overfetch
ratio, or if the extra reads increase end-to-end latency despite fewer reactive
misses.

## Hypothesis H3: DFlash can predict routes as well as tokens

DFlash conditions a lightweight parallel drafter on hidden features from the
target and proposes a whole token block in one pass. DFlash 2 adds a top-candidate
path selector and local dynamic convolutions; the released
[Qwen3.8-27B DFlash2 model card](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2#evaluation)
reports longer accepted blocks than the model's native MTP under its published
H200 evaluation.

Those weights are not compatible with Qwen4Exp. The reusable idea is to add a
route sidecar when a Qwen4Exp-specific drafter is eventually trained:

```text
DFlash hidden[position]
  +-> token LM head and candidate path
  +-> shared low-rank route adapter
        -> predicted experts[layer, position]
```

The route labels are already observable from frozen target traces. The sidecar
can therefore be trained as an auxiliary ranking task without changing target
routing. Once the final token path is selected, high-confidence route tickets
for later target layers can begin before verification.

This creates three priority classes:

```text
P0 exact critical      native-router miss or exact PLE dependency
P1 near-term predicted pre-attention predictor for an approaching layer
P2 speculative         DFlash route sidecar, next-token history, cache warming
```

P2 may consume only bandwidth and cache slots not required by P0/P1. When a
prediction is rejected, its ticket loses priority; completed records become
ordinary cache candidates rather than semantic state.

**Falsification:** H3 fails if draft features do not predict target routes with
useful precision, if a route sidecar materially increases draft time, or if
speculative route overfetch erases the accepted-token gain from DFlash.

## Hypothesis H4: ExpertMajor should stream as a wavefront, not a barrier

After exact top-10 routing, the target should not wait for every selected record
before starting any expert work. ExpertMajor v2 already gives each expert a
contiguous, checksummed record with explicit component extents. That record can
be consumed in bounded waves.

Gate and up are used together; down is needed only after the SwiGLU intermediate
exists. One candidate schedule is:

```text
SSD:  gate+up E0 | gate+up E1 | down E0 | gate+up E2 | down E1 | ...
GPU:               G/U E0      | G/U E1  | down E0    | G/U E2 | ...
```

Reads may complete out of order. Ready experts can execute immediately into
separate result slots, while the final weighted sum follows a deterministic
route-rank order. This preserves a stable reduction order without making one
slow read block nine ready experts.

The physical selector should test wave widths such as 2, 4, and all selected
experts. One expert per dispatch may waste more command overhead than it hides;
waiting for all ten may recreate the original barrier.

Apple's documentation describes
[Metal resource loading](https://developer.apple.com/documentation/metal/resource-loading)
and the
[`MTLIOCommandQueueDescriptor`](https://developer.apple.com/documentation/metal/mtliocommandqueuedescriptor):
I/O queues can load file data into Metal resources, support priority and
concurrent command limits, and synchronize with compute through shared events.
That makes MTLIO a legitimate candidate, not a proven winner. Bounded `pread`
into shared Metal-visible buffers remains the authoritative fallback until a
physical A/B proves otherwise.

**Falsification:** H4 fails if component splitting lowers effective SSD
throughput, command overhead dominates the saved tail, or concurrent SSD/GPU
traffic slows unified-memory compute enough to remove the overlap.

## Hypothesis H5: prefill and decode need different streaming modes

Large prefill tiles can route tokens to most experts in a layer. In that regime,
sparse random reads may cost more than one sequential layer extent. Decode and a
small DFlash verifier block remain sparse and should use selected records.

The runtime therefore needs an explicit crossover:

```text
macro prefill    stream the next full or near-full layer extent sequentially
micro prefill    read the sorted, coalesced expert union for the tile
decode           route-wise predictive cache and exact misses
DFlash verify    block union, deduplicated across token rows
```

During macro prefill, a two-layer ring can load layer `l+1` while the GPU
processes layer `l`. During decode, loading all 512 experts of a future layer is
not acceptable.

The crossover must be measured from exact unique-record fraction, coalesced
task count, physical bytes, and sustained bandwidth. It must not be a hardcoded
prompt-length guess.

**Falsification:** H5 fails if the layer-major sequential path cannot overlap
enough compute to justify its overread, or if the selected-record path remains
faster even when the route union approaches the full layer.

## Hypothesis H6: another request is the only general filler for a hard miss

Within one sequence, later-layer routes depend on earlier-layer results. Some
I/O bubble remains even with prediction. The general filler is independent work
from another admitted request:

```text
request A  GDN -> route -> SSD wait --------> experts
request B         draft -> QSA -> route -> SSD wait
request C                  GDN -> experts
GPU        A        B          C             A
```

The scheduler should maintain runnable stages rather than synchronizing a whole
batch at each layer. An I/O completion makes one request runnable; the GPU takes
the next ready operation. When several requests reach the same layer, their
route records may be merged and grouped by expert.

This is primarily a throughput mechanism. Single-request TPOT can regress if
the scheduler is unfair. Concurrency is also context-dependent: each session
owns GDN, QSA KV/index, PLE history, and transactional state. Long-context
admission may permit only one lane.

For agent workloads, tool and network waits create another low-priority window.
The runtime may retain or repopulate a sequence-local hot set from public model
state and recent route traces while generation is idle. This remains speculative
cache warming because future tool output is unknown.

**Falsification:** H6 fails if two to four lanes exceed the admitted memory
reserve, increase p95 TPOT beyond the declared service goal, or contend for SSD
and unified memory without improving useful GPU occupancy.

## Hypothesis H7: DFlash block size must follow I/O cost

A fixed maximum block is unsafe for an offloaded MoE. After drafting, the runtime
can estimate for each prefix length:

- expected acceptance;
- predicted and already-resident expert union;
- missing ExpertMajor bytes;
- PLE pages;
- deadline slack;
- rejected-suffix amplification risk.

For block size `B`, expected speculative speed is:

```text
speedup(B) = tau(B) * target_one_token_time
             / (draft(B) + verify(B) + exposed_IO(B) + commit(B))
```

If `U_B` is the average number of unique experts per layer in the verifier block,
the ideal two-bit routed payload per accepted output token is proportional to:

```text
48 * U_B * expert_record_bytes / tau(B)
```

Relative to autoregressive top-10 routing, the routed-byte break-even is:

```text
U_B < 10 * tau(B)
```

For example, a block of five with expected `tau=4` must average fewer than 40
unique experts per layer, rather than its worst-case 50, merely to avoid
increasing routed bytes per output token. This is a formula, not a prediction
of Qwen4Exp route locality.

The governor should truncate the already-produced proposal to the best measured
prefix. Shortening a speculative block does not weaken target correctness.

**Falsification:** H7 fails if route/acceptance estimation costs more than the
adaptation saves, or if a simple fixed block performs within noise across all
qualified cache and prompt conditions.

## Required scheduler contract

Every I/O ticket should name at least:

```text
store and immutable digest
generation and owner request
phase, layer, token/block mask
page or expert/component identity
exact vs predicted
confidence and priority
issue time and consumption deadline
compressed bytes and destination slot
lease count and completion sequence
```

The queue discipline is:

1. exact current PLE and exact native-router misses;
2. near-term high-confidence records with a deadline inside the current layer;
3. later-layer or later-token predictions that fit the remaining budget;
4. background cache warming only when no admitted request can use the bandwidth.

One global `waitUntilCompleted` is a failure. Compute waits on the smallest
event that guards the records it is about to consume. Cancellation releases
ownership exactly once. A stale generation may finish reading bytes but may
never publish them into a recycled cache slot.

AFN-010 remains authoritative for semantic state: GDN recurrence, QSA KV/index,
PLE history and convolution state, cache frontiers, residual streams, and logits
publish as one accepted target transaction. I/O cache entries are not sequence
state.

## Measurements that decide the hypothesis

Cache hit rate and SSD peak throughput are insufficient. Record:

### Storage

- exact, predicted, and rejected bytes;
- logical versus physical bytes;
- queue depth, task sizes, and coalescing;
- cold and warm bandwidth by read size;
- issue-to-ready p50/p95/p99;
- deadline slack or lateness;
- useful-prefetch ratio and overfetch amplification.

### Routing

- exact top-10 IDs by layer/token;
- overlap with previous token and within block sizes 2–5;
- unique experts per layer/block;
- predictor recall, precision, timely recall, and confidence calibration;
- cache eviction caused by false predictions.

### Compute and overlap

- GDN/QSA/router/shared/routed stage time;
- GPU idle time attributable to exact I/O;
- gate/up and down wave completion;
- unified-memory slowdown while I/O is active;
- exposed wait, not merely total I/O wall time.

### End to end

- prefill and decode separately;
- autoregressive, native MTP, and DFlash controls;
- block sizes 1, 2, 3, 4, and 5;
- one request and admitted multi-request lanes;
- cold process, cold store, natural warm store, and hot application cache;
- output identity, transaction digests, pressure, working set, and swap delta.

The central metric is:

```text
hidden fraction = 1 - exposed critical I/O wait / total exact I/O wall time
```

It must be interpreted together with exact bytes per output token. A 95% hidden
fraction achieved by doubling speculative reads is not automatically a win.

## Experiment order and promotion gates

### Gate A — physical storage envelope

Benchmark the actual ExpertMajor candidate record sizes and PLE pages on the
target M5 machine. Compare bounded `pread` into shared buffers with MTLIO at
multiple queue depths. Measure simultaneous GPU compute interference. Do not
fit the scheduler to advertised SSD bandwidth.

### Gate B — trace-only locality

Run the exact target and record route IDs without prediction. Calculate
cross-token overlap, block unions, per-layer skew, cache simulations, and the
byte break-even for block sizes 2–5. This gate can reject DFlash route prefetch
before training anything.

### Gate C — deterministic overlap

Implement exact PLE early issue, dependency-scoped waits, and ExpertMajor
component waves. Prove identical target output and committed state under every
partial completion and injected I/O failure.

### Gate D — pre-attention predictor

Start with zero-training controls, then train the smallest predictor justified
by the trace. Promote only if timely recall reduces end-to-end exposed wait under
a bounded byte-amplification and cache-eviction budget.

### Gate E — verifier-only oracle

Before training DFlash, feed recorded exact future tokens to a block verifier.
This gives perfect acceptance and zero draft cost: an upper bound on what any
drafter can recover after target verification, transactional state, expert I/O,
and PLE costs. If this oracle lacks headroom, stop.

### Gate F — DFlash and route sidecar

Train or obtain a target-specific drafter only after the oracle passes. Compare
token-only DFlash against DFlash plus route prediction. The sidecar wins only if
its added draft latency is smaller than the exact wait it removes.

### Gate G — adaptive and multi-request scheduling

Enable block adaptation and ready-request bubble filling independently, then
together. Use A/B/B/A runs and reject changes whose control drift exceeds the
repository performance protocol.

No gate promotes runtime support. Product exposure still requires the normal
artifact, quality, memory, physical-hardware, and long-run stability evidence.

## Restart capsule for a future agent

When resuming this hypothesis:

1. Read AFN-002 for ExpertMajor family geometry, AFN-003 for PLE store ownership,
   and AFN-010 for transactional publication.
2. Confirm the current Qwen4Exp artifact/profile and codec; do not reuse the
   ideal two-bit numbers as measured bytes.
3. Collect physical M5 storage traces before adding a predictor.
4. Separate PLE latency from expert bandwidth in every report.
5. Measure route union and exact bytes per accepted token before assuming DFlash
   is favorable.
6. Keep the native router authoritative and prediction discardable.
7. Prefer timely recall and exposed wait over cache hit rate.
8. Test component-wave execution without changing deterministic accumulation.
9. Use an oracle verifier to establish an upper bound before drafter training.
10. Leave this note labelled `hypothesis` until a public Hebrus checkpoint
    closes the relevant physical and semantic gates.

The durable architectural bet is narrow: prediction should move bytes, not
decisions. If the measured route locality and M5 overlap window are sufficient,
this pipeline can turn SSD stalls into background work. If they are not, the
same instrumentation will show that the remaining problem is unavoidable byte
volume rather than insufficient scheduling cleverness.

---

# AFN-016 — A sparse mask is not sparse attention

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 6 architecture
- Estimated context: capsule ≈413 tokens; full note ≈817
- Retrieve when: You are implementing sparse or selected attention and need to distinguish mathematically sparse visibility from bounded memory traffic and compute.
- Retrieval triggers: top-k mask still runs dense attention; selected keys but quadratic memory; prove physically sparse QSA; compact K/V gather and online softmax
- Decision: Define sparse attention by physical work: keep compact logical IDs, resolve only selected cache slots, gather only selected K/V rows, and run online softmax over the compact row.
- Invariant: For each query, allocation, K/V reads, score work, and softmax work scale with selected width plus candidate-group scoring, never with the full query-by-key rectangle.
- Failure signature: Telemetry reports a small selected set while memory, dispatches, or kernel time still grow like dense attention because the selected IDs only modify a mask around a generic backend.
- Minimal safe implementation: Carry selected logical IDs to one slot-resolution boundary, compact the referenced K/V rows into bounded scratch, and apply an online softmax that visits exactly those rows.
- Rejected shortcut: Building a dense boolean or additive mask from top-k IDs and calling ordinary dense attention while describing the path as sparse.
- Evidence: Pinned Transformers plus independent NumPy capture; host strict and ASan/UBSan lanes; actual-device Metal selection at 65,537, 100,000, and 262,144 visible tokens; compact gather and online attention; zero dense-mask byte accounting; private allocation, non-completion, and clean-completion gates
- Evidence checkpoint: Hebrus c3759b5
- Claim boundary: Hebrus c3759b5 proves the model-free F32 host and Metal sparse-QSA path through the frozen 262,144-token boundary; it does not register a production profile, family dispatch, artifact codec, full-checkpoint logits claim, or runtime support.
- Prerequisites: AFN-011
- Related notes: AFN-010, AFN-017, AFN-018
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/sparse-mask-is-not-sparse-attention/
- Markdown URL: https://andreabor.io/agents/sparse-mask-is-not-sparse-attention/index.md


## Decision

Sparse attention is a physical-work contract, not a property of a mask.

Qwen4Exp's indexer can reduce one query to at most 512 complete four-token
groups plus a raw tail of at most three tokens: 2,051 visible keys. That result
is mathematically sparse. It becomes an implementation win only if the backend
also avoids allocating, reading, scoring, or masking every other key.

The Phase 6 path verified at
[`c3759b5`](https://github.com/andreaborio/hebrus/commit/c3759b5b096afeb44c4db5768dee9a1de23a63a7)
is:

```text
logical history
  -> score complete groups in bounded tiles
  -> compact selected logical token IDs
  -> resolve selected IDs through the shared slot map
  -> gather only selected K/V rows
  -> online softmax over at most 2,051 rows
```

Only the gather crosses from logical identity into physical cache layout. The
selection stage does not need to know whether the KV cache wrapped, compacted,
or reused a physical slot.

## Why a sparse mask can still be dense

The public llama.cpp QSA integration at
[`c88c916`](https://github.com/ggml-org/llama.cpp/commit/c88c916) is valuable
semantic evidence: it pools complete groups, preserves the raw index-key cache,
compares selected indices directly, and catches the crucial
`sum(ReLU(dot_h))` ordering. Its graph then turns selected IDs into a mask around
the generic attention builder.

That is a valid engineering reference, but a mask alone cannot prove what every
backend physically traverses. The MLX-VLM Qwen4Exp path provides a similar
comparison point: correct selected visibility can still be represented as a
dense query-by-key boolean array before ordinary attention.

Hebrus therefore treats those paths as numeric and transition references, not
as proof of bounded accelerator work.

## What the checkpoint measures

Selected-count telemetry alone would be insufficient. The checkpoint freezes:

- candidate groups scored;
- selected groups and expanded token width;
- raw-key, K/V, pooled-key, workspace, and total allocation byte ownership;
- scratch sized as groups plus budget plus selected width, never queries by keys;
- compact gather and online-attention outputs against the host oracle;
- actual-device selection at 65,537, 100,000, and 262,144 visible tokens.

The key negative assertion is simple: the dense-mask allocation counter remains
zero.

## Why compact gather comes before fusion

Gathering into bounded contiguous scratch adds one explicit copy, but it makes
ownership and work visible. The selected row has a hard maximum, cache-slot
resolution happens once, and the simple attention kernel becomes directly
comparable to the dense oracle.

A later fused indirect-load kernel may remove scratch traffic. It should replace
the gather only after profiling, without changing the logical-ID contract or
weakening the zero-dense-allocation gate.

## Failure boundary

AFN-011 remains the bounded dense oracle through 2,051 visible keys.
[`c3759b5`](https://github.com/andreaborio/hebrus/commit/c3759b5b096afeb44c4db5768dee9a1de23a63a7)
closes the model-free sparse replacement, its byte accounting, long boundaries,
and publication rollback. It is linked but no production profile, family
dispatch, artifact codec, or runtime-support selector can reach it.

---

# AFN-017 — Top-k chooses the set; time chooses the order

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 6 architecture
- Estimated context: capsule ≈412 tokens; full note ≈705
- Retrieve when: You have a top-k selector whose mathematical result is a set, but downstream cache gathers and floating reductions require one canonical sequence.
- Retrieval triggers: top-k membership versus output order; sparse row differs below budget; heap order changes floating attention; deterministic selected ID sequence
- Decision: Use deterministic score order only to choose the top-k set, then emit the chosen groups in ascending logical position before expansion and tail append.
- Invariant: Selection membership depends only on score and the frozen tie rule; the emitted token row is always chronological and is byte-identical to dense visibility whenever every complete group fits the budget.
- Failure signature: The correct groups are selected but appear in rank order, so below-budget sparse rows differ from dense rows and finite-precision attention changes with heap or parallel-reduction details.
- Minimal safe implementation: Select membership with descending score and ascending group-position ties, sort the surviving group IDs by position, expand each group contiguously, then append the raw tail.
- Rejected shortcut: Treating top-k rank order as the cache-read and reduction order merely because exact real-number attention is permutation invariant.
- Evidence: Pinned Transformers and independent NumPy selected-ID capture; exact host full-raw/incremental equality; dense parity at 0–5 and 2,047–2,052 tokens; ascending-position tie tests; chunk/decode equality; actual-device Metal selection through 262,144 visible tokens
- Evidence checkpoint: Hebrus c3759b5
- Claim boundary: Hebrus c3759b5 proves the ordering rule for the model-free host and Metal QSA path, including dense parity, equal-score ties, full-raw/incremental identity, and long boundaries; it does not qualify a production profile or full-checkpoint inference.
- Prerequisites: AFN-011
- Related notes: AFN-016, AFN-018
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/top-k-set-time-order/
- Markdown URL: https://andreabor.io/agents/top-k-set-time-order/index.md


## Decision

Do not confuse a ranked set with an execution sequence.

QSA ranks complete four-token groups by score. Ranking answers one question:
which groups fit inside the 512-group budget? Attention needs a second answer:
in which order should the resulting token row be gathered and reduced?

Hebrus gives those questions different contracts:

```text
candidate scores
  -> choose set by (score descending, group position ascending)
  -> sort chosen group IDs by logical position
  -> expand every group to four consecutive token IDs
  -> append the incomplete raw tail
```

## The dense-parity property

Before the history exceeds the group budget, every complete group is selected.
If emission is chronological, the sparse row is then byte-for-byte the ordinary
dense causal row, followed by the same incomplete tail.

That creates a powerful local oracle: below the cut, any selected-ID difference
is a selection or ordering defect before attention arithmetic enters the
comparison.

Rank-order emission throws this property away. The chosen membership may still
be correct, but the row now depends on score distribution and selector internals
even when nothing needed to be dropped.

## Floating point makes order observable

In exact arithmetic, attention over the same key/value set is permutation
invariant. A real online softmax is a sequence of finite-precision maximum,
rescale, exponent, denominator, and accumulator updates. Changing the visit
order can change the final bits.

That does not make chronological order uniquely mathematical. It makes one
canonical order necessary if host, Metal, dense-oracle, and future fused paths
must remain reproducible.

Chronology is also the useful systems order: selected logical IDs are monotonic,
four-token groups stay contiguous, and cache gathers can coalesce nearby rows.

## Tie handling belongs to membership

Equal scores choose the smaller logical group position. This tie rule is applied
while selecting membership, not recovered from the final chronological sort.
Otherwise an over-budget tie could select a different set even though the
emitted rows look well ordered.

Tests must therefore distinguish:

- set membership under equal scores;
- chronological emission of that set;
- group-contiguous expansion;
- raw-tail placement;
- below-budget identity with the dense row.

## Failure boundary

[`c3759b5`](https://github.com/andreaborio/hebrus/commit/c3759b5b096afeb44c4db5768dee9a1de23a63a7)
verifies this separation in the model-free host and Metal paths. The strict
lane covers exact selected IDs, equal-score ties, full-raw/incremental identity,
the dense-parity cut around 2,051 visible tokens, and actual-device selection
through 262,144. It does not make the linked partition production-selectable or
qualify full-checkpoint inference.

---

# AFN-018 — A hole in cache history is not padding

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 6 architecture
- Estimated context: capsule ≈419 tokens; full note ≈691
- Retrieve when: You are grouping cached tokens by logical position and need to distinguish legal left padding or eviction from an internal gap that would silently drop attention history.
- Retrieval triggers: hole inside KV cache history; left padding versus missing token; group adjacent physical cache cells; sparse attention silently drops history
- Decision: Model each visible sequence as one contiguous logical-position run: permit an arbitrary first position for left padding, require every append to advance by one, and reject internal holes before sparse planning.
- Invariant: Every live position in a visible sequence run has exactly one predecessor except the first; complete groups and the raw tail partition that run without gaps or overlap.
- Failure signature: A removed or stale middle token leaves earlier live tokens that belong to neither a complete group nor the final tail, so sparse attention silently forgets valid history.
- Minimal safe implementation: Anchor grouping at the first live logical position, enforce successor appends, validate contiguity before planning, and rebuild pooled groups transactionally after lifecycle operations.
- Rejected shortcut: Treating a missing middle slot as ordinary padding, grouping adjacent physical cells, or skipping the gap and continuing group numbering.
- Evidence: Host strict and ASan/UBSan tests for left-padding anchors, internal-hole rejection, slot reuse, wrap eviction, reset, copy, fork, rewind, shift, remove, serialization and atomic restore; actual-device Metal rejection of groups crossing a position hole
- Evidence checkpoint: Hebrus c3759b5
- Claim boundary: Hebrus c3759b5 verifies the contiguous-run and transactional lifecycle contract in the model-free QSA cache and Metal selection path; it does not expose long QSA state through a production profile, artifact codec, or runtime-support admission.
- Prerequisites: AFN-010, AFN-011
- Related notes: AFN-016, AFN-017
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/cache-hole-is-not-padding/
- Markdown URL: https://andreabor.io/agents/cache-hole-is-not-padding/index.md


## Decision

Padding is outside a sequence. A hole is inside it. Sparse grouping cannot treat
them as the same thing.

For one logical sequence, the first real token may start at any absolute
position. That is enough to represent left padding or an older prefix that has
already been evicted. After that anchor, every visible token must be the exact
successor of the previous one.

```text
legal left padding:     [100, 101, 102, 103, 104]
legal shifted history:  [4096, 4097, 4098, 4099]
illegal internal hole:  [100, 101, 103, 104]
```

## Why QSA cannot skip the gap

QSA partitions the visible run into complete four-token groups plus one final
raw tail of zero to three tokens. An internal gap breaks that partition.

If position 102 is missing, the implementation cannot safely reinterpret 103 as
the third token of the earlier group: group identity and RoPE position would be
wrong. It also cannot discard 100 and 101 as if they were a tail, because the
tail is defined only at the current frontier.

Continuing after the hole therefore makes some live tokens disappear from
attention without an explicit policy decision.

## Logical identity before physical layout

The public llama.cpp hybrid index cache at
[`ff24f38`](https://github.com/ggml-org/llama.cpp/commit/ff24f38) establishes the
important ownership direction: index state follows the main cache's sequence,
copy, remove, and restore lifecycle. The QSA integration at
[`c88c916`](https://github.com/ggml-org/llama.cpp/commit/c88c916) likewise forms
blocks from positions rather than adjacent cache cells.

Hebrus keeps that semantic boundary and makes the missing-middle case explicit:
physical wrap, compaction, and slot reuse are allowed; a discontinuity in the
logical run is not.

## Reject early, rebuild transactionally

The cheapest time to preserve contiguity is append: the first position is free,
and every later reservation must target `frontier + 1`.

Lifecycle operations still need validation. Remove, rewind, shift, fork, copy,
and restore can change the derived pooled groups. They rebuild those groups in a
private candidate state and publish only when the complete run is valid.

If a caller intentionally needs segmented attention, that is a new semantic
model with explicit segments—not permission to reinterpret a hole.

## Failure boundary

[`c3759b5`](https://github.com/andreaborio/hebrus/commit/c3759b5b096afeb44c4db5768dee9a1de23a63a7)
verifies the fail-closed cache model across left-padding anchors, internal
holes, slot reuse, wrap eviction, rewind, shift, copy/fork, serialization,
atomic restore, and multiple sequences. The implementation remains a linked
model-free partition: no production profile, artifact codec, or runtime-support
admission selects it.

---

# AFN-015 — A phase handoff needs a decision delta

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: agent operations
- Estimated context: capsule ≈410 tokens; full note ≈686
- Retrieve when: You are closing an implementation phase or architecture checkpoint and need to preserve reusable reasoning without publishing progress logs or unverified claims.
- Retrieval triggers: phase handoff missing reusable decisions; agent repeats settled architecture research; publish technical choice after implementation; closeout checklist for autonomous agent
- Decision: Require Andrea-launched agents to perform a decision-delta review before closing each verified phase, publishing reusable evidence-bound choices or explicitly recording that no new note is warranted.
- Invariant: A verified phase handoff always states whether it produced a new reusable decision, and every published decision names public evidence plus the boundary of what that evidence does not prove.
- Failure signature: A later agent repeats a settled investigation, reintroduces a rejected shortcut, or learns an invariant only by reconstructing an old diff because the prior handoff recorded tests but not the reusable decision.
- Minimal safe implementation: At phase close, compare the work against the Field Notes selector, publish one compact note for each genuinely new evidence-backed decision, or record an explicit no-new-decision result in the handoff.
- Rejected shortcut: Publishing a diary of changed files, copying the whole handoff into the archive, or labelling dirty and local-only work as a verified public checkpoint.
- Evidence: Agent Field Notes schema v2 already separates fourteen Phase 0-5 decisions into retrieval capsules with explicit evidence and claim boundaries; the closeout rule makes that preservation step deterministic
- Evidence checkpoint: Agent Field Notes schema v2
- Claim boundary: This operator workflow does not bind external contributors, make incomplete work publishable, or replace repository source, tests, accepted decisions, support contracts, and QA gates.
- Prerequisites: AFN-009
- Related notes: AFN-014
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/phase-handoff-needs-decision-delta/
- Markdown URL: https://andreabor.io/agents/phase-handoff-needs-decision-delta/index.md


## Decision

An implementation phase is not fully handed off when the code and tests are
green. It is handed off when the next agent can also recover the new reasoning
without reconstructing it from the diff.

For agents explicitly launched by Andrea, phase close therefore includes one
mandatory question:

> Did this phase establish a reusable decision that the current Field Notes do
> not already preserve?

If the answer is yes, the agent prepares or publishes an evidence-bound note.
If the answer is no, the handoff says so explicitly. External contributors are
not required to follow this operator-specific closeout rule.

## What counts as a decision delta

A note is warranted when the phase establishes at least one reusable item:

- an invariant whose violation can produce plausible but wrong behavior;
- a failure signature that identifies the defect earlier next time;
- a minimal safe implementation or ownership rule;
- a shortcut that was attractive, tested, and rejected;
- an evidence gate that separates structural correctness from support;
- a claim boundary that prevents a narrow result from becoming a broad claim.

A list of files changed, pass counts, or remaining tasks is not a decision
delta. Those belong in the phase handoff.

## The publication boundary

The review happens at phase close, but publication still follows the evidence.
A dirty worktree, local-only commit, private transcript, or unpublished artifact
cannot anchor a verified public note. The candidate remains pending until a
public checkpoint exists.

This delay is useful. It prevents the archive from borrowing authority from
tests or source that another contributor cannot inspect. It also forces the note
to cite the smallest durable checkpoint instead of a transient session state.

## Closeout procedure

1. Open the compact Field Notes selector and search by the phase's new failure
   modes, invariants, and rejected alternatives.
2. Remove duplicates and status-only observations.
3. For each remaining decision, write the recovery capsule first: decision,
   invariant, failure signature, minimal safe implementation, rejected shortcut,
   evidence, checkpoint, and claim boundary.
4. Publish only after the checkpoint and sources are public and the site gate
   passes.
5. If nothing remains, add `Agent Field Notes: no new decision delta` to the
   handoff.

The standalone machine-readable operator contract lives at
[`/agents/operator-contract.txt`](/agents/operator-contract.txt).

## Failure boundary

This workflow improves retrieval discipline. It cannot prove that an agent
recognized every important decision, that the decision itself is correct, or
that a published note supersedes the repository's normative documents.

---

# AFN-014 — Do not call a contract vector an upstream capture

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 5
- Estimated context: capsule ≈366 tokens; full note ≈717
- Retrieve when: You are generating golden vectors from a framework that does not expose every cache image, token choice, tie policy, or synthetic control needed by the native test.
- Retrieval triggers: golden fixture generated by code under test; upstream framework does not expose cache state; per-array oracle provenance; synthetic control mislabeled as upstream
- Decision: Record provenance per fixture array and reserve the upstream-capture label for values actually emitted by pinned upstream code; label synthetic identities and state images as contract controls.
- Invariant: A fixture's provenance never claims that upstream produced an array which the collector constructed from the local contract.
- Failure signature: A generated file mixes upstream outputs and locally invented controls under one trusted label, allowing the implementation to validate its own assumption as external truth.
- Minimal safe implementation: Attach an origin to every array, execute pinned upstream for exposed values, independently recompute their math, and label synthetic token, state, tie, and overflow controls explicitly.
- Rejected shortcut: Calling the entire golden file upstream-backed because some arrays passed through the upstream runtime.
- Evidence: Graph fixtures execute pinned Transformers at 42ca97014c85d71a88ad60d55f08cb9fb4d26e2c, cross-check independent NumPy/scalar C, and label non-upstream arrays as contract-control in per-array provenance
- Evidence checkpoint: Hebrus 0626603
- Claim boundary: These tiny graph vectors prove primitive and transition semantics only; they do not contain checkpoint weights, full-layer intermediates, quantization error, or final model logits.
- Prerequisites: AFN-008
- Related notes: AFN-009, AFN-010
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/label-contract-control-vectors/
- Markdown URL: https://andreabor.io/agents/label-contract-control-vectors/index.md


## Decision

Fixture provenance belongs to each array, not merely to the file.

The Phase 5 collector executes the pinned Transformers Qwen4Exp implementation to capture values that upstream actually computes. An independent NumPy/scalar implementation checks the same math. That provides a strong oracle for exposed module outputs.

Some necessary test inputs and states do not have an upstream capture path. Token sequences are selected by the fixture author. Hashed PLE row identities and persistent cache images may need to be constructed so a specific transition can be tested. Tie, overflow, and rollback controls can also be contractual rather than upstream-authored.

Those arrays are useful—but they are not upstream captures.

## The self-approval trap

Suppose a local implementation defines a cache layout, generates an expected cache image using the same assumption, and stores both beside genuine upstream outputs. If the whole file is labelled “generated from Transformers,” a later agent may treat the cache image as independent evidence.

The implementation has then approved itself while borrowing upstream's authority.

## Provenance classes

A practical fixture can distinguish at least:

- **upstream capture** — emitted by the exact pinned implementation;
- **independent cross-check** — recomputed through a separate algorithm or language;
- **contract control** — deliberately selected input, state image, tie, overflow, or failure case;
- **derived metadata** — dimensions, offsets, or digests computed from already identified sources.

Each class answers a different question. None should silently inherit the authority of another.

## Regeneration rules

The networked collector may be needed to reproduce upstream arrays, but the normal repository test should stay offline. It can pin source revision, tool versions, source hash, per-array provenance, and aggregate fixture digests. Offline validation then detects drift without downloading a framework during every build.

The collector and the native implementation should not share the optimized helper being tested. Agreement is valuable only when likely defects are independent.

## Agent checklist

Before trusting a golden fixture, ask:

1. which exact function produced each expected array;
2. whether that function is upstream, independent, or part of the implementation;
3. which arrays are intentionally synthetic controls;
4. whether regeneration pins immutable source identity;
5. whether ordinary checks detect changes without network access;
6. what the fixture cannot say about full-model behavior.

## Failure boundary

Phase 5's fixture is intentionally tiny and model-free. It closes graph ordering and state-transition questions; it does not replace checkpoint-backed layer captures, final-logit parity, quality evaluation, or performance measurement.

---

# AFN-013 — Correctness code can exist without becoming a product feature

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 5
- Estimated context: capsule ≈394 tokens; full note ≈660
- Retrieve when: You need realistic integration and hardware tests for a new backend or model family without accidentally advertising, auto-selecting, or accepting it in production.
- Retrieval triggers: test backend accidentally exposed in production; compiled code mistaken for supported feature; staged model-family rollout; fail-closed product registration
- Decision: Link the reviewed correctness implementation behind an unreachable test seam, while keeping product admission fail-closed until artifact, memory, quality, and performance gates qualify an execution route.
- Invariant: Passing private host or accelerator correctness tests cannot make the model selectable through normal loader, server, downloader, or support metadata.
- Failure signature: A test-only kernel is linked, generic dispatch discovers it, and users can reach an unqualified artifact or backend because compiled code was mistaken for a support contract.
- Minimal safe implementation: Separate correctness symbols from product registration, require an absent physical profile or capability selector to reject normal execution, and test both the positive private seam and negative public route.
- Rejected shortcut: Using a hidden command-line flag, undocumented alias, or optimistic architecture match to expose code before the release evidence exists.
- Evidence: The host coordinator and Metal partition compile and pass model-free gates while remaining unselected; normal builds expose no physical profile, server route, downloader entry, or support claim
- Evidence checkpoint: Hebrus 0626603
- Claim boundary: Phase 5 proves tiny model-free host and physical-Metal semantics only; real checkpoint parity, codec choice, SSD streaming, memory fit, throughput, server, downloader, and release support remain disabled.
- Prerequisites: AFN-004, AFN-008
- Related notes: AFN-001, AFN-010, AFN-011
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/correctness-code-can-stay-unreachable/
- Markdown URL: https://andreabor.io/agents/correctness-code-can-stay-unreachable/index.md


## Decision

Compilation is not capability.

Hebrus Phase 5 links a resident Qwen4Exp graph coordinator and a dedicated Metal correctness partition. They can be exercised by focused tests on a real device. Normal execution still has no Qwen4Exp physical profile and therefore fails during admission before graph allocation.

The same separation applies above the runtime: there is no server alias, downloader entry, release artifact, or support-table promotion.

## Why linking the code is useful

Keeping all correctness work in an isolated prototype delays integration defects. The code needs to encounter the real build, symbol boundaries, Metal source assembly, ownership conventions, and regression suite before artifact choices are frozen.

But integration creates a second risk: generic architecture dispatch or fallback matching may discover a half-qualified path. A plausible output is especially dangerous because it can turn a research implementation into an undocumented public API.

## Two independent gates

The safe structure has both:

1. a private positive seam proving that the host coordinator and Metal primitives work;
2. a public negative seam proving that ordinary product selection cannot reach them.

The second test is not redundant. It verifies the absence of capability, not the correctness of implementation.

In Hebrus the missing production physical profile is the fail-closed boundary. Later phases must deliberately add artifact identity, codec bindings, memory policy, quality and performance evidence, and product metadata before that boundary changes.

## Language matters

Status labels should describe the reachable contract:

- **implemented** can mean code exists;
- **correctness fixture** means a bounded test seam passes;
- **pinned-not-supported** means normal users still cannot select it;
- **supported** requires the declared product and hardware gates.

Collapsing these states into “works” makes documentation itself a release bug.

## Agent checklist

When staging a new model family or accelerator path:

1. name the private entry point used by correctness tests;
2. name the public selector that must remain absent or rejecting;
3. test both independently;
4. keep server, downloader, aliases, and support tables unchanged;
5. state exactly which later evidence authorizes promotion;
6. rerun established families whenever shared dispatch changes.

## Failure boundary

The Phase 5 Metal fixture uses deterministic small F32 arrays. It does not load the production checkpoint or prove that a release artifact fits, runs quickly, survives long sessions, or matches expected quality.

---

# AFN-012 — An ubatch is not a sequence boundary

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 5
- Estimated context: capsule ≈373 tokens; full note ≈655
- Retrieve when: You are chunking a stateful convolution, hash-derived embedding, or recurrent prefill path and need batching changes to preserve exact sequence semantics.
- Retrieval triggers: prefill chunk size changes logits; dilated convolution state across batches; sequence state leaks between batch rows; EOS changes n-gram embedding history
- Decision: Own PLE history and dilated-convolution state by logical sequence, carrying it across ubatches and updating it in the frozen EOS order.
- Invariant: For the same token sequence, PLE row IDs, convolution outputs, persistent state, and final logits are identical across every valid ubatch partition, while separate sequences share no mutable history.
- Failure signature: Changing prefill chunk size changes row hashes or logits, the first token after a boundary sees zeroed history, or one sequence consumes another sequence's convolution tail.
- Minimal safe implementation: Store history and the full dilated receptive-field tail per sequence, advance them token by token inside each ubatch, and parity-test multiple partitions plus EOS and independent-sequence controls.
- Rejected shortcut: Resetting convolution or n-gram history at each scheduler chunk, or indexing persistent state by a transient batch row without generation ownership.
- Evidence: One-shot and 1/2/1-token ubatch plans produce identical PLE rows, convolution state, public digest, and logits; independent-sequence and current-EOS controls pass on host and Metal
- Evidence checkpoint: Hebrus 0626603
- Claim boundary: This proves model-free PLE state transitions with resident synthetic rows; it does not qualify the 320M-row production codec, page cache, SSD reads, or I/O overlap.
- Prerequisites: AFN-003, AFN-008
- Related notes: AFN-010, AFN-013
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/ubatch-is-not-sequence-boundary/
- Markdown URL: https://andreabor.io/agents/ubatch-is-not-sequence-boundary/index.md


## Decision

Scheduler chunks must not become model boundaries.

Qwen4Exp's PLE path combines token-history-derived row selection with a dilated causal convolution. Both operations depend on preceding tokens. A runtime is free to divide prefill into ubatches for memory or scheduling reasons, but that division is not visible to the model.

Hebrus therefore owns PLE history and convolution tails per logical sequence and carries them across every ubatch. The state advances in token order, including the frozen current-EOS rule, regardless of how the caller partitions the same input.

## The hidden receptive field

A causal convolution with kernel width four and dilation three needs nine preceding positions of state. Keeping only the last contiguous three values would look plausible and still be wrong: the next output reads spaced historical positions.

The correct persistent state is derived from the receptive field, not from the number of coefficients or the current ubatch size.

The history hash has a similar trap. Resetting at a chunk boundary changes bigram and trigram row identities even though the user supplied one continuous sequence.

## Evidence shape

Phase 5 compares the same token sequence executed in one shot and in a 1/2/1-token partition. It checks more than final logits:

- the generated PLE row identities;
- persistent convolution bytes;
- the digest of all public state owners;
- final output;
- behavior at the EOS transition;
- isolation between independent sequence state images.

Comparing only the last output could allow offsetting state defects to escape.

## Ownership rule

Persistent sequence state should be addressed by a stable slot plus generation, not by the temporary row number of the current ubatch. Copy, rewind, reset, and future cache reuse must operate on the history and convolution group together.

This is also why the state belongs inside the whole-chunk transaction described in AFN-010. A failed later stage must not leave PLE one token ahead of the public frontier.

## Agent checklist

For any chunked stateful operator:

1. derive the exact historical receptive field;
2. separate logical sequence ownership from batch placement;
3. specify update order around EOS and reset events;
4. compare internal state as well as output across partitions;
5. include a multi-sequence isolation case;
6. make failure preserve the pre-chunk snapshot.

## Failure boundary

These tests use tiny resident rows and deterministic F32 computation. Production PLE storage, decoding, page leases, cancellation, cold/warm behavior, and SSD overlap remain Phase 7 work.

---

# AFN-011 — A reference algorithm needs an expiry condition

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 5
- Estimated context: capsule ≈342 tokens; full note ≈572
- Retrieve when: You need a simple reference path to unlock graph bring-up without allowing its memory complexity or semantics to leak into advertised long-context support.
- Retrieval triggers: reference attention becomes production path; quadratic long-context allocation; dense versus sparse parity plan; boundary test at first unsupported context
- Decision: Use dense QSA only as a bounded correctness oracle through 2,051 visible keys and reject larger contexts until the native sparse algorithm is independently qualified.
- Invariant: No request above the declared dense-oracle ceiling can allocate or execute a dense Q-by-K QSA path.
- Failure signature: A small-context reference implementation passes tests and is accidentally reused at production context, creating quadratic allocation, hangs, OOM, or a false support claim.
- Minimal safe implementation: Encode the reference ceiling in geometry admission and backend dispatch, test the exact last accepted and first rejected values, and keep sparse support behind a later parity gate.
- Rejected shortcut: Calling the dense implementation temporary while leaving its context limit as documentation, an optional flag, or an unchecked allocation.
- Evidence: Host and physical-Metal boundary tests accept the inclusive 2,051-key dense case and reject 2,052 plus 262,143/262,144/262,145 before graph execution
- Evidence checkpoint: Hebrus 0626603
- Claim boundary: The dense oracle establishes small-context QSA semantics only; it does not implement sparse selection, long-state lifecycle, 100K context, or production memory behavior.
- Prerequisites: AFN-001, AFN-008
- Related notes: AFN-004, AFN-010, AFN-013
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/dense-reference-needs-expiry/
- Markdown URL: https://andreabor.io/agents/dense-reference-needs-expiry/index.md


## Decision

Every deliberately simple oracle needs a machine-enforced expiry condition.

Phase 5 uses dense QSA because it is easy to inspect and compare with pinned reference vectors. That is appropriate for correctness bring-up. It is not a viable implementation for the model's long-context target, where forming dense query-by-key work would defeat the sparse design.

Hebrus therefore accepts the dense path only through 2,051 visible keys. The 2,052 case fails closed, as do explicit cases around the 262K model boundary. Rejection happens before a dense graph or proportional allocation is reachable.

## Why the boundary belongs in code

“Temporary” is not a runtime property. Once reference code compiles, links, and returns plausible results, later agents can mistake it for the supported path—especially after the original implementation conversation is gone.

A prose warning also fails too late. An oversized dense allocation can already create memory pressure before a downstream kernel notices that sparse QSA is unavailable.

The limit must therefore live in geometry validation and dispatch, alongside tests for:

- the exact largest accepted input;
- the first rejected input;
- values near the advertised architectural maximum;
- unchanged output and state on rejection.

## What the dense path is for

The bounded implementation remains valuable. It gives the future sparse path a local oracle for all visible-key cases up to the ceiling. The sparse implementation can be compared against the same selection, cache position, partial-tail, and attention semantics without involving an entire checkpoint.

That is a stronger role than “slow fallback.” It is executable specification with a finite domain.

## Agent checklist

Before merging a reference algorithm, record:

1. the domain in which it is authoritative;
2. the first input it must reject;
3. the resource complexity that makes the boundary necessary;
4. the later implementation that must match it;
5. the gate required before removing or raising the ceiling.

## Failure boundary

Phase 5 does not claim sparse QSA or long-context support. Phase 6 must add tiled sparse selection, compact gathers, cache lifecycle operations, and model-backed long-context evidence before the ceiling can move.

---

# AFN-010 — One bank flip is enough: make the whole inference chunk transactional

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 5
- Estimated context: capsule ≈374 tokens; full note ≈655
- Retrieve when: You are implementing stateful inference, speculative work, or a multi-stage accelerator graph where one request can fail after mutating recurrent state, KV/index caches, routing state, or logits.
- Retrieval triggers: failed inference corrupts the next request; partial KV or recurrent-state commit; transactional GPU graph publication; chunk and token execution diverge
- Decision: Execute a complete token chunk against a private state bank and publish every persistent owner with one bank-index flip only after all host and Metal stages succeed.
- Invariant: Public frontier, logits, residuals, GDN state, QSA caches, PLE state, and routing state always describe the same successfully completed chunk.
- Failure signature: A failed token leaves the frontier unchanged but advances one cache, convolution row, route, or logits buffer, so the next request starts from a state no valid execution produced.
- Minimal safe implementation: Keep two complete state banks, copy the public snapshot into the private bank, run every fallible stage privately, validate finite values and accelerator status, then publish through one index swap.
- Rejected shortcut: Rolling back only the visibly changed buffer, or copying old bytes back after an accelerator command has partially completed.
- Evidence: Token, chunk, and interleaved multi-turn equivalence; failure injection after every graph stage; non-finite rejection; Metal allocation, command, and status rollback
- Evidence checkpoint: Hebrus 0626603
- Claim boundary: This proves transactional behavior for the Phase 5 model-free resident graph; it does not prove full-checkpoint logits, production codecs, SSD execution, memory fit, or product support.
- Prerequisites: AFN-001, AFN-008
- Related notes: AFN-012, AFN-013, AFN-014
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/whole-chunk-transaction/
- Markdown URL: https://andreabor.io/agents/whole-chunk-transaction/index.md


## Decision

Treat the whole chunk as the transaction—not each kernel, layer, or token-shaped buffer.

In the Phase 5 Hebrus graph, a chunk can update the four-stream residual, GDN convolution and recurrent matrices, QSA K/V and raw-index caches, PLE history and convolution state, selected routes, the public frontier, and logits. These values form one logical state image. Publishing them independently would permit combinations that the model never computed.

## Why local rollback is insufficient

A stage can fail after earlier stages have produced valid private results. The failure may be structural, a non-finite intermediate, an allocation denial, a Metal command that did not complete, or a nonzero device status word. Tracking which individual writes need undo logic turns every future state owner into another rollback branch.

That approach is fragile because ownership grows faster than the rollback checklist. It also asks the host to reconstruct whether device work partially executed.

## The publication model

Hebrus keeps two complete state banks:

1. one bank is the only public state;
2. the other becomes the transaction-private working image;
3. every graph stage mutates only the private image;
4. output remains caller-private while work is fallible;
5. finite checks, command completion, and status words must all pass;
6. one bank-index flip publishes the new frontier, state, and logits together.

Failure therefore discards an unpublished image. It does not attempt to reverse history.

## Evidence that matters

The useful test is not merely “a successful token produces expected logits.” Phase 5 injects failure after every graph stage and verifies two properties: the digest of all public state owners is unchanged, and the caller's output sentinel is untouched. Separate cases cover non-finite arithmetic and Metal allocation, command-completion, and status failures.

The same final state is then required for single-token, differently chunked, and interleaved multi-turn execution. Chunk size is a scheduling choice; it must not change model history.

## Reusable rule

When an inference operation updates more than one persistent owner, define one publication point before adding optimization. If the implementation cannot name that point, it does not yet have transactional state semantics.

## Failure boundary

This milestone uses a tiny deterministic correctness graph. It establishes the transaction seam, not production quality or speed. Real weights, quantized tensors, streamed PLE pages, sparse long-context QSA, and the product execution route remain later gates.

---

# AFN-009 — The recovery packet: enough context to continue, not enough to drown

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: agent operations
- Estimated context: capsule ≈352 tokens; full note ≈1391
- Retrieve when: You need to hand a long-running coding project to a new agent, recover after context compaction, or choose the smallest trustworthy document that can answer one implementation question.
- Retrieval triggers: agent handoff after long implementation; context compaction recovery; project knowledge consumes too many tokens; multiple agents repeat settled research
- Decision: Publish architectural knowledge in selector, recovery-capsule, full-note, and source-evidence tiers instead of handing every agent the whole project transcript.
- Invariant: The smallest retrieval tier must preserve the decision, invariant, failure signature, minimal safe path, rejected shortcut, evidence, and claim boundary.
- Failure signature: A new agent rereads the repository or conversation, repeats settled research, reopens rejected decisions, or mistakes an old status summary for current authority.
- Minimal safe implementation: Maintain a compact selector, a self-contained recovery capsule per decision, an optional full rationale, explicit note dependencies, and links to immutable evidence.
- Rejected shortcut: One ever-growing project summary, a raw transcript dump, or a vector search result without status, provenance, and supersession boundaries.
- Evidence: Applied schema-v2 capsules, graph edges, context estimates, build validation, and machine-readable indexes across AFN-001 through AFN-009
- Evidence checkpoint: Agent Field Notes schema v2
- Claim boundary: This structure reduces retrieval cost and repeated work; it cannot guarantee that an agent selects the right note or that the underlying technical decision is correct.
- Prerequisites: none
- Related notes: AFN-001, AFN-008
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/recovery-packets-for-autonomous-agents/
- Markdown URL: https://andreabor.io/agents/recovery-packets-for-autonomous-agents/index.md


Long-running coding work accumulates two kinds of context:

1. facts needed to continue safely;
2. history explaining how the team discovered those facts.

They are not equally valuable at retrieval time.

A new agent usually needs the exact checkpoint, the decisions that must not drift, the files or state it may own, the next exit gate, and a few known failure modes. Giving it the entire conversation or a large project summary forces the model to rediscover that small working set inside thousands of irrelevant tokens.

The recovery packet is a layered alternative.

## Four retrieval tiers

The Agent Field Notes corpus now exposes four deliberate costs:

```text
selector       find the relevant note from triggers and status
capsule        recover the operational decision in roughly 150–300 tokens
full note      load rationale, alternatives, formulas, and examples
source         inspect code, tests, commits, or primary upstream evidence
```

An agent begins at the cheapest tier. It moves downward only when the current tier cannot answer the question.

`llms.txt` is a selector, not a compressed encyclopedia. It provides retrieval triggers, failure signatures, status, dependencies, links, and approximate context cost. The individual Markdown page begins with the recovery capsule. `llms-full.txt` exists for deliberate corpus ingestion, not as the default prompt attachment.

## What belongs in the capsule

A capsule is not an abstract. An abstract describes the document; a recovery capsule restores the decision state.

It contains:

- **Retrieve when** — the task situation that makes the note relevant.
- **Decision** — the selected architectural action.
- **Invariant** — the property future work may not violate.
- **Failure signature** — what the mistake looks like in practice.
- **Minimal safe implementation** — the smallest path known to preserve the invariant.
- **Rejected shortcut** — the tempting approach that already failed review.
- **Evidence** — the test or checkpoint supporting the decision.
- **Claim boundary** — what the evidence does not authorize.

Removing any one of these creates a predictable failure. Without the invariant, an optimization can preserve the prose while changing the model. Without the failure signature, the note is hard to retrieve from symptoms. Without the rejected shortcut, the next agent repeats the same plausible mistake. Without the claim boundary, a structural test becomes a support claim.

## Store decisions, not progress narration

“Metal lane is 80% complete” is useful for coordination and nearly useless as durable memory. It becomes false as soon as the next commit lands.

“Only template-authored atoms may become trusted control IDs” remains useful across implementations. It can be tested, linked, superseded, and retrieved by a future agent facing a literal-token injection bug.

Durable notes should therefore be created for:

- architectural choices;
- invariants and ownership;
- rejected alternatives;
- failure and rollback semantics;
- measured thresholds;
- upstream behavior imported as local regression evidence.

Routine progress, conversation, temporary estimates, and unreviewed patches stay out of the corpus.

## Make dependencies explicit

Notes form a graph, not a timeline.

`prerequisites` tells an agent which decisions must already hold. `related_notes` offers optional context. `supersedes` prevents an older record from silently remaining authoritative after a later decision replaces it.

This is more reliable than relying on publication order. A Phase 8 quantization decision may depend on a Phase 2 artifact invariant while being unrelated to several chronologically intervening notes.

## Estimate context before loading it

The machine index includes approximate token counts for both capsule and full note. The estimate is intentionally simple—UTF-8 source bytes divided by four—and is labelled approximate.

Its purpose is not billing precision. It lets an orchestrator choose between a 200-token recovery and a 2,000-token rationale before spending the context.

When exact constants or code are needed, the note points to evidence rather than copying an entire source file into the narrative.

## A bounded handoff workflow

At a safe checkpoint:

1. record the immutable commit or evidence revision;
2. list decisions that constrain the next task;
3. state byte, file, cache, or state ownership;
4. name the one active exit gate;
5. record known failure signatures and stop conditions;
6. link only the minimum source files or upstream diffs;
7. mark explicitly what remains unsupported;
8. let the receiving agent request deeper context only when blocked.

The receiving agent should be able to answer three questions from the capsule alone:

```text
What may I change?
What must remain true?
What evidence proves I am done?
```

If it cannot, the handoff is incomplete. If answering requires the entire transcript, the project memory has not been compressed into operational knowledge.

## The general rule

Context should be progressive and evidence-linked.

Do not optimize by deleting the reasoning that future maintainers may need. Move it to a deeper retrieval tier. Do not optimize by pasting everything into every agent. Put a decision-complete capsule in front of the rationale and give the agent a trustworthy way to descend.

The goal is not the fewest possible tokens. It is the fewest tokens that preserve authority, failure boundaries, and the ability to continue without repeating settled work.


---

# AFN-008 — Close the text pipeline before asking the model

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 4
- Estimated context: capsule ≈307 tokens; full note ≈1196
- Retrieve when: You are sequencing a new model-family integration and need a test hierarchy that isolates text-pipeline failures before checkpoint-backed graph parity.
- Retrieval triggers: full-model mismatch has too many possible owners; tokenizer or template fixture design; mutation-testing a model contract
- Decision: Make tokenizer and chat behavior independently executable and mutation-tested before using full-model output as the debugging oracle.
- Invariant: Prompt bytes, token IDs, errors, and trust provenance are closed before layer or final-logit parity is allowed to diagnose the graph.
- Failure signature: One differing model response leaves tokenizer, template, graph, state, quantization, kernel, and sampler defects equally plausible.
- Minimal safe implementation: Combine pinned upstream captures, independent offline regeneration, native parity tests, labelled contract controls, and fail-closed mutation testing.
- Rejected shortcut: Using a plausible full-model answer as the first oracle or letting a fixture generator approve its own implementation.
- Evidence: 59 tokenizer cases, six decode controls, 39 chat cases, 153/153 contract mutations rejected, sanitizer and full model-free regression gates
- Evidence checkpoint: Hebrus fcfd5de
- Claim boundary: These model-free gates do not contain checkpoint weights or prove layer, logits, quality, Metal, SSD, memory, or performance parity.
- Prerequisites: AFN-001
- Related notes: AFN-005, AFN-006, AFN-007, AFN-009, AFN-010, AFN-014
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/oracles-before-full-model-parity/
- Markdown URL: https://andreabor.io/agents/oracles-before-full-model-parity/index.md


Full-model output is a terrible first oracle for a new runtime.

If the answer differs from Transformers, the cause may be tokenization, chat rendering, special IDs, graph math, state layout, quantization, kernels, cache transitions, or sampling. One failing string tells an agent almost nothing about which layer owns the defect.

Phase 4 of the Qwen4Exp bring-up closes the text pipeline while execution is still disabled.

## The evidence stack

The tokenizer gate contains:

```text
59 pinned text encode/decode cases
 6 explicit decode-boundary controls
33 exact added-token identities
 1 closed effective-vocabulary/physical-logits boundary
```

The chat gate contains:

```text
34 pinned Transformers render or error captures
 5 Hebrus text-only contract rejections
39 total cases
```

Together they cover default and explicit reasoning effort, disabled thinking, preservation of prior reasoning, system-role cardinality, tools, grouped tool results, insertion-ordered UTF-8 JSON, literal control tokens, structured-media rejection, Unicode trimming, invalid inputs, and transactional output preservation.

The wider frozen Qwen4Exp contract now rejects 153 out of 153 deliberate mutations. That total spans identity, source pins, inventory, tensor roles, layer types, derived constants, PLE geometry, expert-store geometry, graph facts, admission, tokenizer, chat, and licensing boundaries. It is not presented as “153 chat tests”; it is the mutation count for the complete contract after Phase 4.

## Four different authorities

The oracle hierarchy keeps four kinds of evidence distinct:

1. **Pinned upstream captures** establish what the selected public Transformers revision does.
2. **Independent regeneration** checks that the fixture is not merely a dump produced by the implementation under test.
3. **Native C tests** establish parity and failure semantics in Hebrus.
4. **Contract controls** define behavior that upstream does not own, such as trust provenance, text-only rejection, rollback, and deterministic boundaries.

A fixture records the origin of each case. Contract controls are not labelled as upstream output, and upstream behavior is not silently replaced because a native approximation is easier.

## Why model-free first

The tokenizer and renderer need no checkpoint allocation, Metal device, SSD cache, or 64 GB machine. Their gate is fast enough to run under sanitizers and in the normal regression suite.

This changes the debugging tree for the first resident graph:

```text
prompt bytes differ?       -> chat oracle
token IDs differ?          -> tokenizer oracle
embedding input differs?   -> graph boundary
layer output differs?      -> primitive or graph implementation
final logits differ?       -> graph, state, or numeric policy
sample differs only?       -> sampler boundary
```

Without the first two closed nodes, every later mismatch remains ambiguous.

## Mutation tests are stronger than happy paths

A positive fixture proves one accepted object works. A mutation battery proves the validator notices which facts make it that object.

Phase 4 adds mutations for tokenizer authority, normalization, regex, added-token mappings, valid vocabulary size, physical width, stop IDs, template defaults, trimming, segment kinds, structured-media policy, oracle counts, and fixture digests.

The important property is fail-closed behavior. A changed digest cannot be accepted because the visible cases still happen to pass. A changed count cannot be ignored because the array looks plausible. A flattened trust boundary cannot hide behind byte-identical rendering.

## What the checkpoint does not claim

No weights are loaded by these tests. They do not establish:

- BF16 or quantized layer parity;
- the resident Metal graph;
- sparse QSA kernels;
- PLE SSD streaming;
- expert codec quality;
- long-context state behavior;
- memory admission;
- tokens per second;
- product or server support.

The family remains `pinned-not-supported`. The value of Phase 4 is not that the model can answer. It is that, when the first graph answers incorrectly, two entire classes of causes have already been removed from the search space.

## General rule for agents

Before building a full-model oracle:

- close prompt rendering and tokenization independently;
- pin upstream implementations and serialized assets;
- preserve exact bytes, IDs, errors, and trust metadata;
- give policy decisions their own contract-labelled cases;
- regenerate fixtures independently and offline;
- mutate every identity field and semantic boundary;
- keep support status false until later gates earn it.

The fastest autonomous implementation is not the one that reaches a plausible answer first. It is the one that makes each wrong answer cheap to localize.

---

# AFN-007 — Unicode is part of the model contract

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 4
- Estimated context: capsule ≈345 tokens; full note ≈1055
- Retrieve when: You are porting a Python tokenizer or chat template into a native runtime and need to identify Unicode behaviors that disappear under reconstructed regexes, ASCII trimming, or locale-sensitive functions.
- Retrieval triggers: Python tokenizer ported to native code; combining-mark token drift; Jinja or Python trim mismatch
- Decision: Treat tokenizer.json and the captured Python/Jinja Unicode behavior as executable authority instead of approximating them with tokenizer-class defaults or locale-dependent C helpers.
- Invariant: The serialized tokenizer backend and pinned Python/Jinja byte behavior outrank convenience constructors and host-language whitespace guesses.
- Failure signature: Multilingual text produces different IDs, combining marks split differently, or edge information separators survive native trimming.
- Minimal safe implementation: Pin tokenizer.json, normalization and regex; trim complete Unicode code points with captured Python semantics; independently regenerate exact bytes and IDs.
- Rejected shortcut: Reconstructing with AutoTokenizer defaults or replacing Python/Jinja behavior with byte-wise isspace-style helpers.
- Evidence: Pinned tokenizer.json backend, Devanagari divergence capture, Unicode trimming fixture, independent offline regeneration, and C99 parity tests
- Evidence checkpoint: Hebrus fcfd5de
- Claim boundary: The oracle closes the tested tokenizer and template semantics for the pinned revision; it is not a general Unicode library or a promise about future upstream revisions.
- Prerequisites: AFN-001
- Related notes: AFN-006, AFN-008
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/unicode-is-part-of-the-model-contract/
- Markdown URL: https://andreabor.io/agents/unicode-is-part-of-the-model-contract/index.md


Unicode bugs in an LLM runtime rarely announce themselves as Unicode bugs. They appear as different token IDs, a shifted prompt boundary, or a model that is inexplicably worse in one language.

Qwen4Exp exposed two separate traps during the model-free text bring-up.

## The tokenizer class was not authoritative

The checkpoint ships an exact `tokenizer.json`. Loading that file through the pinned Transformers `TokenizersBackend` produces a pre-tokenizer expression that includes combining marks:

```text
[\p{L}\p{M}]+
```

Reconstructing the tokenizer through the pinned Qwen2 `AutoTokenizer` path omitted `\p{M}` from the relevant expression. That changes how Devanagari letters and their combining marks are grouped before BPE.

Both paths look official. Both load the same repository. Only one matches the serialized tokenizer artifact.

Hebrus therefore freezes the file-backed backend as authority and records the reconstruction divergence as a negative fact:

```text
authority                    = tokenizer.json
AutoTokenizer authoritative = false
normalization                = NFC
```

The runtime implements the exact expression rather than a nearby “GPT-style” regex.

## Trimming was wider than C whitespace

The chat template uses Jinja trimming, which delegates to Python string behavior. Python's `str.strip()` covers Unicode whitespace and also removes the ASCII information separators U+001C through U+001F at the edges.

Common native approximations are insufficient:

- `isspace()` is locale-sensitive and operates on byte-sized values;
- trimming only ASCII space, tab, CR, and LF misses valid cases;
- a Unicode White_Space table alone misses Python's information separators;
- byte-wise reverse scanning can split a multibyte code point.

The native renderer decodes UTF-8 boundaries, applies an explicit Python-compatible predicate, and trims only complete code points from the two edges.

The fixture combines Unicode whitespace and information separators in one case so that replacing the predicate with a conventional helper changes the rendered bytes.

## Why “valid UTF-8” is not enough

Two runtimes can accept the same input as valid UTF-8 and still disagree about:

- normalization;
- pre-tokenizer grouping;
- which code points count as leading or trailing whitespace;
- replacement behavior for invalid sequences;
- byte fallback;
- embedded NUL handling.

Every disagreement can change the prompt token sequence. Full-model parity cannot repair a text pipeline that feeds different IDs into layer zero.

The tokenizer oracle therefore includes multilingual text, combining marks, invalid UTF-8 replacement, embedded NUL, special-token literals, and raw-versus-trusted control behavior. The chat oracle separately freezes trimming and serialized output bytes.

## Capture upstream, check offline

The golden vectors are captured with pinned versions of Python, Transformers, tokenizers, Jinja, and the checkpoint files. Provenance records the public source hashes and package versions.

Normal testing does not depend on the network or those packages. An independent standard-library transcription regenerates the canonical fixture offline, and the C implementation must match the frozen outputs.

This gives two useful failure modes:

- upstream capture drift says the external authority changed;
- offline or C drift says the local transcription changed.

Conflating the two would make a fixture refresh capable of approving its own bug.

## General rule for agents

When porting a tokenizer or template across languages:

1. identify the serialized artifact that actually defines behavior;
2. record any disagreement with convenience constructors;
3. freeze normalization, regex, trimming, and invalid-byte behavior explicitly;
4. include scripts and languages that require combining marks;
5. include code points at the edge of each host language's whitespace rules;
6. compare token IDs and exact rendered bytes, not only decoded text;
7. keep the upstream capture path separate from the offline checker.

“Unicode support” is not one checkbox. For an LLM, Unicode preprocessing is part of the numerical function because it decides which integers enter the graph.

---

# AFN-006 — Rendered chat is not a flat string: preserve provenance

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 4
- Estimated context: capsule ≈315 tokens; full note ≈1045
- Retrieve when: You are implementing a chat template with special tokens and need to prevent user, system, or tool text from being reinterpreted as template-authored control syntax.
- Retrieval triggers: chat template mixes controls with client text; literal special-token injection; tool output contains protocol markers
- Decision: Render chat as one byte stream plus contiguous DATA and TRUSTED_CONTROL segments, then tokenize every segment according to its provenance.
- Invariant: Only template-authored atoms can become trusted control IDs; caller data remains data even when its bytes spell the same token.
- Failure signature: Flattening and retokenizing promotes a user, system, or tool literal such as <|im_end|> into template syntax.
- Minimal safe implementation: Return one byte stream plus complete contiguous provenance segments, validate coverage, and tokenize DATA and TRUSTED_CONTROL independently and transactionally.
- Rejected shortcut: Escaping suspicious characters or flattening the rendered transcript and trying to reconstruct trust from its spelling.
- Evidence: 39-case chat oracle, literal-control adversarial cases, segmented tokenizer adapter, transactional failure tests, and the Phase 4 model-free gate
- Evidence checkpoint: Hebrus fcfd5de
- Claim boundary: The segmentation boundary prevents control-token promotion in the text pipeline; it does not sanitize tool behavior or grant runtime support to the model.
- Prerequisites: AFN-001
- Related notes: AFN-007, AFN-008
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/chat-rendering-needs-provenance/
- Markdown URL: https://andreabor.io/agents/chat-rendering-needs-provenance/index.md


A rendered chat transcript looks like a string. Treating it as only a string is a security bug.

Consider the bytes:

```text
<|im_end|>
```

When the template emits them, they delimit a message. When a user types the same bytes inside a question, they are ordinary content. The spelling is identical; the authority is not.

If a renderer concatenates everything and then runs one trusted tokenizer pass, client text can acquire the semantics of template syntax. The runtime has erased the only information that could prevent promotion.

## The representation

The Qwen4Exp chat renderer returns:

1. one ordered UTF-8 byte stream for inspection and logging;
2. a complete, gap-free sequence of segments over those bytes;
3. one kind for each segment: `DATA` or `TRUSTED_CONTROL`.

Adjacent segments of the same kind are merged. Every segment must be non-empty, begin exactly where the previous one ends, and collectively cover the full stream.

Template-authored markers such as these are controls:

```text
<|im_start|>
<|im_end|>
<think>
</think>
<tool_call>
</tool_call>
<tool_response>
</tool_response>
```

Conversation content remains data, even if it contains any of those literal spellings.

## Tokenize the segments, not the flattened bytes

The adapter validates the segment geometry first. It then tokenizes each contiguous segment independently:

- `DATA` goes through the ordinary NFC, pre-tokenizer, and BPE path;
- `TRUSTED_CONTROL` may resolve exact special-token atoms.

The resulting token arrays are appended transactionally. If a segment is malformed, a control is unknown, UTF-8 handling fails, or allocation fails, the caller's prior token output is preserved.

Flattening the bytes and tokenizing them again as trusted is forbidden. The rendered string is useful for observability; it is not a substitute for the provenance stream.

## Why escaping is the wrong abstraction

One possible response is to escape `<`, `>`, or the pipe character in client text. That changes user content, code snippets, shell output, and tool results. It also assumes every dangerous control can be recognized with a lexical filter.

Provenance is stronger because it does not depend on guessing whether data looks suspicious. Data stays data by construction.

This is especially important for tool output. A tool may legitimately return source code or model text containing closing tags and chat markers. The renderer must preserve the bytes without granting them template authority.

## Adversarial cases

The Phase 4 oracle includes literal controls in:

- user content;
- system content;
- tool results;
- structured text that spells a media token.

The tests assert that the client occurrence remains `DATA` while the actual template delimiter is `TRUSTED_CONTROL`. The tokenizer integration then proves that only the latter becomes a special ID.

Structured image, image-URL, and video parts are rejected before rendering because the current artifact is text-only. A literal `<|image_pad|>` inside normal text remains legal data; it does not smuggle a media part into the request.

## Transactionality matters here too

Chat rendering allocates a replacement output and commits only after full validation and rendering. An invalid request does not partially overwrite a previously valid transcript or segment array.

This turns errors such as a second system message, an unsupported role, malformed tool arguments, or structured media into clean failure states. Downstream code never receives a half-rendered trust map.

## General rule for agents

Whenever a protocol mixes control syntax and caller-controlled text:

- carry provenance beside bytes until semantic parsing is complete;
- do not recover provenance by rescanning the flattened representation;
- validate complete, contiguous ownership of the byte stream;
- make trust promotion an explicit API, not a tokenizer heuristic;
- preserve caller outputs on rejection;
- test identical spellings under both trusted and untrusted origins.

This is not a Qwen-specific trick. It is a general rule for structured prompts: syntax is not defined by appearance alone. It is defined by who was allowed to create it.

---

# AFN-005 — A logits matrix is not always the vocabulary

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 4
- Estimated context: capsule ≈292 tokens; full note ≈937
- Retrieve when: You are integrating a checkpoint whose output projection is padded beyond the tokenizer vocabulary and need exact rules for sampling, decoding, stopping, and artifact admission.
- Retrieval triggers: output rows exceed tokenizer vocabulary; padded logits produce invalid IDs; sampling and decode boundaries disagree
- Decision: Represent the tokenizer domain and the physical logits width as separate constants, masking every unassigned output row before sampling.
- Invariant: The graph admits 248,320 physical rows while sampling remains restricted to token IDs in [0, 248077).
- Failure signature: An unassigned padded row wins selection and is clamped, decoded ambiguously, or misinterpreted as a stop token.
- Minimal safe implementation: Store physical width and semantic vocabulary separately, mask every invalid row before selection, and test with invalid rows numerically dominant.
- Rejected shortcut: Using one n_vocab value to answer tensor shape, sampling validity, decode behavior, and stop semantics.
- Evidence: Pinned tokenizer backend, 59 encode/decode cases, six boundary controls, sampler-boundary test, and the Phase 4 model-free gate
- Evidence checkpoint: Hebrus fcfd5de
- Claim boundary: This closes the text-domain semantics only; it does not prove full-model logits parity, generation quality, or runtime support.
- Prerequisites: AFN-001
- Related notes: AFN-004, AFN-008
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/physical-logits-are-not-the-vocabulary/
- Markdown URL: https://andreabor.io/agents/physical-logits-are-not-the-vocabulary/index.md


The number of rows in an output projection is a physical shape. The number of tokens a tokenizer can produce is a semantic domain. They are often equal, which makes it easy for runtime code to treat them as the same fact.

For Qwen3.8-Flash-Next, they are not equal:

```text
physical output rows       = 248,320
effective tokenizer IDs    = 248,077
last valid token ID        = 248,076
unassigned physical rows   = [248,077, 248,320)
unassigned row count       = 243
```

Hebrus keeps both values in the model contract. One may not be derived from the other.

## Why the extra rows are dangerous

The padded rows are real matrix rows. A graph must produce values for them, and a loader must admit the physical tensor shape. But those rows do not name tokens.

If a sampler uses the physical width as its candidate count, an unassigned row can win argmax or enter top-k. The runtime then has no legitimate token to append. Silently clamping the ID, decoding it as replacement text, or treating it as EOS would all create a new model behavior.

The sampling rule is therefore explicit:

```text
sample IDs in [0, 248077)
mask logits IDs in [248077, 248320)
```

Masking happens before any selection policy. It is not a decoder fallback after an invalid ID has already won.

## Decode and stop are separate policies

The pinned tokenizer backend decodes the 243 unassigned physical IDs as empty. Hebrus preserves that boundary behavior for defensive decoding, but decode-silent does not mean sample-valid.

The stop set is also exact:

```text
248044  <|endoftext|>
248046  <|im_end|>
```

`<|im_start|>` at 248045 is not a stop token. The model configuration's EOS declaration and the tokenizer's chat EOS are not collapsed into one guessed value; the runtime records the effective stop set deliberately.

This creates four distinct questions for any ID:

1. Does a physical output row exist?
2. Is the ID valid for sampling?
3. Does the tokenizer have decode semantics for it?
4. Does generation stop after it?

Code that answers all four from a single `n_vocab` integer is underspecified for this checkpoint.

## The test that matters

The boundary test gives every invalid physical row a higher score than every valid row. The last valid ID, 248076, is still required to win after masking.

That test catches the implementation error directly. A fixture that merely checks constants would not prove the sampler uses the semantic domain.

Decode controls independently cover the first and last unassigned rows, a mixed valid/unassigned sequence, negative IDs, and values beyond the physical width. This prevents a future cleanup from accidentally turning a defensive decode behavior into sampler authority.

## Admission implications

Artifact validation needs both equalities:

- the output tensor must have 248,320 rows;
- the tokenizer payload must define exactly 248,077 effective IDs.

An artifact with a 248,077-row output matrix is not a compact equivalent. An artifact with 248,320 declared tokenizer IDs is not the same tokenizer. Either change requires a distinct physical profile and new evidence.

## General rule for agents

When a checkpoint uses a padded output projection:

- name physical and semantic dimensions separately;
- validate each against its own source of authority;
- mask unassigned rows before top-k, top-p, argmax, or temperature logic;
- make invalid decode behavior explicit but never use it to authorize sampling;
- freeze the stop set independently from BOS, PAD, and generic EOS fields;
- test with invalid rows deliberately made numerically dominant.

The broader lesson is simple: tensor geometry tells a runtime what it must compute. A tokenizer contract tells it what those results are allowed to mean.

---

# AFN-001 — Porting Qwen3.8-Flash-Next to Hebrus: evidence before execution

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 0–3
- Estimated context: capsule ≈292 tokens; full note ≈1414
- Retrieve when: You are adding a new model family to an inference runtime and need an order of operations that prevents a plausible but incorrect graph from becoming the de facto specification.
- Retrieval triggers: new model-family bring-up; uncertain checkpoint identity; graph work beginning before executable contracts
- Decision: Freeze the source identity and executable contracts before implementing model execution.
- Invariant: Execution cannot become authoritative until source identity, model math, artifact geometry, and admission failures are independently executable.
- Failure signature: Kernel debugging begins before the exact input contract is closed, so local fixes accumulate around a plausible but incorrect model.
- Minimal safe implementation: Pin source and inventory, build scalar oracles, freeze physical owners, and require CPU-only admission before creating the execution graph.
- Rejected shortcut: Starting from a model name or generic architecture match and allowing the first plausible graph to define the specification.
- Evidence: Model-free, sanitizer, admission, CPU, Metal-regression, and build-isolation gates
- Evidence checkpoint: Hebrus 6c3ae19
- Claim boundary: The checkpoint is structurally admitted but remains pinned-not-supported; this note makes no throughput, quality, or runtime-support claim.
- Prerequisites: none
- Related notes: AFN-002, AFN-003, AFN-004, AFN-008, AFN-009
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/evidence-before-execution/
- Markdown URL: https://andreabor.io/agents/evidence-before-execution/index.md


The fastest way to make a new model family difficult to debug is to start with the graph.

Once tensors are flowing through kernels, every mismatch looks like a kernel problem. A shifted layer pattern looks like bad attention. A wrong norm convention looks like quantization damage. A tokenizer drift looks like incoherent logits. The implementation becomes a pile of plausible local fixes around an identity that was never closed.

For the Qwen3.8-Flash-Next integration, Hebrus took the opposite route: execution was forbidden until the source, equations, artifacts, and admission failures were independently testable.

## The object being integrated

The exact target is the public checkpoint [`Qwen/Qwen3.8-Flash-Next`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) at revision `de4b8e4d43b917e7706784d8bb445c9af86a3540`.

The closed text profile has:

| Property | Exact value |
|---|---:|
| Architecture | `Qwen4ExpForConditionalGeneration` |
| Runtime family spelling | `qwen4exp` |
| Text layers | 48 |
| Hidden width | 2,560 |
| Residual streams | 4 |
| Layer pattern | GDN, GDN, GDN, QSA; repeated 12 times |
| Routed experts | 512 |
| Active experts/token | 10 |
| Expert width | 640 |
| Maximum context | 262,144 tokens |
| Source tensors | 1,658 |
| Source shards | 131 |
| Indexed source bytes | 359,999,963,128 |

These values are admission equalities, not hints. A model with a similar name or dimensions is a different model.

## Phase 0: make identity executable

The source revision, tokenizer, chat template, configuration, Transformers implementation, shard ownership, tensor inventory, license, and file digests were pinned first.

The important output was not prose. It was a contract that can reject mutations. If an agent changes one prime, one layer type, one dtype, one shard span, or one source hash, the validator must fail on that field.

This turns “we are implementing Qwen4Exp” into a finite statement:

```text
repository + revision
+ exact file digests
+ exact tensor inventory
+ exact configuration
+ exact exclusions
= one admissible source identity
```

The first artifact is text-only. The 333 vision tensors and 31 MTP tensors in the source are accounted for but excluded by named rules. They are not silently ignored, and the base runtime is not allowed to accept image/video input or execute MTP.

## Phase 1: separate math from acceleration

Allocation-free C references were added before Metal work for:

- zero-centred RMS normalization;
- four-stream gated residual mixing;
- Gated DeltaNet recurrence and convolution state;
- float32 router softmax and normalized top-10 selection;
- Qwen sparse-attention grouping and deterministic selection;
- PLE hashing, gating, convolution, and history transitions.

The fixtures are captured from a pinned Transformers commit, then cross-checked against an independent transcription. Contract-only cases—tie policy, integer wrap, reset/copy/rewind—are labelled as such rather than misrepresented as upstream output.

This matters because several operations are unusually easy to “optimize” into a different model. Examples include adding one to the wrong norm weight, reassociating the GDN recurrence, pooling physical cache slots rather than logical token groups, or changing tie order in sparse selection.

## Phase 2: freeze physical structure without choosing a winner

The artifact has three disjoint owners:

1. dense/non-routed tensors;
2. one embedded `ds4.expert_major.v2` routed-expert store;
3. one embedded `ds4.ple_rows.v1` row store.

ExpertMajor retained its v2 container. A new family ID closes the 48-layer, 512-expert, gate/up/down geometry without reviving an abandoned v3 format.

The PLE table is not disguised as experts. It has a different access pattern, checksum unit, cache policy, and future codec decision, so it received a separate fixed-page format.

Crucially, the structural phase did not pretend to have selected production quantization. MLX affine4 with group size 64 is sufficient to prove the ExpertMajor geometry, but it is explicitly marked `phase2-structural-not-release-qualified`. The PLE production codec remains open.

## Phase 3: reject before the GPU exists

The structural admission binary is deliberately CPU-only. It validates the family, profile, source hashes, model constants, layer array, tokenizer/template identity, exact tensor set, embedded store manifests, byte extents, ownership, and text-only policy.

Only after those checks may a future runtime allocate pipelines, graph state, or caches. Negative fixtures independently mutate fields and assert that rejection happens at the intended boundary.

This ordering makes the failure useful. “Wrong `ple.rows`” is actionable. “The model emitted nonsense after 40 seconds” is not.

## The implementation rule for other agents

When continuing this port:

1. Read the current contract, not the model name.
2. Preserve `pinned-not-supported` until execution, quality, memory, and performance gates pass.
3. Treat every new optimization as a hypothesis behind an oracle.
4. Keep one owner for every state buffer and physical byte range.
5. Never weaken existing model-family contracts to admit the new family.

The checkpoint through structural admission is recorded as Hebrus commit `6c3ae19`. The public repository is [`andreaborio/hebrus`](https://github.com/andreaborio/hebrus); the note keeps the exact checkpoint even when a working branch has not yet been published upstream.

The result is not yet a running model. It is something more useful at this stage: a model that cannot quietly become the wrong model while agents implement it.

---

# AFN-002 — Why Qwen4Exp kept ExpertMajor v2

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified structural decision
- Phase: 2
- Estimated context: capsule ≈288 tokens; full note ≈1045
- Retrieve when: You need to add a model with new expert geometry to an existing routed-expert container without conflating model family, storage format, and release-qualified codec.
- Retrieval triggers: new expert geometry in an existing container; logical row width does not divide codec block size; structural acceptance confused with codec qualification
- Decision: Extend ExpertMajor v2 with a distinct family-4 descriptor while keeping its wire format unchanged.
- Invariant: Container version, model-family geometry, and physical codec identity remain independent decisions.
- Failure signature: A structurally plausible store parses even though one role cannot be represented by the declared block geometry, or a new family changes legacy output bytes.
- Minimal safe implementation: Add one immutable family descriptor, validate every count/role/shape/order exactly, preserve legacy branches, and label a geometry-only codec candidate as non-release.
- Rejected shortcut: Bumping the wire version, padding width 640 opportunistically, or reusing another family ID merely to obtain a positive fixture.
- Evidence: Sparse 48×512 fixture, unchanged legacy fixture hashes, strict C and sanitizer tests
- Evidence checkpoint: Hebrus c5cedb0
- Claim boundary: MLX affine4 G64 proves the geometry only; no Qwen4Exp routed-expert codec is release-qualified by this decision.
- Prerequisites: AFN-001
- Related notes: AFN-003, AFN-004
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/expertmajor-family-four/
- Markdown URL: https://andreabor.io/agents/expertmajor-family-four/index.md


Adding 512 experts did not require a new container. It required a new family contract.

That distinction sounds small, but it prevents two common mistakes in artifact work:

1. incrementing a format version when only the admitted geometry changed;
2. reusing a family ID because the byte layout happens to parse.

Hebrus already had `ds4.expert_major.v2`: a family-tagged, checksummed store that keeps routed gate/up/down records contiguous by expert. Qwen4Exp needs the same physical idea but has different model invariants.

## The closed family-4 geometry

The Qwen4Exp descriptor requires:

```text
layers            = 48
experts/layer     = 512
experts used      = 10
components        = gate | up | down
gate              = [2560, 640, 512]
up                = [2560, 640, 512]
down              = [640, 2560, 512]
layer IDs         = 0..47, contiguous
minimum alignment = existing ExpertMajor v2 rules
```

The v2 header and record layout did not need to change. The parser needed a fourth immutable descriptor and a higher structural maximum: 384 to 512 experts.

Families 1–3 continue through their previous branches. Their regression artifacts remain byte-identical. Family 4 is admitted only when every exact count, component role, dimension, order, offset, digest, and storage rule matches.

## The 640-wide row problem

The down projection has logical row width 640. Existing admitted GGML K-quant layouts use 256-element blocks.

`640 % 256 != 0`.

That is not a tail to hand-wave away. The container descriptor describes exact physical records. If the codec cannot represent the logical row under its closed block rules, the builder must fail.

The structural fixture therefore uses the already-understood MLX affine4 representation with group size 64:

```text
group elements = 64
group bytes    = 36
640 / 64       = 10 exact groups
```

This establishes that the v2 record geometry, extents, offsets, and reader can represent the family. It does **not** establish that affine4 G64 has acceptable quality or is the best runtime codec.

The implementation labels it accordingly:

```text
phase2-structural-not-release-qualified
```

## Why not invent padding?

Padding 640 to 768 would make a 256-element block codec fit, but it would also create a new physical profile with new byte formulas, kernel behavior, quality implications, and manifest semantics.

That may eventually be a valid candidate. It is not a parser fix.

An autonomous agent should never introduce that kind of padding locally just to get a positive fixture. The correct sequence is:

1. define the candidate codec and padding rule;
2. give it a distinct profile identity;
3. implement checked byte formulas and tail semantics;
4. add decoder/kernel parity tests;
5. measure model quality and target-hardware performance;
6. promote only after evidence.

## Why not call the PLE table 512 more experts?

Because representation should follow access geometry.

Routed experts are selected in small sets and consumed as matrices. PLE is a 320-million-row lookup table with page locality, hash-derived row IDs, independent checksums, and a future SSD page cache. Giving both the same container would save a type name and destroy ownership clarity.

The artifact therefore contains one ExpertMajor extent and one PLE extent. Their spans must be disjoint, and neither may be accidentally included in dense warm-up ranges.

## Agent checklist

When adding another ExpertMajor family:

- change the family descriptor before touching the shared parser;
- preserve legacy branches and verify old output hashes;
- distinguish structural parser acceptance from builder availability;
- reject a codec whose block geometry does not exactly cover every role;
- avoid padding, transposition, or logical-type aliases without a named profile;
- keep support status separate from “the fixture parses”.

The general lesson is that a stable container is valuable precisely because new model geometry can be admitted without changing its wire format. Version the structure when the structure changes. Use a family/profile identity when the model contract changes. Use a codec identity when the bytes change.

---

# AFN-003 — 320 million embedding rows without a 320 million-entry index

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified structural decision
- Phase: 2
- Estimated context: capsule ≈284 tokens; full note ≈1146
- Retrieve when: You are designing storage for hundreds of millions of fixed-width lookup rows and need bounded metadata, independent verification, and future SSD caching.
- Retrieval triggers: hundreds of millions of fixed-width rows; bounded SSD lookup metadata; TOCTOU-safe page verification
- Decision: Represent PLE as a fixed-page embedded extent with affine row location and one digest per page, never a per-row offset table.
- Invariant: Every row maps affinely to one fixed page, and caller output is published only from the same verified page snapshot.
- Failure signature: Metadata grows with row count, lazy mapping creates unbounded first-touch, or bytes are verified in one read and copied from another.
- Minimal safe implementation: Use a fixed manifest, one digest per fixed-stride page, checked locate arithmetic, single-snapshot verified reads, and an atomic verified writer.
- Rejected shortcut: A per-row offset table, OS page faults as the scheduler, or a two-read verify-then-copy path.
- Evidence: Strict parser/writer tests, ASan/UBSan, whole-payload corruption, overflow, fsync, rename, and short-read failures
- Evidence checkpoint: Hebrus c5cedb0
- Claim boundary: The wire structure is frozen; production row codec, rows per page, page stride, and cache policy remain unqualified.
- Prerequisites: AFN-001
- Related notes: AFN-002, AFN-004, AFN-012
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/ple-fixed-page-store/
- Markdown URL: https://andreabor.io/agents/ple-fixed-page-store/index.md


Qwen3.8-Flash-Next includes a large PLE table: 320,001,536 rows, each 160 values wide.

A naïve runtime design reaches for an offset table. At eight bytes per row, that index alone would be roughly 2.38 GiB before allocator and page overhead. It would also solve a problem that does not exist: the rows are fixed-size inside a closed codec profile.

The PLE v1 design uses fixed physical pages and affine arithmetic instead.

## Logical geometry

The sixteen hash heads use cumulative prime-sized segments. Their exact logical end is:

```text
head rows = 320,001,446
```

The checkpoint pads this to a row alignment of 128:

```text
align_up(320,001,446, 128) = 320,001,536
padding rows                         = 90
```

The 90 rows matter. Requiring the prime segments to sum to the padded extent would make the real checkpoint impossible to admit. Treating the trailing rows as arbitrary slack would weaken the identity. The manifest closes both values and the alignment rule.

## Physical geometry

PLE v1 has:

- a fixed 512-byte little-endian manifest header;
- a compact table with one 32-byte SHA-256 digest per page;
- alignment padding before payload;
- fixed-stride physical pages;
- a 64-byte duplicated page header;
- no per-row index.

For row `r`:

```text
page = r / rows_per_page
slot = r % rows_per_page
page_offset = payload_offset + page * page_stride
row_offset  = page_header_bytes + slot * encoded_row_bytes
```

Every multiply, add, alignment, extent, and `off_t` conversion is checked before I/O.

The page header repeats enough geometry to detect a page copied from another store or position: page index, first row, valid row count, logical width, encoded row bytes, rows per page, codec version, codec group size, and family.

## Digest domains

Three digest levels have different jobs:

1. **Manifest digest** — authenticates the header with its own digest field zeroed, the page-digest table, and alignment padding.
2. **Page digest** — authenticates one complete fixed-stride physical page, including its header and unused stride bytes.
3. **Payload digest** — authenticates every complete physical page in order.

Opening the store validates structure and the manifest without reading hundreds of millions of rows. Offline publication runs every page digest and the whole-payload digest.

The test for the final payload comparison is deliberately isolated: mutate payload bytes, recompute the affected page digest and manifest digest, leave the whole-payload digest stale, then require `open` and `verify_page` to pass while `verify_all` fails specifically on the whole payload.

## Transactional row reads

A row read takes one page snapshot.

It does not verify a page and then issue a second direct read for the row. That two-read design creates a time-of-check/time-of-use window: the bytes copied to the caller may not be the bytes that were hashed.

The correct sequence is:

```text
pread complete page into bounded buffer
verify duplicated page header
verify page SHA-256
copy requested encoded row to caller
```

On corruption or short read, the caller's output remains unchanged.

## Atomic publication

The writer uses a sibling temporary file and bounded one-page memory:

```text
create sibling temp
write header, digest table, padding, and pages
fsync temp file
close and reopen through the runtime parser
verify complete payload and boundary pages
fsync parent directory
rename temp over target       <- commit point
fsync parent directory again
```

A failure before rename preserves the old target and removes the temporary file. A post-rename directory-fsync error is reported, but the writer does not attempt an unsafe rollback after the new complete file has become visible.

Deterministic test hooks inject fsync failure before the commit point. Rename failure is tested against a non-replaceable target. Both prove that no apparently valid partial target or sibling temp remains.

## What remains open

The structural format does not pick a production codec.

Codec ID, version, group size, encoded row bytes, rows per page, page alignment, and derived stride belong to a future artifact profile. They need quality, capacity, cold-read, cache, and Metal decode evidence on the target machine.

This separation is intentional. The storage invariants are stable:

- fixed pages;
- affine lookup;
- exact geometry;
- bounded metadata;
- independent verification;
- transactional publication.

The performance choices are not yet stable. Freezing both at once would turn an early benchmark guess into a wire-format promise.

---

# AFN-004 — Admission before Metal: make every bad artifact fail cheaply

Project: Hebrus — https://github.com/andreaborio/hebrus
Contribute: https://github.com/andreaborio/hebrus/blob/main/CONTRIBUTING.md

## Retrieval capsule

- Status: verified checkpoint
- Phase: 3
- Estimated context: capsule ≈291 tokens; full note ≈1334
- Retrieve when: You are integrating a complex artifact into a GPU runtime and need deterministic field-level failures before expensive allocation or opaque backend errors.
- Retrieval triggers: GPU loader fails too late; artifact ownership or extent ambiguity; need deterministic field-level rejection
- Decision: Complete exact artifact admission in a dedicated CPU-only path before registering Qwen4Exp execution.
- Invariant: No GPU symbol, allocation, graph, or sparse-payload read is reachable until identity, policy, tensors, extents, and ownership all pass.
- Failure signature: A malformed artifact reaches the backend and fails as an opaque missing-tensor, allocation, overlap, or device error.
- Minimal safe implementation: Run ordered CPU-only admission over exact metadata, tensor identities, manifests, spans, exclusions, and owners, with one negative fixture per boundary.
- Rejected shortcut: Treating a loader as successful once it finds enough familiar tensors to start building a graph.
- Evidence: Sparse GGUF fixture, field-specific negative matrix, GPU-symbol isolation, sanitizer and full regression gates
- Evidence checkpoint: Hebrus 6c3ae19
- Claim boundary: The dedicated binary proves structural admission only and intentionally rejects generation; normal builds still register no Qwen4Exp physical runtime profile.
- Prerequisites: AFN-001, AFN-002, AFN-003
- Related notes: AFN-005, AFN-008, AFN-013
- Supersedes: none
- Canonical URL: https://andreabor.io/agents/admission-before-metal/
- Markdown URL: https://andreabor.io/agents/admission-before-metal/index.md


A loader is not successful because it can find enough tensors to start a graph.

It is successful when it can prove that one exact artifact owns every byte and satisfies every policy before expensive or irreversible work begins.

For Qwen4Exp, Hebrus added a dedicated structural-admission binary before adding execution. The binary is CPU-only and test-hook-only. It admits one sparse miniature fixture, emits a machine-readable ownership report, and refuses generation.

Normal builds still register no Qwen4Exp physical profile.

## Ordered failure

Admission is intentionally ordered from cheap identity checks to deeper physical verification:

1. parse GGUF bounds with checked arithmetic;
2. require architecture and artifact profile IDs;
3. require source revision and inventory digests;
4. validate every scalar and array model constant;
5. bind tokenizer, template, vocabulary, regex, and special IDs;
6. classify every tensor as base, PLE, MTP, or vision;
7. require the exact text-artifact tensor identity set;
8. validate the ExpertMajor manifest and all 48×512 records;
9. validate the PLE manifest, geometry, hash identity, and extent;
10. enforce text-only and MTP-not-executed policy;
11. sort physical ranges and prove disjoint ownership;
12. only then permit future platform and memory admission.

The order is part of the interface. A bad family should not fail later as a missing tensor. An overlapping extent should not survive until an SSD read. A vision tensor in a text-only artifact should not become ignored padding.

## Exact metadata, including arrays

The loader does not infer identity from a few familiar dimensions.

It validates the 48-entry layer-type array, PLE primes and offsets, 64-bit hash seeds, rotary sections, sparse-index parameters, GR dimensions, router rules, source revision, tokenizer digest, template digest, and explicit exclusions.

Metadata type is also closed. An integer with the same printed value is not accepted when the schema requires an unsigned 64-bit value or an array of a specific element type.

This protects against a subtle class of converter bugs: values that survive JSON or command-line inspection but change representation in GGUF.

## One tensor, one role, one owner

The source inventory has four classifications:

| Class | Source tensors | Base-artifact policy |
|---|---:|---|
| Base | 1,294 | Required, subject to routed/PLE physical replacement rules |
| PLE | 137 | Collapsed into one embedded PLE extent |
| Vision | 333 | Explicitly excluded from text profile |
| MTP | 31 | Explicitly excluded and not executed |

The converter dry run consumes all 1,658 source identities exactly once. “Skipped” is legal only as a named exclusion whose bytes remain accounted.

At runtime, dense tensors, the ExpertMajor tensor, the PLE tensor, GGUF metadata, alignment padding, and file bounds are separate owners. Sorted spans must not overlap.

Page-rounded dense warm-up spans receive an additional check: they must not accidentally include the adjacent PLE payload. Otherwise a harmless-looking residency optimization could fault a huge lookup table into the primary working set.

## Negative fixtures as an API

The compact fixture generator can mutate one field at a time:

- family or profile;
- revision or digest;
- context, head dimensions, top-k, expert count, GR rank;
- layer pattern;
- one PLE prime, offset, row count, width, or hash version;
- tokenizer/template hash or special ID;
- tensor name, rank, shape, type, offset, or length;
- overlapping extents;
- missing/extra tensors;
- second store;
- canonical routed or PLE tensors that should have been replaced;
- vision/MTP policy;
- embedded manifest or page geometry.

Each negative asserts the intended diagnostic and proves that the process has not reached GPU allocation.

This is more than test coverage. It makes error ordering a stable contract for future agents. If a refactor changes “profile mismatch” into “tensor missing”, the test explains that the loader stopped failing at the cheapest authoritative boundary.

## Why the binary has no GPU symbols

“We do not call Metal” is weaker than “Metal is not link-reachable.”

The structural test binary is linked without GPU entry points, and the test inspects its unresolved symbols. That turns pre-GPU rejection from a control-flow intention into a build property.

It also keeps sanitizer runs small and deterministic. Artifact validation can be fuzzed without initializing devices, compiling kernels, or depending on host GPU state.

## Handoff to execution

The admission report includes exact owners and byte ranges but marks runtime and payload support false. Generation exits with a structural-admission-only diagnostic.

The future resident graph may consume the admitted descriptor. It may not reinterpret the file, guess a family, discover a sidecar, or weaken an equality because a kernel prefers a different shape.

That is the boundary:

```text
loader proves what the artifact is
graph proves that implementation matches the model
benchmarks prove whether the implementation is useful
release gates decide whether anyone may call it supported
```

Keeping those proofs separate makes autonomous work faster. Agents can change one layer of evidence without silently granting authority to the next.