Andrea Borio

Agent-native knowledge base

Inference Architecture Field Notes

Reusable decisions from the open-source Hebrus inference engine: sparse attention, transactional state, fail-closed admission, and checkpoint-scale storage. Each note records the invariant, rejected shortcut, evidence, and claim boundary.

Agents launched by Andrea follow the scoped operator closeout contract; external contributors do not.

21 notes · public architecture only · evidence-bound

Start with the architecture

4 core decisions

Four boundaries that determine whether the runtime is merely plausible or mechanically defensible.

  1. Sparse attention AFN-016

    A sparse mask is not sparse attention

    Selecting 2,051 keys does not make attention sparse if the backend still allocates or traverses the full query-by-key matrix.

    verified checkpoint Read decision →
  2. Transactional state AFN-010

    One bank flip is enough: make the whole inference chunk transactional

    A token chunk can touch recurrent state, caches, routing, residual streams, and logits; publishing any one of them early creates a state that never existed mathematically.

    verified checkpoint Read decision →
  3. Storage AFN-003

    320 million embedding rows without a 320 million-entry index

    The fixed-page PLE store: affine lookup, independent checksums, transactional reads, and a codec decision intentionally left open.

    verified structural decision Read decision →
  4. Admission AFN-004

    Admission before Metal: make every bad artifact fail cheaply

    A CPU-only structural loader that proves metadata, tensor identity, physical ownership, and policy before any GPU symbol or graph allocation is reachable.

    verified checkpoint Read decision →

All decision records

21 records
  1. AFN-021 hypothesis

    Replay decode as prefill: a training flywheel for predictive MoE I/O

    A decode-shaped, prefill-executed data pipeline for training expert-prefetch predictors without putting optimization in the inference runtime.

    Phase Qwen4Exp predictive I/O training
    • prefill replay
    • decode traces
    • DFlash 2
    • Qwen4Exp
    • MoE routing
  2. AFN-020 verified structural decision

    Port the DFlash2 protocol, not its checkpoint

    DFlash2 weights are bound to one target model, but proposal, candidate selection, verification, accepted-prefix replay, and route-sidecar contracts can be rebuilt for another target.

    Phase DFlash2 compatibility boundary
    • DFlash 2
    • speculative decoding
    • checkpoint compatibility
    • Qwen4Exp
    • Qwen3.8-Flash-Next
  3. AFN-019 hypothesis

    Hide SSD misses before the router knows they exist

    A falsifiable pipeline hypothesis for SSD-heavy Qwen4Exp inference: exact PLE lookahead, predictive expert loading, DFlash route forecasts, component-level streaming, and deadline-aware scheduling.

    Phase 8–11 performance architecture
    • SSD streaming
    • ExpertMajor v2
    • Qwen4Exp
    • DFlash 2
    • speculative decoding
  4. AFN-016 verified checkpoint

    A sparse mask is not sparse attention

    Selecting 2,051 keys does not make attention sparse if the backend still allocates or traverses the full query-by-key matrix.

    Phase 6 architecture
    • sparse attention
    • Qwen4Exp
    • QSA
    • Metal
    • online softmax
  5. AFN-017 verified checkpoint

    Top-k chooses the set; time chooses the order

    Ranking decides which sparse-attention groups survive; canonical logical order decides how the selected row is represented and reduced.

    Phase 6 architecture
    • top-k selection
    • deterministic ordering
    • sparse attention
    • floating-point reproducibility
    • QSA
  6. AFN-018 verified checkpoint

    A hole in cache history is not padding

    Left padding may move the first real token; a missing position inside the visible run destroys sparse-group semantics and must fail closed.

    Phase 6 architecture
    • KV cache
    • logical positions
    • sparse attention
    • cache holes
    • left padding
  7. AFN-015 verified checkpoint

    A phase handoff needs a decision delta

    Finishing code is not the same as preserving reusable reasoning: every verified phase closes with either a new evidence-bound note or an explicit no-new-decision record.

    Phase agent operations
    • agent handoff
    • decision delta
    • architecture knowledge
    • evidence checkpoint
    • retrieval capsule
  8. AFN-014 verified checkpoint

    Do not call a contract vector an upstream capture

    Pinned Transformers can author module outputs, but synthetic tokens, hashed PLE rows, and cache images need a different provenance label or the fixture overstates its authority.

    Phase 5
    • Hebrus
    • oracle provenance
    • golden vectors
    • Transformers
    • contract control
  9. AFN-013 verified checkpoint

    Correctness code can exist without becoming a product feature

    Phase 5 links a resident graph coordinator and real Metal correctness kernels, yet normal Hebrus execution still rejects Qwen4Exp at physical-profile admission.

    Phase 5
    • Hebrus
    • staged rollout
    • capability admission
    • Metal correctness
    • pinned not supported
  10. AFN-012 verified checkpoint

    An ubatch is not a sequence boundary

    Dilated PLE convolution and token-history hashing must survive arbitrary chunking without leaking state between sequences or changing the current-EOS transition.

    Phase 5
    • Hebrus
    • ubatch parity
    • PLE
    • dilated convolution
    • sequence state
  11. AFN-011 verified checkpoint

    A reference algorithm needs an expiry condition

    Dense QSA is a useful correctness oracle through 2,051 visible keys; without a hard ceiling, temporary reference code silently becomes an impossible long-context implementation.

    Phase 5
    • Hebrus
    • sparse attention
    • dense reference
    • QSA
    • long context
  12. AFN-010 verified checkpoint

    One bank flip is enough: make the whole inference chunk transactional

    A token chunk can touch recurrent state, caches, routing, residual streams, and logits; publishing any one of them early creates a state that never existed mathematically.

    Phase 5
    • Hebrus
    • transactional inference
    • double buffering
    • Metal rollback
    • recurrent state
  13. AFN-009 verified checkpoint

    The recovery packet: enough context to continue, not enough to drown

    A layered handoff format that lets a new coding agent recover decisions, invariants, ownership, and the next gate without ingesting the project history.

    Phase agent operations
    • agent handoff
    • context engineering
    • recovery packet
    • llms.txt
    • project memory
  14. AFN-008 verified checkpoint

    Close the text pipeline before asking the model

    Phase 4 used pinned captures, independent regeneration, native tests, and 153 contract mutations to close tokenizer and chat semantics before a runtime graph existed.

    Phase 4
    • tokenizer oracle
    • chat template oracle
    • mutation testing
    • model bring-up
    • Transformers parity
  15. AFN-007 verified checkpoint

    Unicode is part of the model contract

    A tokenizer-class reconstruction dropped combining marks, while Jinja trimming followed Python semantics beyond common C whitespace helpers.

    Phase 4
    • Unicode tokenizer
    • combining marks
    • tokenizer.json
    • Jinja trim
    • Python str.strip
  16. AFN-006 verified checkpoint

    Rendered chat is not a flat string: preserve provenance

    A byte-identical control-token spelling can be trusted template syntax or untrusted client data; flattening before tokenization destroys that distinction.

    Phase 4
    • chat template security
    • control token provenance
    • prompt injection boundary
    • tool messages
    • Qwen4Exp
  17. AFN-005 verified checkpoint

    A logits matrix is not always the vocabulary

    Qwen4Exp exposes 248,320 output rows but only 248,077 valid token IDs; collapsing those domains creates invalid sampling and ambiguous decoding.

    Phase 4
    • padded logits
    • tokenizer vocabulary
    • sampling mask
    • Qwen4Exp
    • special tokens
  18. AFN-001 verified checkpoint

    Porting Qwen3.8-Flash-Next to Hebrus: evidence before execution

    Why the Qwen4Exp bring-up froze identity, math, artifact geometry, and failure semantics before the first Metal graph was allowed to exist.

    Phase 0–3
    • autonomous coding agents
    • Qwen3.8-Flash-Next
    • Qwen4Exp
    • Hebrus
    • model bring-up
  19. AFN-002 verified structural decision

    Why Qwen4Exp kept ExpertMajor v2

    A new 512-expert family needed stricter geometry, not a new container version—and a 640-wide row exposed the difference between structural proof and codec qualification.

    Phase 2
    • ExpertMajor v2
    • mixture of experts
    • Qwen4Exp
    • quantization geometry
    • MLX affine4
  20. AFN-003 verified structural decision

    320 million embedding rows without a 320 million-entry index

    The fixed-page PLE store: affine lookup, independent checksums, transactional reads, and a codec decision intentionally left open.

    Phase 2
    • PLE row store
    • SSD streaming
    • fixed-page storage
    • checksummed embeddings
    • affine indexing
  21. AFN-004 verified checkpoint

    Admission before Metal: make every bad artifact fail cheaply

    A CPU-only structural loader that proves metadata, tensor identity, physical ownership, and policy before any GPU symbol or graph allocation is reachable.

    Phase 3
    • model admission
    • fail closed
    • GGUF validation
    • GPU isolation
    • tensor ownership
For agents: retrieval formats and archive policy

Start from the compact index, load one retrieval capsule, and open the full note only when its decision matches the task. Treat hypothesis, structural-only, and pinned-not-supported as hard boundaries.

Public architecture only. The archive excludes personal information, account data, credentials, private prompts, conversations, local paths, network addresses, and unpublished infrastructure.

Andrea-launched agents. Before closing a verified phase, follow the scoped operator contract: publish a new evidence-bound decision or record that no new decision delta exists. External contributors are not bound by this rule.

Audience
Autonomous coding agents and systems engineers
Claims
Evidence-bound; hypotheses stay labelled
Canonical path
/agents/
Format
HTML · Markdown · LLM text · JSON