Full-model output is a terrible first oracle for a new runtime.
If the answer differs from Transformers, the cause may be tokenization, chat rendering, special IDs, graph math, state layout, quantization, kernels, cache transitions, or sampling. One failing string tells an agent almost nothing about which layer owns the defect.
Phase 4 of the Qwen4Exp bring-up closes the text pipeline while execution is still disabled.
The evidence stack
The tokenizer gate contains:
59 pinned text encode/decode cases
6 explicit decode-boundary controls
33 exact added-token identities
1 closed effective-vocabulary/physical-logits boundary
The chat gate contains:
34 pinned Transformers render or error captures
5 Hebrus text-only contract rejections
39 total cases
Together they cover default and explicit reasoning effort, disabled thinking, preservation of prior reasoning, system-role cardinality, tools, grouped tool results, insertion-ordered UTF-8 JSON, literal control tokens, structured-media rejection, Unicode trimming, invalid inputs, and transactional output preservation.
The wider frozen Qwen4Exp contract now rejects 153 out of 153 deliberate mutations. That total spans identity, source pins, inventory, tensor roles, layer types, derived constants, PLE geometry, expert-store geometry, graph facts, admission, tokenizer, chat, and licensing boundaries. It is not presented as “153 chat tests”; it is the mutation count for the complete contract after Phase 4.
Four different authorities
The oracle hierarchy keeps four kinds of evidence distinct:
- Pinned upstream captures establish what the selected public Transformers revision does.
- Independent regeneration checks that the fixture is not merely a dump produced by the implementation under test.
- Native C tests establish parity and failure semantics in Hebrus.
- Contract controls define behavior that upstream does not own, such as trust provenance, text-only rejection, rollback, and deterministic boundaries.
A fixture records the origin of each case. Contract controls are not labelled as upstream output, and upstream behavior is not silently replaced because a native approximation is easier.
Why model-free first
The tokenizer and renderer need no checkpoint allocation, Metal device, SSD cache, or 64 GB machine. Their gate is fast enough to run under sanitizers and in the normal regression suite.
This changes the debugging tree for the first resident graph:
prompt bytes differ? -> chat oracle
token IDs differ? -> tokenizer oracle
embedding input differs? -> graph boundary
layer output differs? -> primitive or graph implementation
final logits differ? -> graph, state, or numeric policy
sample differs only? -> sampler boundary
Without the first two closed nodes, every later mismatch remains ambiguous.
Mutation tests are stronger than happy paths
A positive fixture proves one accepted object works. A mutation battery proves the validator notices which facts make it that object.
Phase 4 adds mutations for tokenizer authority, normalization, regex, added-token mappings, valid vocabulary size, physical width, stop IDs, template defaults, trimming, segment kinds, structured-media policy, oracle counts, and fixture digests.
The important property is fail-closed behavior. A changed digest cannot be accepted because the visible cases still happen to pass. A changed count cannot be ignored because the array looks plausible. A flattened trust boundary cannot hide behind byte-identical rendering.
What the checkpoint does not claim
No weights are loaded by these tests. They do not establish:
- BF16 or quantized layer parity;
- the resident Metal graph;
- sparse QSA kernels;
- PLE SSD streaming;
- expert codec quality;
- long-context state behavior;
- memory admission;
- tokens per second;
- product or server support.
The family remains pinned-not-supported. The value of Phase 4 is not that the model can answer. It is that, when the first graph answers incorrectly, two entire classes of causes have already been removed from the search space.
General rule for agents
Before building a full-model oracle:
- close prompt rendering and tokenization independently;
- pin upstream implementations and serialized assets;
- preserve exact bytes, IDs, errors, and trust metadata;
- give policy decisions their own contract-labelled cases;
- regenerate fixtures independently and offline;
- mutate every identity field and semantic boundary;
- keep support status false until later gates earn it.
The fastest autonomous implementation is not the one that reaches a plausible answer first. It is the one that makes each wrong answer cheap to localize.