ww/docs/test-system-v2.md

# WW test architecture

Status: active architecture as of 2026-08-07. The target graph, corpus
accounting, and complete `test-commit` gate have been rechecked, including its
two concurrency controls separately and together. The separately gated
byte-identity and test-infrastructure proofs have also run. Bootstrap and
platform proofs have not run as part of this revision.

## Why the old suite was slow

The former suite made shell, Make, and one C executable per legacy case share
ownership of discovery, registration, phase selection, scheduling, result
records, timeout classification, and a last-green cache. Language behavior and
byte-identity loops repeatedly rebuilt complete package graphs. The measured
serial result was 411.403 seconds wall time, with a static lower bound of about
11,040 compiler-stage processes and 4,800 archive creations.

The replacement assigns each assertion to one owner and keeps expensive proof
categories out of the ordinary developer target.

## Owners

| Category | Owner |
| --- | --- |
| Arena, lexer, parser, checker, module-decl parse, narrow codegen | Six in-process C unit binaries |
| Compile success/rejection, stage-routed diagnostics, and runtime exit | `test/wcc/data/*/case.ww`, executed by `wwfixture` |
| Package semantics | `test/package`, `test/sep/sepinit_test.ww`, and the native package-test coordinator |
| Language behavior | `test/lang/*_test.ww` through the language `@test` runtime |
| Library behavior | The package coordinator's `lib/...` tree walk (`ww test -j N lib/...`) |
| Standalone library-source compilation | Three import-free real source paths named by `LIBRARY_STANDALONE_SOURCES`, compiled directly by both frontends |
| Compiler-output identity | `test-lang-byteid` and `test-data-byteid` plus residual byte/artifact carriers |
| Fixed point and self-host | `test-bootstrap` |
| Host linker/platform behavior | `test-platform` |

The live declarative compiler corpus has 1,759 fixtures and 3,518 C/WW cells:
351 expected rejections, 22 compile-only successes, 209 exit-zero
programs, and 1,177 explicit-exit programs.

15 native C carriers remain. They are partitioned exactly once into
the two terminal classes: six in-process units (`738_module_decl`
re-homed from the residuals — Go does not distinguish "bootstrap
observers" from unit tests, and its parsefile-call shape is exactly
the `200_parse` class), six bootstrap gates, and three platform-claim
carriers (`996_dyn_ww` under `test-platform`; `810_dyn` and
`989_dynentry_run` hold host loader/glibc claims and run with
`test-compiler`). The residual class is EMPTY: `794_xmod_ident_prefer`
— the last bug-pinned survivor — retired with its #55-sibling cgen
fix (its assertions migrated to `test/xmod/collide_test.ww`).

The 2026-08-08 residual drain retired the other 107: 5 deleted against
verified owners, 196 rows migrated into corpus fixtures (700_e2e's 136,
the fixture-class 46, the 926/929/671 run legs), and the rest ported to
ww-native observer suites on `test/testenv`, each carrier retired in
the same commit as its replacement: `test/sep/` (sep-layout/driver),
`test/xmod/` (module-tree collisions, mangle, typecheck, direnum),
`test/asm/` (positional asm-window needle observers), `test/object/`
(w6a/w6l ELF, archive, and link facts), `test/misc/` (divergence pins,
stamp probes, wwi round-trips), `test/tool/` (driver CLI, wwdump
gates, FFI, the c6 soak), and `test/libenv/` (the lib/os + lib/dirs
env-arranger legs; those lib suites now pass bare with loud SKIPs).
All suites run under `test-compiler`. The byte/artifact partition
stays EMPTY under `test/byteid/` exactly as before; wwstage-driver-leg
byte identity keeps its one owner, `test/byteid/libbyteid_test.ww`
(44-entry roster plus the lib/ completeness scan). Known cs≠ww
divergences exposed by row migration are pinned in
`DATABYTEID_DIVERGED` (3 entries) on the graduation discipline.

## Carrier endgame

The declarative corpus and the native `@test` owners are the permanent
test surface — the `go/test/` analogy. The native C carrier fleet is the
pre-Go-1.5 artifact and shrinks toward exactly two terminal classes:

1. **C-bootstrap observers.** The six in-process units watch the C
   frontend from inside its own process and are irreplaceable while
   `cmd/` is the live frontend. The six bootstrap gates are the same
   class's proof column: they compare the wwstage tools against the C
   stage to a fixed point. At the eventual selfhost flip both freeze
   into a bootstrap smoke gate (Go 1.5 deleted its C toolchain and that
   toolchain's tests; it did not port them).
2. **Host ABI/platform gates.** Behavior owned by the host linker,
   loader, or ABI (`996_dyn_ww` today). These observe the platform, not
   the compiler, and stay native exactly as long as the claim is about
   the platform.

Everything else gets a ww owner. Byte and artifact observations —
assembly-pattern greps, symbol tables, frame layouts, `.wwi`
round-trips, driver-leg comparisons — are subprocess plus file IO plus
string search, which `test/package/package_test.ww` already performs
natively (runcommand + in-language assertions). A C carrier whose
assertions fit that shape is ported and retired in the same commit,
assertions preserved or strengthened, with no compatibility execution
path left behind. A carrier that is merely historical is deleted
outright; git history is the archive.

## Public targets

| Target | Composition |
| --- | --- |
| `test` | Six in-process units plus one compile-only C/WW compiler-fixture smoke case |
| `test-compiler` | Complete fixture corpus plus residual compiler/integration carriers |
| `test-package` | Package planning, grouping, routing, and package runtime only |
| `test-lang` | Language-owned `@test` behavior |
| `test-library` | The coordinator's `lib/...` walk plus three import-free standalone-source C/WW compilation checks |
| `test-commit` | Unit + compiler + package + language + library behavior |
| `test-byteid` | Compiler-output identity gates |
| `test-bootstrap` | Fixed-point bootstrap plus the 950/991–995 native gates |
| `test-platform` | Host-dependent dynamic-link gate |
| `test-wwfixture` | Fixture CLI/process/protocol integration boundary |
| `test-all` | Commit + byte-ID + bootstrap + platform + test-infrastructure checks |

`test-commit` deliberately excludes byte identity, bootstrap, and platform
work. `test` is intentionally smaller than the old target and is the ordinary
developer feedback gate. Its purpose is a short, direct path from a compiler
edit to useful evidence, not compliance with an arbitrary wall-clock cutoff.

Make and fixture scheduling have separate, explicit owners. The Makefile does
not detect CPU count or add `-j` to `MAKEFLAGS`; the caller selects Make
parallelism with the standard `make -jN` option. `JOBS ?= 1` controls only the
`wwfixture -j N` value passed by `test-compiler`. A normal fast gate can use
both layers deliberately:

```sh
make -j4 JOBS=4 test-commit
```

For deterministic failure reproduction, make both layers serial explicitly:

```sh
make -j1 JOBS=1 test-commit
```

Serial execution is a debugging mode, not a correctness requirement. `JOBS`
is not inferred from `MAKEFLAGS`, and there is no jobserver adapter or second
scheduler hidden in Make.

`nocc` remains the separate, explicit reproduction route from a checked-in
stage-0 snapshot. It is not an implicit prerequisite of ordinary tests or of
`test-bootstrap`, because it has an external stage-0 precondition.

## Compiler fixtures

Each fixture is one directory with one `case.ww`. Its first line is exactly one
of:

```text
//ww:error "required diagnostic fragment"
//ww:error c "C-stage fragment" ww "WW-stage fragment"
//ww:compile
//ww:run
//ww:run-exit N
```

- `error` requires normal nonzero frontend termination and the declared stderr
  fragment. The labeled form routes distinct fragments to the C and WW cells;
  labels are fixed-order and neither fragment is treated as a shared fallback.
- `compile` requires frontend exit zero and produces no executable.
- `run` builds and requires normal program exit zero.
- `run-exit N` builds and requires normal program exit `N`.

Every fixture is run against the C and WW frontends. Signals, launch failures,
timeouts, build failures, and runtime exits are distinct outcomes.
`test-compiler` passes `-j $(JOBS)` and therefore uses one fixture slot by
default. The direct `wwfixture` CLI retains its own four-slot default; pass
`-j N` when its concurrency must be explicit. Its existing
`os.exec.start`/`poll` loop supervises the independent processes. Each cell has
its own working directory; filesystem fixtures use that directory or an
existing `temp.named` path rather than a shared fixed pathname.

The old `TESTS` list and all explicit per-wrapper Make rules are gone. A
surviving C carrier is registered only by its source file and built through one
generic pattern rule. Four arena/frontend units share a static rule; the codegen
unit links the existing private `cgen` and text-emitter objects directly.

`test/wwfixture/integration.sh` remains the direct black-box owner for behavior
that exists only at the command/process/protocol boundary: filtering and list
output, phase and outcome classification, diagnostic routing, malformed-corpus
and identity-drift rejection, signal/timeout/interruption cleanup, publication
failure, and strict result-stream decoding. Semantic fixtures cannot prove
those observations about their coordinator. `test/wwfixture/process/main.ww`
owns the lower-level `os.exec` primitives, not the CLI policy layered over
them, so it is complementary rather than a duplicate owner.

The six unit sources have zero active `system`, `popen`, `fork`, or `exec`
calls. Before direct conversion they contained two `popen` call sites and a full
`test-unit` run launched one `ww -V` plus twelve `w6c` processes. `400_w6c` now
checks the same twelve assembly fragments in process, grouped under ten unique
sources. The CLI version assertion lives with C/WW-driver parity in the existing
`949_driver_flagargs` integration carrier; `000_smoke` retains its arena-growth
and nonempty-version-constant assertions.

## Package and language behavior

`ww test` delegates directory package requests to the native package
coordinator. The coordinator owns request-pattern expansion, package grouping,
same-package and external-package variant selection, filtering, result
aggregation, and its internal temporary workspace. Test sources are exclusively
`*_test.ww` (Go's `_test.go` contract): a line-leading `@test` declaration in
any other source is rejected loudly — by both driver stages at directory
enumeration and by the coordinator at source classification. A local spelling
containing `...` (for example `ww test lib/...`, Go's `./...` form) is
recognized by both driver stages before path resolution and walks from the
directory prefix before the first wildcard. Recursive children beginning `.`
or `_` and exact `testdata` subtrees are pruned; child directory symlinks are
not followed. A wildcard cannot consume a non-terminal exact `vendor` element,
while an explicitly vendor-rooted pattern remains legal. Every selected
test-bearing directory becomes one package run, and selected source-bearing
directories without tests report the usual `?` line. The coordinator first
launches one driver plan for the complete command-global package/action union.
After that shared build completes, `-j N` schedules up to N successful selected
test binaries concurrently under `os.exec` start/poll supervision (no threads);
emission stays strictly in group order, so the byte stream is identical at
every `-j` level, and `-j 1` — the default — matches the former sequential
run loop exactly. Measured on the 31-package `lib/...` walk:
7.0s sequential, 2.4s at `-j 4`. With `-c`, it publishes each exact
`<package>.test` binary in the package directory; the first output owns the one
shared cold sepwork containing the command-global action universe. Those become
caller-owned artifacts. `-c -o <name>`
names that artifact instead of the fixed stem, for exactly one package:
the coordinator rejects a multi-package fan-out ("cannot use -o with
multiple packages", Go's `go test -o` rule), and `-o` without `-c` is
rejected at the driver ("needs -c for a package target") because a plain
run always executes from the temp root. Without `-c`, it removes the
temporary binary and scratch with its workspace. The language runtime
owns individual `@test` functions.

Separate compilation is the only driver build path; no compatibility mode
switch remains.

`test/package/package_test.ww` also owns the dual-stage
`declared_name_identity_and_file_import_scope` and
`explicit_import_alias_binding_modes` observers. They generate temporary
directory trees proving that canonical import identity, physical directory,
declared package name, optional source alias, and effective source-file
qualifier remain distinct. The alias matrix covers default and explicit-only
qualification, bare function/type/def/const/variable rejection with exact
unused-before-undefined diagnostics, same-file duplicate bindings, accepted
same-path imports under distinct bindings, alias reuse across files, sibling
scope isolation, package-declaration collisions, unused aliases, blank
side-effect imports, repeated blank/default/explicit combinations, and blank
no-binding/no-unused behavior. It also pins canonical `.wwi` spelling,
archive/action ownership, one canonical direct edge for repeated occurrences,
stage-equal diagnostics and binaries, and clean rejection state.
The declared-name observer proves imports are file-scoped while dependency
edges are the package-wide sorted union; command and
production/internal/external/generated-main variants retain canonical action
ownership; vendor expansion changes identity but not the effective qualifier;
compiler argv contains only direct `.wwi` inputs; and its named rejected
actions leave neither committed nor staged action artifacts or a published
binary. It also forces a request transaction to reject after dependency work
has staged, then proves that both stages restore every prior artifact, unit,
tool record, stamp, and product rather than accepting a mixed warm generation.
Together with the existing directory, recursive, vendor,
exact-argv, command, and persistent-workdir observers, the package suite proves
archive-only link argv and exact warm/rejection-state behavior without
duplicating those broader mechanisms in this observer.

`test/sep/sepinit_test.ww` is the single focused package-initialization owner.
It generates all source trees temporarily and runs independent cold/persistent
Cstage and WWstage legs. Its matrix covers blank-only reachability; dependency,
diamond, and independent ready-task order; runtime call/allocation and aggregate
package lets; multiple init declarations and invalid forms; direct/qualified
init invisibility; initialization cycles and diagnostic order; command,
internal-test, external-test, support, and generated-main ownership; canonical
task and dispatcher symbols; deterministic one- and two-member archives;
`.wwi` exclusion; byte-identical artifacts/binaries; init-only invalidation;
warm invalid-init rollback; direct/recursive variant equivalence; exact
production, support, and generated-main task ordering; complete normalized
rejection diagnostics; dangling staging/rollback no-follow rejection; and
checked compiler-output failure/non-regular-destination rollback; and
bounded-memory allocation-failure parity under one shared ceiling supplied by
the repository-built `sep-limitexec` helper.

`ww build`, an explicit single-file `ww test -o <stem>`, and each successful
directory-package `ww test -c` build publish `<stem>.sepwork` as a caller-owned
artifact directory. The driver acquires it with one fresh `mkdir` and refuses
an existing path; it never clears a collision. A caller keeps only the exact
artifacts it observes and removes that exact tree on every later success or
failure. `ww run` and no-output single-file `ww test` use driver-owned scratch
instead; both driver stages place that scratch and their temporary executable
beneath one freshly acquired directory, remove both after every build result,
and make cleanup failure fail the command. Make recipes build driver-produced
tools in invocation-owned directories and apply the same exact cleanup rule.

`ww build -w DIR` and single-file `ww test -w DIR` replace that scratch with a
caller-owned persistent package-artifact workdir: for these direct routes the
directory must already exist, is never cleaned by the driver, and holds one
committed unit, `.wwi`, `.s`, `.o`, and dep `.a` per package plus byte copies of
the invoking driver, compiler, and assembler and a small mode stamp. A package
is reused only when
its freshly composed owner unit byte-equals the committed unit, no recompiled
direct dependency emitted changed export bytes, and every applicable recorded
executable byte-equals the live executable — content identity only, no mtimes,
no hashes, every decision reproducible with `cmp` against plain files. The
driver identity covers graph construction, owner-unit composition,
direct-export argument construction, archive creation, and commit algorithms
that neither unit bytes nor compiler identity can name.
Recompiled artifacts, dispatcher artifacts, products, statuses, tool copies,
and the stamp land at staged `.new` names. After every requested product stages
successfully, one rollback-capable transaction installs the whole generation;
any producer or installation failure preserves the prior committed bytes and
removes remaining stages. An interrupted or rejected request therefore cannot
create false reuse or a mixed generation; a successful executable link still
reruns when required. One workdir serves one invocation at a time and may hold
the complete command-global action universe for all selected roots and isolated
variants; semantic identity, rather than request shape, controls reuse. Both
driver stages implement the identical contract. This is build staleness in the
Make/mk/Go sense, not a result cache:
tests always run, and the byte-identity and bootstrap gates keep building on
fresh scratch. `make clean` reclaims every workdir under `out/`.

On a package or tree target, `ww build/test -w DIR` forwards that exact caller
path as the one command-global semantic-action store; neither pattern spelling
nor package-group name derives a persistent subdirectory. The delegated driver
may create a missing store only after graph and request preflight, and rolls
back newly created empty prefixes if later setup fails. Every reuse decision
stays with the driver's content-identity contract above. `-w` and test `-c` do
not mix — the coordinator rejects the combination rather than guess which
artifact tree the caller owns. `test-library` uses exactly this: one
`ww test -j $(JOBS) -w out/wwbuild/wwtest-lib lib/...` line owns library
behavior (measured: 2.3s cold, 1.0s warm at
`-j 4`, vs 4.7s for the retired 59-target per-file fan-out).

`test/lang` currently uses one package per source file, so its complete gate
still performs independent package builds. That remaining source layout is not
hidden behind caching or concurrency; it is outside the small `test` target.

## Compiler-only byte identity

`ww build -S` and `ww test -S -o <stem>` run source discovery, dependency
ordering, unit composition, and each required `w6c -c` invocation. They return
after the complete `.s`/`.wwi` set exists. The producer loop does not invoke
`w6a`, create per-package archives, or invoke `w6l` when `-S` is active.

`test-lang-byteid` runs the 158 selected language files once with C `w6c` and
once with WW `w6c`, requires identical emitted `.s` filename sets including
`__root.s`, and compares those bytes. Relative to the old two-leg full builds,
this removes at least 3,476 assembler launches, 3,160 archive writes, and 316
linker launches from the comparison loop. A cold Make invocation may still
build prerequisite compiler binaries; the compiler-only claim applies to the
per-language-file comparison path.

`test-data-byteid` applies the same comparator to the declarative corpus:
every non-error `case.ww` builds twice through the fixed cstage driver with
only `WW_W6C` swapped, and every emitted per-package `.s` must be
byte-identical. `//ww:error` fixtures have no `.s`; their both-stage reject
parity is owned by the fixture corpus itself. Known cs/ww divergences are
pinned in `DATABYTEID_DIVERGED` with the `989_lib_byteid` discipline: a
pinned fixture must still build on both stages and still differ, so a
compiler fix fails the gate demanding graduation rather than silently
widening coverage. The full sweep compares the 1,157 non-error fixtures
in about a minute and is scratch-rooted under `out/`, not `/tmp`.

Byte identity is an explicit proof gate. It is not a prerequisite of `test` or
`test-commit`.

## Bootstrap, subprocesses, and CSP

Stage-2-through-stage-4 fixed-point proofs and the 950/991–995 self-host gates
are reachable through `test-bootstrap` and `test-all`, never through `test` or
`test-commit`. Cold ordinary targets may still build their C- and WW-stage tool
prerequisites once; they do not iterate those tools to a fixed point. The C
bootstrap source and the standalone `nocc` route remain intact.

The bootstrap recipe owns the fixed `out/bootstrap` tree. Make schedules that
target once within one invocation, but two independent `make bootstrap`
invocations are not safe to run concurrently and remain mutually exclusive.

`lib/os/exec` is the sole reusable WW subprocess mechanism. Fixture and package
coordinators use its captured asynchronous path. The WW driver directly uses
`os.exec.runstdio` for inherited-stdio, inherited-environment, leader-only
compiler, assembler, linker, cleanup, run, and single-file-test calls. The
local WW `procrun` implementation is deleted. The C bootstrap retains its C
process implementation because it cannot consume a WW standard-library
module.

WW has tokens and an opaque type for future CSP/channel work, but no mature
production channel operations, task runtime, or scheduler. No channel,
goroutine, thread, worker-runtime, or CSP library was added. Coordinators stay
single-threaded; OS process polling does not require language-level threads.

## Retired mechanisms

The following are deleted, not adapted:

- `test/run`, including its phase classifier, `xargs` scheduler, atomic private
  records, timeout-text classifier, result collector, and last-green cache;
- `test/run_test.sh`, the synthetic shell tests for that protocol;
- every explicit `TESTS` registration and all 329 explicit wrapper rules;
- the unregistered `test/runww.ww` corpus runner;
- frozen duplicate compiler-corpus code under `internal/wwtest`, `test/wwtest`,
  and `test/compiler`;
- the redundant standalone `smoke`, `test-run`, and `test-harness` routes; and
- library-launcher wrappers whose only assertion was an existing `@test`
  source's exit status, including the declarative `900_stdlib.c` launcher.

There is no test cache, daemon, scanner, generated manifest, database, new
framework, compatibility API, concurrency runtime, dependency, or changed
timeout policy in this architecture.

## Open driver work

None; the package-level `-o` contract (the last carried bullet) landed as
`-c -o <name>` for exactly one package.

## Validation policy

Use `test-unit` as the inner loop for lexer, parser, checker, and narrow codegen
changes. Run `test` for the ordinary local compiler check, followed by the
focused owner for the changed behavior. Use `test-commit` for ordinary
pre-commit behavior; use `make -j4 JOBS=4 test-commit` when parallel feedback is
desired, and `make -j1 JOBS=1 ...` to reproduce failures deterministically.
Run `test-byteid` and `test-bootstrap` only when those proof categories are
intended. `test-all` is the exhaustive CI/release composition and should not be
launched casually.