ww/docs/test-system-v2.md

WW test architecture

Status: active architecture as of 2026-08-07. The target graph, corpus accounting, and complete test-commit gate have been rechecked, including its two concurrency controls separately and together. The separately gated byte-identity and test-infrastructure proofs have also run. Bootstrap and platform proofs have not run as part of this revision.

Why the old suite was slow

The former suite made shell, Make, and one C executable per legacy case share ownership of discovery, registration, phase selection, scheduling, result records, timeout classification, and a last-green cache. Language behavior and byte-identity loops repeatedly rebuilt complete package graphs. The measured serial result was 411.403 seconds wall time, with a static lower bound of about 11,040 compiler-stage processes and 4,800 archive creations.

The replacement assigns each assertion to one owner and keeps expensive proof categories out of the ordinary developer target.

Owners

Category	Owner
Arena, lexer, parser, checker, module-decl parse, narrow codegen	Six in-process C unit binaries
Compile success/rejection, stage-routed diagnostics, and runtime exit	test/wcc/data/*/case.ww, executed by wwfixture
Package semantics	test/package and the native package-test coordinator
Language behavior	test/lang/*_test.ww through the language @test runtime
Library behavior	The package coordinator's lib/... tree walk (ww test -j N lib/...)
Standalone library-source compilation	Three import-free real source paths named by LIBRARY_STANDALONE_SOURCES, compiled directly by both frontends
Compiler-output identity	test-lang-byteid and test-data-byteid plus residual byte/artifact carriers
Fixed point and self-host	test-bootstrap
Host linker/platform behavior	test-platform

The live declarative compiler corpus has 1,759 fixtures and 3,518 C/WW cells: 351 expected rejections, 22 compile-only successes, 209 exit-zero programs, and 1,177 explicit-exit programs.

15 native C carriers remain. They are partitioned exactly once into the two terminal classes: six in-process units (738_module_decl re-homed from the residuals — Go does not distinguish "bootstrap observers" from unit tests, and its parsefile-call shape is exactly the 200_parse class), six bootstrap gates, and three platform-claim carriers (996_dyn_ww under test-platform; 810_dyn and 989_dynentry_run hold host loader/glibc claims and run with test-compiler). The residual class is EMPTY: 794_xmod_ident_prefer — the last bug-pinned survivor — retired with its #55-sibling cgen fix (its assertions migrated to test/xmod/collide_test.ww).

The 2026-08-08 residual drain retired the other 107: 5 deleted against verified owners, 196 rows migrated into corpus fixtures (700_e2e's 136, the fixture-class 46, the 926/929/671 run legs), and the rest ported to ww-native observer suites on test/testenv, each carrier retired in the same commit as its replacement: test/sep/ (sep-layout/driver), test/xmod/ (module-tree collisions, mangle, typecheck, direnum), test/asm/ (positional asm-window needle observers), test/object/ (w6a/w6l ELF, archive, and link facts), test/misc/ (divergence pins, stamp probes, wwi round-trips), test/tool/ (driver CLI, wwdump gates, FFI, the c6 soak), and test/libenv/ (the lib/os + lib/dirs env-arranger legs; those lib suites now pass bare with loud SKIPs). All suites run under test-compiler. The byte/artifact partition stays EMPTY under test/byteid/ exactly as before; wwstage-driver-leg byte identity keeps its one owner, test/byteid/libbyteid_test.ww (44-entry roster plus the lib/ completeness scan). Known cs≠ww divergences exposed by row migration are pinned in DATABYTEID_DIVERGED (3 entries) on the graduation discipline.

Carrier endgame

The declarative corpus and the native @test owners are the permanent test surface — the go/test/ analogy. The native C carrier fleet is the pre-Go-1.5 artifact and shrinks toward exactly two terminal classes:

1. C-bootstrap observers. The six in-process units watch the C frontend from inside its own process and are irreplaceable while cmd/ is the live frontend. The six bootstrap gates are the same class's proof column: they compare the wwstage tools against the C stage to a fixed point. At the eventual selfhost flip both freeze into a bootstrap smoke gate (Go 1.5 deleted its C toolchain and that toolchain's tests; it did not port them).
2. Host ABI/platform gates. Behavior owned by the host linker, loader, or ABI (996_dyn_ww today). These observe the platform, not the compiler, and stay native exactly as long as the claim is about the platform.

Everything else gets a ww owner. Byte and artifact observations — assembly-pattern greps, symbol tables, frame layouts, .wwi round-trips, driver-leg comparisons — are subprocess plus file IO plus string search, which test/package/package_test.ww already performs natively (runcommand + in-language assertions). A C carrier whose assertions fit that shape is ported and retired in the same commit, assertions preserved or strengthened, with no compatibility execution path left behind. A carrier that is merely historical is deleted outright; git history is the archive.

Public targets

Target	Composition
test	Six in-process units plus one compile-only C/WW compiler-fixture smoke case
test-compiler	Complete fixture corpus plus residual compiler/integration carriers
test-package	Package planning, grouping, routing, and package runtime only
test-lang	Language-owned @test behavior
test-library	The coordinator's lib/... walk plus three import-free standalone-source C/WW compilation checks
test-commit	Unit + compiler + package + language + library behavior
test-byteid	Compiler-output identity gates
test-bootstrap	Fixed-point bootstrap plus the 950/991–995 native gates
test-platform	Host-dependent dynamic-link gate
test-wwfixture	Fixture CLI/process/protocol integration boundary
test-all	Commit + byte-ID + bootstrap + platform + test-infrastructure checks

test-commit deliberately excludes byte identity, bootstrap, and platform work. test is intentionally smaller than the old target and is the ordinary developer feedback gate. Its purpose is a short, direct path from a compiler edit to useful evidence, not compliance with an arbitrary wall-clock cutoff.

Make and fixture scheduling have separate, explicit owners. The Makefile does not detect CPU count or add -j to MAKEFLAGS; the caller selects Make parallelism with the standard make -jN option. JOBS ?= 1 controls only the wwfixture -j N value passed by test-compiler. A normal fast gate can use both layers deliberately:

make -j4 JOBS=4 test-commit

For deterministic failure reproduction, make both layers serial explicitly:

make -j1 JOBS=1 test-commit

Serial execution is a debugging mode, not a correctness requirement. JOBS is not inferred from MAKEFLAGS, and there is no jobserver adapter or second scheduler hidden in Make.

nocc remains the separate, explicit reproduction route from a checked-in stage-0 snapshot. It is not an implicit prerequisite of ordinary tests or of test-bootstrap, because it has an external stage-0 precondition.

Compiler fixtures

Each fixture is one directory with one case.ww. Its first line is exactly one of:

//ww:error "required diagnostic fragment"
//ww:error c "C-stage fragment" ww "WW-stage fragment"
//ww:compile
//ww:run
//ww:run-exit N

• error requires normal nonzero frontend termination and the declared stderr fragment. The labeled form routes distinct fragments to the C and WW cells; labels are fixed-order and neither fragment is treated as a shared fallback.
• compile requires frontend exit zero and produces no executable.
• run builds and requires normal program exit zero.
• run-exit N builds and requires normal program exit N.

Every fixture is run against the C and WW frontends. Signals, launch failures, timeouts, build failures, and runtime exits are distinct outcomes. test-compiler passes -j $(JOBS) and therefore uses one fixture slot by default. The direct wwfixture CLI retains its own four-slot default; pass -j N when its concurrency must be explicit. Its existing os.exec.start/poll loop supervises the independent processes. Each cell has its own working directory; filesystem fixtures use that directory or an existing temp.named path rather than a shared fixed pathname.

The old TESTS list and all explicit per-wrapper Make rules are gone. A surviving C carrier is registered only by its source file and built through one generic pattern rule. Four arena/frontend units share a static rule; the codegen unit links the existing private cgen and text-emitter objects directly.

test/wwfixture/integration.sh remains the direct black-box owner for behavior that exists only at the command/process/protocol boundary: filtering and list output, phase and outcome classification, diagnostic routing, malformed-corpus and identity-drift rejection, signal/timeout/interruption cleanup, publication failure, and strict result-stream decoding. Semantic fixtures cannot prove those observations about their coordinator. test/wwfixture/process/main.ww owns the lower-level os.exec primitives, not the CLI policy layered over them, so it is complementary rather than a duplicate owner.

The six unit sources have zero active system, popen, fork, or exec calls. Before direct conversion they contained two popen call sites and a full test-unit run launched one ww -V plus twelve w6c processes. 400_w6c now checks the same twelve assembly fragments in process, grouped under ten unique sources. The CLI version assertion lives with C/WW-driver parity in the existing 949_driver_flagargs integration carrier; 000_smoke retains its arena-growth and nonempty-version-constant assertions.

Package and language behavior

ww test delegates directory package requests to the native package coordinator. The coordinator owns request-pattern expansion, package grouping, same-package and external-package variant selection, filtering, result aggregation, and its internal temporary workspace. Test sources are exclusively *_test.ww (Go's _test.go contract): a line-leading @test declaration in any other source is rejected loudly — by both driver stages at directory enumeration and by the coordinator at source classification. A local spelling containing ... (for example ww test lib/..., Go's ./... form) is recognized by both driver stages before path resolution and walks from the directory prefix before the first wildcard. Recursive children beginning . or _ and exact testdata subtrees are pruned; child directory symlinks are not followed. A wildcard cannot consume a non-terminal exact vendor element, while an explicitly vendor-rooted pattern remains legal. Every selected test-bearing directory becomes one package run, and selected source-bearing directories without tests report the usual ? line. The coordinator first launches one driver plan for the complete command-global package/action union. After that shared build completes, -j N schedules up to N successful selected test binaries concurrently under os.exec start/poll supervision (no threads); emission stays strictly in group order, so the byte stream is identical at every -j level, and -j 1 — the default — matches the former sequential run loop exactly. Measured on the 31-package lib/... walk: 7.0s sequential, 2.4s at -j 4. With -c, it publishes each exact <package>.test binary in the package directory; the first output owns the one shared cold sepwork containing the command-global action universe. Those become caller-owned artifacts. -c -o <name> names that artifact instead of the fixed stem, for exactly one package: the coordinator rejects a multi-package fan-out ("cannot use -o with multiple packages", Go's go test -o rule), and -o without -c is rejected at the driver ("needs -c for a package target") because a plain run always executes from the temp root. Without -c, it removes the temporary binary and scratch with its workspace. The language runtime owns individual @test functions.

Separate compilation is the only driver build path; no compatibility mode switch remains.

test/package/package_test.ww also owns the dual-stage declared_name_identity_and_file_import_scope observer. It generates temporary directory trees proving that canonical import identity, physical directory, declared package name, and source-file default qualifier remain distinct; imports are file-scoped while dependency edges are the package-wide sorted union; command and production/internal/external/generated-main variants retain canonical action ownership; vendor expansion changes identity but not the declared qualifier; compiler argv contains only direct .wwi inputs; and its named rejected actions leave neither committed nor staged action artifacts or a published binary. Together with the existing directory, recursive, vendor, exact-argv, command, and persistent-workdir observers, the package suite proves archive-only link argv and exact warm/rejection-state behavior without duplicating those broader mechanisms in this observer.

ww build, an explicit single-file ww test -o <stem>, and each successful directory-package ww test -c build publish <stem>.sepwork as a caller-owned artifact directory. The driver acquires it with one fresh mkdir and refuses an existing path; it never clears a collision. A caller keeps only the exact artifacts it observes and removes that exact tree on every later success or failure. ww run and no-output single-file ww test use driver-owned scratch instead; both driver stages place that scratch and their temporary executable beneath one freshly acquired directory, remove both after every build result, and make cleanup failure fail the command. Make recipes build driver-produced tools in invocation-owned directories and apply the same exact cleanup rule.

ww build -w DIR and single-file ww test -w DIR replace that scratch with a caller-owned persistent package-artifact workdir: for these direct routes the directory must already exist, is never cleaned by the driver, and holds one committed unit, .wwi, .s, .o, and dep .a per package plus byte copies of the invoking driver, compiler, and assembler and a small mode stamp. A package is reused only when its freshly composed owner unit byte-equals the committed unit, no recompiled direct dependency emitted changed export bytes, and every applicable recorded executable byte-equals the live executable — content identity only, no mtimes, no hashes, every decision reproducible with cmp against plain files. The driver identity covers graph construction, owner-unit composition, direct-export argument construction, archive creation, and commit algorithms that neither unit bytes nor compiler identity can name. Recompiled artifacts land at staged .new names and commit by rename with the unit renamed last, so an interrupted build forces a recompile rather than a false reuse; the link always reruns. One workdir serves one invocation at a time and may hold the complete command-global action universe for all selected roots and isolated variants; semantic identity, rather than request shape, controls reuse. Both driver stages implement the identical contract. This is build staleness in the Make/mk/Go sense, not a result cache: tests always run, and the byte-identity and bootstrap gates keep building on fresh scratch. make clean reclaims every workdir under out/.

On a package or tree target, ww build/test -w DIR forwards that exact caller path as the one command-global semantic-action store; neither pattern spelling nor package-group name derives a persistent subdirectory. The delegated driver may create a missing store only after graph and request preflight, and rolls back newly created empty prefixes if later setup fails. Every reuse decision stays with the driver's content-identity contract above. -w and test -c do not mix — the coordinator rejects the combination rather than guess which artifact tree the caller owns. test-library uses exactly this: one ww test -j $(JOBS) -w out/wwbuild/wwtest-lib lib/... line owns library behavior (measured: 2.3s cold, 1.0s warm at -j 4, vs 4.7s for the retired 59-target per-file fan-out).

test/lang currently uses one package per source file, so its complete gate still performs independent package builds. That remaining source layout is not hidden behind caching or concurrency; it is outside the small test target.

Compiler-only byte identity

ww build -S and ww test -S -o <stem> run source discovery, dependency ordering, unit composition, and each required w6c -c invocation. They return after the complete .s/.wwi set exists. The producer loop does not invoke w6a, create per-package archives, or invoke w6l when -S is active.

test-lang-byteid runs the 158 selected language files once with C w6c and once with WW w6c, requires identical emitted .s filename sets including __root.s, and compares those bytes. Relative to the old two-leg full builds, this removes at least 3,476 assembler launches, 3,160 archive writes, and 316 linker launches from the comparison loop. A cold Make invocation may still build prerequisite compiler binaries; the compiler-only claim applies to the per-language-file comparison path.

test-data-byteid applies the same comparator to the declarative corpus: every non-error case.ww builds twice through the fixed cstage driver with only WW_W6C swapped, and every emitted per-package .s must be byte-identical. //ww:error fixtures have no .s; their both-stage reject parity is owned by the fixture corpus itself. Known cs/ww divergences are pinned in DATABYTEID_DIVERGED with the 989_lib_byteid discipline: a pinned fixture must still build on both stages and still differ, so a compiler fix fails the gate demanding graduation rather than silently widening coverage. The full sweep compares the 1,157 non-error fixtures in about a minute and is scratch-rooted under out/, not /tmp.

Byte identity is an explicit proof gate. It is not a prerequisite of test or test-commit.

Bootstrap, subprocesses, and CSP

Stage-2-through-stage-4 fixed-point proofs and the 950/991–995 self-host gates are reachable through test-bootstrap and test-all, never through test or test-commit. Cold ordinary targets may still build their C- and WW-stage tool prerequisites once; they do not iterate those tools to a fixed point. The C bootstrap source and the standalone nocc route remain intact.

The bootstrap recipe owns the fixed out/bootstrap tree. Make schedules that target once within one invocation, but two independent make bootstrap invocations are not safe to run concurrently and remain mutually exclusive.

lib/os/exec is the sole reusable WW subprocess mechanism. Fixture and package coordinators use its captured asynchronous path. The WW driver directly uses os.exec.runstdio for inherited-stdio, inherited-environment, leader-only compiler, assembler, linker, cleanup, run, and single-file-test calls. The local WW procrun implementation is deleted. The C bootstrap retains its C process implementation because it cannot consume a WW standard-library module.

WW has tokens and an opaque type for future CSP/channel work, but no mature production channel operations, task runtime, or scheduler. No channel, goroutine, thread, worker-runtime, or CSP library was added. Coordinators stay single-threaded; OS process polling does not require language-level threads.

Retired mechanisms

The following are deleted, not adapted:

• test/run, including its phase classifier, xargs scheduler, atomic private records, timeout-text classifier, result collector, and last-green cache;
• test/run_test.sh, the synthetic shell tests for that protocol;
• every explicit TESTS registration and all 329 explicit wrapper rules;
• the unregistered test/runww.ww corpus runner;
• frozen duplicate compiler-corpus code under internal/wwtest, test/wwtest, and test/compiler;
• the redundant standalone smoke, test-run, and test-harness routes; and
• library-launcher wrappers whose only assertion was an existing @test source's exit status, including the declarative 900_stdlib.c launcher.

There is no test cache, daemon, scanner, generated manifest, database, new framework, compatibility API, concurrency runtime, dependency, or changed timeout policy in this architecture.

Open driver work

None; the package-level -o contract (the last carried bullet) landed as -c -o <name> for exactly one package.

Validation policy

Use test-unit as the inner loop for lexer, parser, checker, and narrow codegen changes. Run test for the ordinary local compiler check, followed by the focused owner for the changed behavior. Use test-commit for ordinary pre-commit behavior; use make -j4 JOBS=4 test-commit when parallel feedback is desired, and make -j1 JOBS=1 ... to reproduce failures deterministically. Run test-byteid and test-bootstrap only when those proof categories are intended. test-all is the exhaustive CI/release composition and should not be launched casually.