Hojun-Cho
659e859f34
Go compiles only _test.go files as tests; discovery now keys on the
_test.ww suffix alone. The line-leading-@test compatibility allowance
(noncanonical filenames admitted as test sources) is removed from both
driver stages and the coordinator. An @test declaration outside a
*_test.ww file is rejected loudly ("@test declaration outside
*_test.ww", wording byte-identical cs/ww) instead of silently running
under compose or silently dropping in a non-T build (#6). Tree audit
found zero real carriers; the two allowance fixtures flip canonical
(dep_test.ww, widget_test.ww). New pins: direnum attest-noncanon
reject row (both-stage stderr parity) and the coordinator
noncanonical_attest_rejected package row.
354 lines
19 KiB
Markdown
354 lines
19 KiB
Markdown
# WW test architecture
|
||
|
||
Status: active architecture as of 2026-08-07. The target graph, corpus
|
||
accounting, and complete `test-commit` gate have been rechecked, including its
|
||
two concurrency controls separately and together. The separately gated
|
||
byte-identity and test-infrastructure proofs have also run. Bootstrap and
|
||
platform proofs have not run as part of this revision.
|
||
|
||
## Why the old suite was slow
|
||
|
||
The former suite made shell, Make, and one C executable per legacy case share
|
||
ownership of discovery, registration, phase selection, scheduling, result
|
||
records, timeout classification, and a last-green cache. Language behavior and
|
||
byte-identity loops repeatedly rebuilt complete package graphs. The measured
|
||
serial result was 411.403 seconds wall time, with a static lower bound of about
|
||
11,040 compiler-stage processes and 4,800 archive creations.
|
||
|
||
The replacement assigns each assertion to one owner and keeps expensive proof
|
||
categories out of the ordinary developer target.
|
||
|
||
## Owners
|
||
|
||
| Category | Owner |
|
||
| --- | --- |
|
||
| Arena, lexer, parser, checker, module-decl parse, narrow codegen | Six in-process C unit binaries |
|
||
| Compile success/rejection, stage-routed diagnostics, and runtime exit | `test/wcc/data/*/case.ww`, executed by `wwfixture` |
|
||
| Package semantics | `test/package` and the native package-test coordinator |
|
||
| Language behavior | `test/lang/*_test.ww` through the language `@test` runtime |
|
||
| Library behavior | The package coordinator's `lib/...` tree walk (`ww test -j N -I lib/ww lib/...`) |
|
||
| Standalone library-source compilation | Four real source paths named by `LIBRARY_STANDALONE_SOURCES`, compiled directly by both frontends |
|
||
| Compiler-output identity | `test-lang-byteid` and `test-data-byteid` plus residual byte/artifact carriers |
|
||
| Fixed point and self-host | `test-bootstrap` |
|
||
| Host linker/platform behavior | `test-platform` |
|
||
|
||
The live declarative compiler corpus has 1,745 fixtures and 3,490 C/WW cells:
|
||
346 expected rejections, 21 compile-only successes, 209 exit-zero
|
||
programs, and 1,169 explicit-exit programs.
|
||
|
||
15 native C carriers remain. They are partitioned exactly once into
|
||
the two terminal classes: six in-process units (`738_module_decl`
|
||
re-homed from the residuals — Go does not distinguish "bootstrap
|
||
observers" from unit tests, and its parsefile-call shape is exactly
|
||
the `200_parse` class), six bootstrap gates, and three platform-claim
|
||
carriers (`996_dyn_ww` under `test-platform`; `810_dyn` and
|
||
`989_dynentry_run` hold host loader/glibc claims and run with
|
||
`test-compiler`). The residual class is EMPTY: `794_xmod_ident_prefer`
|
||
— the last bug-pinned survivor — retired with its #55-sibling cgen
|
||
fix (its assertions migrated to `test/xmod/collide_test.ww`).
|
||
|
||
The 2026-08-08 residual drain retired the other 107: 5 deleted against
|
||
verified owners, 196 rows migrated into corpus fixtures (700_e2e's 136,
|
||
the fixture-class 46, the 926/929/671 run legs), and the rest ported to
|
||
ww-native observer suites on `test/testenv`, each carrier retired in
|
||
the same commit as its replacement: `test/sep/` (sep-layout/driver),
|
||
`test/xmod/` (module-tree collisions, mangle, typecheck, direnum),
|
||
`test/asm/` (positional asm-window needle observers), `test/object/`
|
||
(w6a/w6l ELF, archive, and link facts), `test/misc/` (divergence pins,
|
||
stamp probes, wwi round-trips), `test/tool/` (driver CLI, wwdump
|
||
gates, FFI, the c6 soak), and `test/libenv/` (the lib/os + lib/dirs
|
||
env-arranger legs; those lib suites now pass bare with loud SKIPs).
|
||
All suites run under `test-compiler`. The byte/artifact partition
|
||
stays EMPTY under `test/byteid/` exactly as before; wwstage-driver-leg
|
||
byte identity keeps its one owner, `test/byteid/libbyteid_test.ww`
|
||
(44-entry roster plus the lib/ completeness scan). Known cs≠ww
|
||
divergences exposed by row migration are pinned in
|
||
`DATABYTEID_DIVERGED` (3 entries) on the graduation discipline.
|
||
|
||
## Carrier endgame
|
||
|
||
The declarative corpus and the native `@test` owners are the permanent
|
||
test surface — the `go/test/` analogy. The native C carrier fleet is the
|
||
pre-Go-1.5 artifact and shrinks toward exactly two terminal classes:
|
||
|
||
1. **C-bootstrap observers.** The six in-process units watch the C
|
||
frontend from inside its own process and are irreplaceable while
|
||
`cmd/` is the live frontend. The six bootstrap gates are the same
|
||
class's proof column: they compare the wwstage tools against the C
|
||
stage to a fixed point. At the eventual selfhost flip both freeze
|
||
into a bootstrap smoke gate (Go 1.5 deleted its C toolchain and that
|
||
toolchain's tests; it did not port them).
|
||
2. **Host ABI/platform gates.** Behavior owned by the host linker,
|
||
loader, or ABI (`996_dyn_ww` today). These observe the platform, not
|
||
the compiler, and stay native exactly as long as the claim is about
|
||
the platform.
|
||
|
||
Everything else gets a ww owner. Byte and artifact observations —
|
||
assembly-pattern greps, symbol tables, frame layouts, `.wwi`
|
||
round-trips, driver-leg comparisons — are subprocess plus file IO plus
|
||
string search, which `test/package/package_test.ww` already performs
|
||
natively (runcommand + in-language assertions). A C carrier whose
|
||
assertions fit that shape is ported and retired in the same commit,
|
||
assertions preserved or strengthened, with no compatibility execution
|
||
path left behind. A carrier that is merely historical is deleted
|
||
outright; git history is the archive.
|
||
|
||
## Public targets
|
||
|
||
| Target | Composition |
|
||
| --- | --- |
|
||
| `test` | Six in-process units plus one compile-only C/WW compiler-fixture smoke case |
|
||
| `test-compiler` | Complete fixture corpus plus residual compiler/integration carriers |
|
||
| `test-package` | Package planning, grouping, routing, and package runtime only |
|
||
| `test-lang` | Language-owned `@test` behavior |
|
||
| `test-library` | The coordinator's `lib/...` walk plus four direct standalone-source C/WW compilation checks |
|
||
| `test-commit` | Unit + compiler + package + language + library behavior |
|
||
| `test-byteid` | Compiler-output identity gates |
|
||
| `test-bootstrap` | Fixed-point bootstrap plus the 950/991–995 native gates |
|
||
| `test-platform` | Host-dependent dynamic-link gate |
|
||
| `test-wwfixture` | Fixture CLI/process/protocol integration boundary |
|
||
| `test-all` | Commit + byte-ID + bootstrap + platform + test-infrastructure checks |
|
||
|
||
`test-commit` deliberately excludes byte identity, bootstrap, and platform
|
||
work. `test` is intentionally smaller than the old target and is the ordinary
|
||
developer feedback gate. Its purpose is a short, direct path from a compiler
|
||
edit to useful evidence, not compliance with an arbitrary wall-clock cutoff.
|
||
|
||
Make and fixture scheduling have separate, explicit owners. The Makefile does
|
||
not detect CPU count or add `-j` to `MAKEFLAGS`; the caller selects Make
|
||
parallelism with the standard `make -jN` option. `JOBS ?= 1` controls only the
|
||
`wwfixture -j N` value passed by `test-compiler`. A normal fast gate can use
|
||
both layers deliberately:
|
||
|
||
```sh
|
||
make -j4 JOBS=4 test-commit
|
||
```
|
||
|
||
For deterministic failure reproduction, make both layers serial explicitly:
|
||
|
||
```sh
|
||
make -j1 JOBS=1 test-commit
|
||
```
|
||
|
||
Serial execution is a debugging mode, not a correctness requirement. `JOBS`
|
||
is not inferred from `MAKEFLAGS`, and there is no jobserver adapter or second
|
||
scheduler hidden in Make.
|
||
|
||
`nocc` remains the separate, explicit reproduction route from a checked-in
|
||
stage-0 snapshot. It is not an implicit prerequisite of ordinary tests or of
|
||
`test-bootstrap`, because it has an external stage-0 precondition.
|
||
|
||
## Compiler fixtures
|
||
|
||
Each fixture is one directory with one `case.ww`. Its first line is exactly one
|
||
of:
|
||
|
||
```text
|
||
//ww:error "required diagnostic fragment"
|
||
//ww:error c "C-stage fragment" ww "WW-stage fragment"
|
||
//ww:compile
|
||
//ww:run
|
||
//ww:run-exit N
|
||
```
|
||
|
||
- `error` requires normal nonzero frontend termination and the declared stderr
|
||
fragment. The labeled form routes distinct fragments to the C and WW cells;
|
||
labels are fixed-order and neither fragment is treated as a shared fallback.
|
||
- `compile` requires frontend exit zero and produces no executable.
|
||
- `run` builds and requires normal program exit zero.
|
||
- `run-exit N` builds and requires normal program exit `N`.
|
||
|
||
Every fixture is run against the C and WW frontends. Signals, launch failures,
|
||
timeouts, build failures, and runtime exits are distinct outcomes.
|
||
`test-compiler` passes `-j $(JOBS)` and therefore uses one fixture slot by
|
||
default. The direct `wwfixture` CLI retains its own four-slot default; pass
|
||
`-j N` when its concurrency must be explicit. Its existing
|
||
`os.exec.start`/`poll` loop supervises the independent processes. Each cell has
|
||
its own working directory; filesystem fixtures use that directory or an
|
||
existing `temp.named` path rather than a shared fixed pathname.
|
||
|
||
The old `TESTS` list and all explicit per-wrapper Make rules are gone. A
|
||
surviving C carrier is registered only by its source file and built through one
|
||
generic pattern rule. Four arena/frontend units share a static rule; the codegen
|
||
unit links the existing private `cgen` and text-emitter objects directly.
|
||
|
||
`test/wwfixture/integration.sh` remains the direct black-box owner for behavior
|
||
that exists only at the command/process/protocol boundary: filtering and list
|
||
output, phase and outcome classification, diagnostic routing, malformed-corpus
|
||
and identity-drift rejection, signal/timeout/interruption cleanup, publication
|
||
failure, and strict result-stream decoding. Semantic fixtures cannot prove
|
||
those observations about their coordinator. `test/wwfixture/process/main.ww`
|
||
owns the lower-level `os.exec` primitives, not the CLI policy layered over
|
||
them, so it is complementary rather than a duplicate owner.
|
||
|
||
The six unit sources have zero active `system`, `popen`, `fork`, or `exec`
|
||
calls. Before direct conversion they contained two `popen` call sites and a full
|
||
`test-unit` run launched one `ww -V` plus twelve `w6c` processes. `400_w6c` now
|
||
checks the same twelve assembly fragments in process, grouped under ten unique
|
||
sources. The CLI version assertion lives with C/WW-driver parity in the existing
|
||
`949_driver_flagargs` integration carrier; `000_smoke` retains its arena-growth
|
||
and nonempty-version-constant assertions.
|
||
|
||
## Package and language behavior
|
||
|
||
`ww test` delegates directory package requests to the native package
|
||
coordinator. The coordinator owns discovery, package grouping, same-package and
|
||
external-package test composition, filtering, result aggregation, and its
|
||
internal temporary workspace. Test sources are exclusively `*_test.ww` (Go's
|
||
`_test.go` contract): a line-leading `@test` declaration in any other source is
|
||
rejected loudly — by both driver stages at directory enumeration and by the
|
||
coordinator at source classification. A trailing `...` path element (`ww test lib/...`,
|
||
Go's `./...` form) is recognized by both driver stages before path resolution
|
||
and walks the tree rooted at the prefix: every subdirectory whose name does not
|
||
begin with `.` or `_` is descended with the same lstat/no-symlink discipline,
|
||
each test-bearing directory becomes one package run, and source-bearing
|
||
directories without tests report the usual `?` line. The coordinator's
|
||
`-j N` schedules up to N package groups concurrently (build-then-run
|
||
process chains supervised with `os.exec` start/poll, no threads);
|
||
emission stays strictly in group order, so the byte stream is identical
|
||
at every `-j` level, and `-j 1` — the default — matches the former
|
||
sequential loop exactly. Measured on the 31-package `lib/...` walk:
|
||
7.0s sequential, 2.4s at `-j 4`. With `-c`, it publishes each exact
|
||
`<package>.test` binary and adjacent `<package>.test.sepwork` tree in the
|
||
package directory; those become caller-owned artifacts. `-c -o <name>`
|
||
names that artifact instead of the fixed stem, for exactly one package:
|
||
the coordinator rejects a multi-package fan-out ("cannot use -o with
|
||
multiple packages", Go's `go test -o` rule), and `-o` without `-c` is
|
||
rejected at the driver ("needs -c for a package target") because a plain
|
||
run always executes from the temp root. Without `-c`, it removes the
|
||
temporary binary and scratch with its workspace. The language runtime
|
||
owns individual `@test` functions.
|
||
|
||
Separate compilation is the only driver build path; no compatibility mode
|
||
switch remains.
|
||
|
||
`ww build`, an explicit single-file `ww test -o <stem>`, and each successful
|
||
directory-package `ww test -c` build publish `<stem>.sepwork` as a caller-owned
|
||
artifact directory. The driver acquires it with one fresh `mkdir` and refuses
|
||
an existing path; it never clears a collision. A caller keeps only the exact
|
||
artifacts it observes and removes that exact tree on every later success or
|
||
failure. `ww run` and no-output single-file `ww test` use driver-owned scratch
|
||
instead; both driver stages place that scratch and their temporary executable
|
||
beneath one freshly acquired directory, remove both after every build result,
|
||
and make cleanup failure fail the command. Make recipes build driver-produced
|
||
tools in invocation-owned directories and apply the same exact cleanup rule.
|
||
|
||
`ww build -w DIR` and single-file `ww test -w DIR` replace that scratch with a
|
||
caller-owned persistent package-artifact workdir: the directory must already
|
||
exist, is never cleaned by the driver, and holds one committed unit, `.wwi`,
|
||
`.s`, `.o`, and dep `.a` per package plus byte copies of the compiler and
|
||
assembler and a small mode stamp. A package is reused only when its freshly
|
||
composed unit byte-equals the committed unit and the recorded tool copies
|
||
byte-equal the live tools — content identity only, no mtimes, no hashes, every
|
||
decision reproducible with `cmp` against plain files. Recompiled artifacts
|
||
land at staged `.new` names and commit by rename with the unit renamed last,
|
||
so an interrupted build forces a recompile rather than a false reuse; the
|
||
link always reruns. One workdir serves one invocation at a time and one
|
||
(root, mode) shape; both driver stages implement the identical contract.
|
||
This is build staleness in the Make/mk/Go sense, not a result cache: tests
|
||
always run, and the byte-identity and bootstrap gates keep building on fresh
|
||
scratch. `make clean` reclaims every workdir under `out/`.
|
||
|
||
On a package or tree target, `ww test -w DIR` forwards to the coordinator,
|
||
which keys one persistent driver workdir per package group under `DIR`
|
||
(`<dir>_<pkg>` with `/` flattened) and passes it to each inner build; every
|
||
reuse decision stays with the driver's content-identity contract above.
|
||
`-w` and `-c` do not mix there — the coordinator rejects the combination
|
||
rather than guess which artifact tree the caller owns. `test-library` uses
|
||
exactly this: one `ww test -j $(JOBS) -w out/wwbuild/wwtest-lib -I lib/ww
|
||
lib/...` line owns library behavior (measured: 2.3s cold, 1.0s warm at
|
||
`-j 4`, vs 4.7s for the retired 59-target per-file fan-out).
|
||
|
||
`test/lang` currently uses one package per source file, so its complete gate
|
||
still performs independent package builds. That remaining source layout is not
|
||
hidden behind caching or concurrency; it is outside the small `test` target.
|
||
|
||
## Compiler-only byte identity
|
||
|
||
`ww build -S` and `ww test -S -o <stem>` run source discovery, dependency
|
||
ordering, unit composition, and each required `w6c -c` invocation. They return
|
||
after the complete `.s`/`.wwi` set exists. The producer loop does not invoke
|
||
`w6a`, create per-package archives, or invoke `w6l` when `-S` is active.
|
||
|
||
`test-lang-byteid` runs the 158 selected language files once with C `w6c` and
|
||
once with WW `w6c`, requires identical emitted `.s` filename sets including
|
||
`__root.s`, and compares those bytes. Relative to the old two-leg full builds,
|
||
this removes at least 3,476 assembler launches, 3,160 archive writes, and 316
|
||
linker launches from the comparison loop. A cold Make invocation may still
|
||
build prerequisite compiler binaries; the compiler-only claim applies to the
|
||
per-language-file comparison path.
|
||
|
||
`test-data-byteid` applies the same comparator to the declarative corpus:
|
||
every non-error `case.ww` builds twice through the fixed cstage driver with
|
||
only `WW_W6C` swapped, and every emitted per-package `.s` must be
|
||
byte-identical. `//ww:error` fixtures have no `.s`; their both-stage reject
|
||
parity is owned by the fixture corpus itself. Known cs/ww divergences are
|
||
pinned in `DATABYTEID_DIVERGED` with the `989_lib_byteid` discipline: a
|
||
pinned fixture must still build on both stages and still differ, so a
|
||
compiler fix fails the gate demanding graduation rather than silently
|
||
widening coverage. The full sweep compares the 1,157 non-error fixtures
|
||
in about a minute and is scratch-rooted under `out/`, not `/tmp`.
|
||
|
||
Byte identity is an explicit proof gate. It is not a prerequisite of `test` or
|
||
`test-commit`.
|
||
|
||
## Bootstrap, subprocesses, and CSP
|
||
|
||
Stage-2-through-stage-4 fixed-point proofs and the 950/991–995 self-host gates
|
||
are reachable through `test-bootstrap` and `test-all`, never through `test` or
|
||
`test-commit`. Cold ordinary targets may still build their C- and WW-stage tool
|
||
prerequisites once; they do not iterate those tools to a fixed point. The C
|
||
bootstrap source and the standalone `nocc` route remain intact.
|
||
|
||
The bootstrap recipe owns the fixed `out/bootstrap` tree. Make schedules that
|
||
target once within one invocation, but two independent `make bootstrap`
|
||
invocations are not safe to run concurrently and remain mutually exclusive.
|
||
|
||
`lib/os/exec` is the sole reusable WW subprocess mechanism. Fixture and package
|
||
coordinators use its captured asynchronous path. The WW driver directly uses
|
||
`os.exec.runstdio` for inherited-stdio, inherited-environment, leader-only
|
||
compiler, assembler, linker, cleanup, run, and single-file-test calls. The
|
||
local WW `procrun` implementation is deleted. The C bootstrap retains its C
|
||
process implementation because it cannot consume a WW standard-library
|
||
module.
|
||
|
||
WW has tokens and an opaque type for future CSP/channel work, but no mature
|
||
production channel operations, task runtime, or scheduler. No channel,
|
||
goroutine, thread, worker-runtime, or CSP library was added. Coordinators stay
|
||
single-threaded; OS process polling does not require language-level threads.
|
||
|
||
## Retired mechanisms
|
||
|
||
The following are deleted, not adapted:
|
||
|
||
- `test/run`, including its phase classifier, `xargs` scheduler, atomic private
|
||
records, timeout-text classifier, result collector, and last-green cache;
|
||
- `test/run_test.sh`, the synthetic shell tests for that protocol;
|
||
- every explicit `TESTS` registration and all 329 explicit wrapper rules;
|
||
- the unregistered `test/runww.ww` corpus runner;
|
||
- frozen duplicate compiler-corpus code under `internal/wwtest`, `test/wwtest`,
|
||
and `test/compiler`;
|
||
- the redundant standalone `smoke`, `test-run`, and `test-harness` routes; and
|
||
- library-launcher wrappers whose only assertion was an existing `@test`
|
||
source's exit status, including the declarative `900_stdlib.c` launcher.
|
||
|
||
There is no test cache, daemon, scanner, generated manifest, database, new
|
||
framework, compatibility API, concurrency runtime, dependency, or changed
|
||
timeout policy in this architecture.
|
||
|
||
## Open driver work
|
||
|
||
None; the package-level `-o` contract (the last carried bullet) landed as
|
||
`-c -o <name>` for exactly one package.
|
||
|
||
## Validation policy
|
||
|
||
Use `test-unit` as the inner loop for lexer, parser, checker, and narrow codegen
|
||
changes. Run `test` for the ordinary local compiler check, followed by the
|
||
focused owner for the changed behavior. Use `test-commit` for ordinary
|
||
pre-commit behavior; use `make -j4 JOBS=4 test-commit` when parallel feedback is
|
||
desired, and `make -j1 JOBS=1 ...` to reproduce failures deterministically.
|
||
Run `test-byteid` and `test-bootstrap` only when those proof categories are
|
||
intended. `test-all` is the exhaustive CI/release composition and should not be
|
||
launched casually.
|