Hojun-Cho
f6628b985e
Audit-driven migrate-and-retire wave (one read-only auditor per carrier batch, verdicts row-checked through the full corpus gate): every retired carrier's assertions are now owned by declarative fixtures or an existing owner. 238 new fixtures land (corpus 1,239 -> 1,477; cells 2,954), covering enum/str/slice/tagged array elements, inferred-length arrays, global slice/str zero-init and literals, global array/pointer field reads and stores, struct-literal slice fields, struct returns, float arithmetic, size/int limits, sort and log-vstream behavior, place-addressed stores, and append places. Rejection rows STRENGTHEN the old nonzero-exit checks to required diagnostic fragments; all byte-id claims fold into the blanket test-data-byteid sweep (now 1,142 compared, 0 pinned-divergent). Four carriers were fully redundant with existing fixtures/lang tests (749, 722, 765, and 771's byteid rows) and retire without new rows. Native carriers 171 -> 147 (residual 135 -> 111).
280 lines
15 KiB
Markdown
280 lines
15 KiB
Markdown
# WW test architecture
|
||
|
||
Status: active architecture as of 2026-08-07. The target graph, corpus
|
||
accounting, and complete `test-commit` gate have been rechecked, including its
|
||
two concurrency controls separately and together. The separately gated
|
||
byte-identity and test-infrastructure proofs have also run. Bootstrap and
|
||
platform proofs have not run as part of this revision.
|
||
|
||
## Why the old suite was slow
|
||
|
||
The former suite made shell, Make, and one C executable per legacy case share
|
||
ownership of discovery, registration, phase selection, scheduling, result
|
||
records, timeout classification, and a last-green cache. Language behavior and
|
||
byte-identity loops repeatedly rebuilt complete package graphs. The measured
|
||
serial result was 411.403 seconds wall time, with a static lower bound of about
|
||
11,040 compiler-stage processes and 4,800 archive creations.
|
||
|
||
The replacement assigns each assertion to one owner and keeps expensive proof
|
||
categories out of the ordinary developer target.
|
||
|
||
## Owners
|
||
|
||
| Category | Owner |
|
||
| --- | --- |
|
||
| Arena, lexer, parser, checker, narrow codegen | Five in-process C unit binaries |
|
||
| Compile success/rejection, stage-routed diagnostics, and runtime exit | `test/wcc/data/*/case.ww`, executed by `wwfixture` |
|
||
| Package semantics | `test/package` and the native package-test coordinator |
|
||
| Language behavior | `test/lang/*_test.ww` through the language `@test` runtime |
|
||
| Library behavior | Library-owned `@test` sources named by `LIBRARY_TESTS` |
|
||
| Standalone library-source compilation | Four real source paths named by `LIBRARY_STANDALONE_SOURCES`, compiled directly by both frontends |
|
||
| Compiler-output identity | `test-lang-byteid` and `test-data-byteid` plus residual byte/artifact carriers |
|
||
| Fixed point and self-host | `test-bootstrap` |
|
||
| Host linker/platform behavior | `test-platform` |
|
||
|
||
The live declarative compiler corpus has 1,477 fixtures and 2,954 C/WW cells:
|
||
335 expected rejections (311 shared and 24 stage-specific), 17 compile-only
|
||
successes, 191 exit-zero programs, and 934 explicit-exit programs.
|
||
|
||
147 native C carriers remain. They are partitioned exactly once as five
|
||
in-process units, 24 byte/artifact gates, six bootstrap gates, one platform
|
||
gate, and 111 residual compiler, package-layout, ABI, diagnostic-observer,
|
||
driver, linker, or FFI gates. Rows migrated to fixtures or native `@test`
|
||
owners were removed from those carriers; there is no compatibility execution
|
||
path for retired rows. 61 former byte/artifact carriers whose only assertion
|
||
was a cstage-vs-wwstage `.s` byte-compare of corpus fixtures were retired
|
||
into the blanket `test-data-byteid` comparator; the survivors observe
|
||
something the blanket cannot: assembly patterns, symbol tables, frame
|
||
layouts, `.wwi` round-trips, inline sources with no corpus twin, or the
|
||
wwstage-driver leg. Wwstage-driver-leg byte identity (`ww_ww` versus `ww`
|
||
over the emitted `.s` set) has one owner, `989_lib_byteid`, whose unit
|
||
sweep spans lib test fixtures, import probes, and a zero-dep root-only
|
||
build; the former 815/940/951 driver-parity carriers were folded into it,
|
||
their content identity already owned by their corpus twins.
|
||
|
||
## Public targets
|
||
|
||
| Target | Composition |
|
||
| --- | --- |
|
||
| `test` | Five in-process units plus one compile-only C/WW compiler-fixture smoke case |
|
||
| `test-compiler` | Complete fixture corpus plus residual compiler/integration carriers |
|
||
| `test-package` | Package planning, grouping, routing, and package runtime only |
|
||
| `test-lang` | Language-owned `@test` behavior |
|
||
| `test-library` | Library-owned `@test` behavior plus four direct standalone-source C/WW compilation checks |
|
||
| `test-commit` | Unit + compiler + package + language + library behavior |
|
||
| `test-byteid` | Compiler-output identity gates |
|
||
| `test-bootstrap` | Fixed-point bootstrap plus the 950/991–995 native gates |
|
||
| `test-platform` | Host-dependent dynamic-link gate |
|
||
| `test-wwfixture` | Fixture CLI/process/protocol integration boundary |
|
||
| `test-all` | Commit + byte-ID + bootstrap + platform + test-infrastructure checks |
|
||
|
||
`test-commit` deliberately excludes byte identity, bootstrap, and platform
|
||
work. `test` is intentionally smaller than the old target and is the ordinary
|
||
developer feedback gate. Its purpose is a short, direct path from a compiler
|
||
edit to useful evidence, not compliance with an arbitrary wall-clock cutoff.
|
||
|
||
Make and fixture scheduling have separate, explicit owners. The Makefile does
|
||
not detect CPU count or add `-j` to `MAKEFLAGS`; the caller selects Make
|
||
parallelism with the standard `make -jN` option. `JOBS ?= 1` controls only the
|
||
`wwfixture -j N` value passed by `test-compiler`. A normal fast gate can use
|
||
both layers deliberately:
|
||
|
||
```sh
|
||
make -j4 JOBS=4 test-commit
|
||
```
|
||
|
||
For deterministic failure reproduction, make both layers serial explicitly:
|
||
|
||
```sh
|
||
make -j1 JOBS=1 test-commit
|
||
```
|
||
|
||
Serial execution is a debugging mode, not a correctness requirement. `JOBS`
|
||
is not inferred from `MAKEFLAGS`, and there is no jobserver adapter or second
|
||
scheduler hidden in Make.
|
||
|
||
`nocc` remains the separate, explicit reproduction route from a checked-in
|
||
stage-0 snapshot. It is not an implicit prerequisite of ordinary tests or of
|
||
`test-bootstrap`, because it has an external stage-0 precondition.
|
||
|
||
## Compiler fixtures
|
||
|
||
Each fixture is one directory with one `case.ww`. Its first line is exactly one
|
||
of:
|
||
|
||
```text
|
||
//ww:error "required diagnostic fragment"
|
||
//ww:error c "C-stage fragment" ww "WW-stage fragment"
|
||
//ww:compile
|
||
//ww:run
|
||
//ww:run-exit N
|
||
```
|
||
|
||
- `error` requires normal nonzero frontend termination and the declared stderr
|
||
fragment. The labeled form routes distinct fragments to the C and WW cells;
|
||
labels are fixed-order and neither fragment is treated as a shared fallback.
|
||
- `compile` requires frontend exit zero and produces no executable.
|
||
- `run` builds and requires normal program exit zero.
|
||
- `run-exit N` builds and requires normal program exit `N`.
|
||
|
||
Every fixture is run against the C and WW frontends. Signals, launch failures,
|
||
timeouts, build failures, and runtime exits are distinct outcomes.
|
||
`test-compiler` passes `-j $(JOBS)` and therefore uses one fixture slot by
|
||
default. The direct `wwfixture` CLI retains its own four-slot default; pass
|
||
`-j N` when its concurrency must be explicit. Its existing
|
||
`os.exec.start`/`poll` loop supervises the independent processes. Each cell has
|
||
its own working directory; filesystem fixtures use that directory or an
|
||
existing `temp.named` path rather than a shared fixed pathname.
|
||
|
||
The old `TESTS` list and all explicit per-wrapper Make rules are gone. A
|
||
surviving C carrier is registered only by its source file and built through one
|
||
generic pattern rule. Four arena/frontend units share a static rule; the codegen
|
||
unit links the existing private `cgen` and text-emitter objects directly.
|
||
|
||
`test/wwfixture/integration.sh` remains the direct black-box owner for behavior
|
||
that exists only at the command/process/protocol boundary: filtering and list
|
||
output, phase and outcome classification, diagnostic routing, malformed-corpus
|
||
and identity-drift rejection, signal/timeout/interruption cleanup, publication
|
||
failure, and strict result-stream decoding. Semantic fixtures cannot prove
|
||
those observations about their coordinator. `test/wwfixture/process/main.ww`
|
||
owns the lower-level `os.exec` primitives, not the CLI policy layered over
|
||
them, so it is complementary rather than a duplicate owner.
|
||
|
||
The five unit sources have zero active `system`, `popen`, `fork`, or `exec`
|
||
calls. Before direct conversion they contained two `popen` call sites and a full
|
||
`test-unit` run launched one `ww -V` plus twelve `w6c` processes. `400_w6c` now
|
||
checks the same twelve assembly fragments in process, grouped under ten unique
|
||
sources. The CLI version assertion lives with C/WW-driver parity in the existing
|
||
`949_driver_flagargs` integration carrier; `000_smoke` retains its arena-growth
|
||
and nonempty-version-constant assertions.
|
||
|
||
## Package and language behavior
|
||
|
||
`ww test` delegates directory package requests to the native package
|
||
coordinator. The coordinator owns discovery, package grouping, same-package and
|
||
external-package test composition, filtering, result aggregation, and its
|
||
internal temporary workspace. With `-c`, it publishes each exact
|
||
`<package>.test` binary and adjacent `<package>.test.sepwork` tree in the
|
||
package directory; those become caller-owned artifacts. Without `-c`, it
|
||
removes the temporary binary and scratch with its workspace. The language
|
||
runtime owns individual `@test` functions.
|
||
|
||
Separate compilation is the only driver build path; no compatibility mode
|
||
switch remains.
|
||
|
||
`ww build`, an explicit single-file `ww test -o <stem>`, and each successful
|
||
directory-package `ww test -c` build publish `<stem>.sepwork` as a caller-owned
|
||
artifact directory. The driver acquires it with one fresh `mkdir` and refuses
|
||
an existing path; it never clears a collision. A caller keeps only the exact
|
||
artifacts it observes and removes that exact tree on every later success or
|
||
failure. `ww run` and no-output single-file `ww test` use driver-owned scratch
|
||
instead; both driver stages place that scratch and their temporary executable
|
||
beneath one freshly acquired directory, remove both after every build result,
|
||
and make cleanup failure fail the command. Make recipes build driver-produced
|
||
tools in invocation-owned directories and apply the same exact cleanup rule.
|
||
|
||
`test/lang` currently uses one package per source file, so its complete gate
|
||
still performs independent package builds. That remaining source layout is not
|
||
hidden behind caching or concurrency; it is outside the small `test` target.
|
||
|
||
## Compiler-only byte identity
|
||
|
||
`ww build -S` and `ww test -S -o <stem>` run source discovery, dependency
|
||
ordering, unit composition, and each required `w6c -c` invocation. They return
|
||
after the complete `.s`/`.wwi` set exists. The producer loop does not invoke
|
||
`w6a`, create per-package archives, or invoke `w6l` when `-S` is active.
|
||
|
||
`test-lang-byteid` runs the 158 selected language files once with C `w6c` and
|
||
once with WW `w6c`, requires identical emitted `.s` filename sets including
|
||
`__root.s`, and compares those bytes. Relative to the old two-leg full builds,
|
||
this removes at least 3,476 assembler launches, 3,160 archive writes, and 316
|
||
linker launches from the comparison loop. A cold Make invocation may still
|
||
build prerequisite compiler binaries; the compiler-only claim applies to the
|
||
per-language-file comparison path.
|
||
|
||
`test-data-byteid` applies the same comparator to the declarative corpus:
|
||
every non-error `case.ww` builds twice through the fixed cstage driver with
|
||
only `WW_W6C` swapped, and every emitted per-package `.s` must be
|
||
byte-identical. `//ww:error` fixtures have no `.s`; their both-stage reject
|
||
parity is owned by the fixture corpus itself. Known cs/ww divergences are
|
||
pinned in `DATABYTEID_DIVERGED` with the `989_lib_byteid` discipline: a
|
||
pinned fixture must still build on both stages and still differ, so a
|
||
compiler fix fails the gate demanding graduation rather than silently
|
||
widening coverage. The full sweep compares 915 fixtures in about a minute
|
||
and is scratch-rooted under `out/`, not `/tmp`.
|
||
|
||
Byte identity is an explicit proof gate. It is not a prerequisite of `test` or
|
||
`test-commit`.
|
||
|
||
## Bootstrap, subprocesses, and CSP
|
||
|
||
Stage-2-through-stage-4 fixed-point proofs and the 950/991–995 self-host gates
|
||
are reachable through `test-bootstrap` and `test-all`, never through `test` or
|
||
`test-commit`. Cold ordinary targets may still build their C- and WW-stage tool
|
||
prerequisites once; they do not iterate those tools to a fixed point. The C
|
||
bootstrap source and the standalone `nocc` route remain intact.
|
||
|
||
The bootstrap recipe owns the fixed `out/bootstrap` tree. Make schedules that
|
||
target once within one invocation, but two independent `make bootstrap`
|
||
invocations are not safe to run concurrently and remain mutually exclusive.
|
||
|
||
`lib/os/exec` is the sole reusable WW subprocess mechanism. Fixture and package
|
||
coordinators use its captured asynchronous path. The WW driver directly uses
|
||
`os.exec.runstdio` for inherited-stdio, inherited-environment, leader-only
|
||
compiler, assembler, linker, cleanup, run, and single-file-test calls. The
|
||
local WW `procrun` implementation is deleted. The C bootstrap retains its C
|
||
process implementation because it cannot consume a WW standard-library
|
||
module.
|
||
|
||
WW has tokens and an opaque type for future CSP/channel work, but no mature
|
||
production channel operations, task runtime, or scheduler. No channel,
|
||
goroutine, thread, worker-runtime, or CSP library was added. Coordinators stay
|
||
single-threaded; OS process polling does not require language-level threads.
|
||
|
||
## Retired mechanisms
|
||
|
||
The following are deleted, not adapted:
|
||
|
||
- `test/run`, including its phase classifier, `xargs` scheduler, atomic private
|
||
records, timeout-text classifier, result collector, and last-green cache;
|
||
- `test/run_test.sh`, the synthetic shell tests for that protocol;
|
||
- every explicit `TESTS` registration and all 329 explicit wrapper rules;
|
||
- the unregistered `test/runww.ww` corpus runner;
|
||
- frozen duplicate compiler-corpus code under `internal/wwtest`, `test/wwtest`,
|
||
and `test/compiler`;
|
||
- the redundant standalone `smoke`, `test-run`, and `test-harness` routes; and
|
||
- library-launcher wrappers whose only assertion was an existing `@test`
|
||
source's exit status, including the declarative `900_stdlib.c` launcher.
|
||
|
||
There is no test cache, daemon, scanner, generated manifest, database, new
|
||
framework, compatibility API, concurrency runtime, dependency, or changed
|
||
timeout policy in this architecture.
|
||
|
||
## Open driver work
|
||
|
||
Carried over from the dissolved project plan; each is deliberate scope, not
|
||
drift:
|
||
|
||
- Recursive package discovery (a `./...`-equivalent pattern) for `ww test`.
|
||
- Package-level concurrency for the coordinator's builds (it is
|
||
single-threaded by design today).
|
||
- A package-level `-o` contract (single-file `ww test -o` exists; directory
|
||
packages publish fixed `<package>.test` stems under `-c`).
|
||
- Universal `*_test.ww` filename migration for library tests (several
|
||
library suites still use the older `*test.ww` names, enumerated by
|
||
`LIBRARY_TESTS` and the `989_lib_byteid` roster).
|
||
- The remaining invalid-`@test` attribute-shape diagnostics (exported
|
||
tests, prototypes, arguments, variadics, non-void returns, duplicate
|
||
annotations) with stable text in both frontends.
|
||
|
||
## Validation policy
|
||
|
||
Use `test-unit` as the inner loop for lexer, parser, checker, and narrow codegen
|
||
changes. Run `test` for the ordinary local compiler check, followed by the
|
||
focused owner for the changed behavior. Use `test-commit` for ordinary
|
||
pre-commit behavior; use `make -j4 JOBS=4 test-commit` when parallel feedback is
|
||
desired, and `make -j1 JOBS=1 ...` to reproduce failures deterministically.
|
||
Run `test-byteid` and `test-bootstrap` only when those proof categories are
|
||
intended. `test-all` is the exhaustive CI/release composition and should not be
|
||
launched casually.
|