Files

Hojun-Cho 78c9eee82d ww imports: reject init bindings before recovery

2026-08-22 04:27:23 +09:00

53 KiB

Raw Blame History

WW test architecture

Status: active architecture as of 2026-08-07. The target graph, corpus accounting, and complete test-commit gate have been rechecked, including its two concurrency controls separately and together. The separately gated byte-identity and test-infrastructure proofs have also run. Bootstrap and platform proofs have not run as part of this revision.

Why the old suite was slow

The former suite made shell, Make, and one C executable per legacy case share ownership of discovery, registration, phase selection, scheduling, result records, timeout classification, and a last-green cache. Language behavior and byte-identity loops repeatedly rebuilt complete package graphs. The measured serial result was 411.403 seconds wall time, with a static lower bound of about 11,040 compiler-stage processes and 4,800 archive creations.

The replacement assigns each assertion to one owner and keeps expensive proof categories out of the ordinary developer target.

Owners

Category	Owner
Arena, lexer, parser, checker, module-decl parse, narrow codegen	Six in-process C unit binaries
Compile success/rejection, stage-routed diagnostics, and runtime exit	`test/wcc/data/*/case.ww`, executed by `wwfixture`
Package semantics	`test/package`, `test/sep/sepinit_test.ww`, and the native package-test coordinator
Language behavior	`test/lang/*_test.ww` through the language `@test` runtime
Library behavior	The package coordinator's `lib/...` tree walk (`ww test -j N lib/...`)
Standalone library-source compilation	Three import-free real source paths named by `LIBRARY_STANDALONE_SOURCES`, compiled directly by both frontends
Compiler-output identity	`test-lang-byteid` and `test-data-byteid` plus residual byte/artifact carriers
Fixed point and self-host	`test-bootstrap`
Host linker/platform behavior	`test-platform`

The live declarative compiler corpus has 1,759 fixtures and 3,518 C/WW cells: 351 expected rejections, 22 compile-only successes, 209 exit-zero programs, and 1,177 explicit-exit programs.

15 native C carriers remain. They are partitioned exactly once into the two terminal classes: six in-process units (738_module_decl re-homed from the residuals — Go does not distinguish "bootstrap observers" from unit tests, and its parsefile-call shape is exactly the 200_parse class), six bootstrap gates, and three platform-claim carriers (996_dyn_ww under test-platform; 810_dyn and 989_dynentry_run hold host loader/glibc claims and run with test-compiler). The residual class is EMPTY: 794_xmod_ident_prefer — the last bug-pinned survivor — retired with its #55-sibling cgen fix (its assertions migrated to test/xmod/collide_test.ww).

The 2026-08-08 residual drain retired the other 107: 5 deleted against verified owners, 196 rows migrated into corpus fixtures (700_e2e's 136, the fixture-class 46, the 926/929/671 run legs), and the rest ported to ww-native observer suites on test/testenv, each carrier retired in the same commit as its replacement: test/sep/ (sep-layout/driver), test/xmod/ (module-tree collisions, mangle, typecheck, direnum), test/asm/ (positional asm-window needle observers), test/object/ (w6a/w6l ELF, archive, and link facts), test/misc/ (divergence pins, stamp probes, wwi round-trips), test/tool/ (driver CLI, wwdump gates, FFI, the c6 soak), and test/libenv/ (the lib/os + lib/dirs env-arranger legs; those lib suites now pass bare with loud SKIPs). All suites run under test-compiler. The byte/artifact partition stays EMPTY under test/byteid/ exactly as before; wwstage-driver-leg byte identity keeps its one owner, test/byteid/libbyteid_test.ww (44-entry roster plus the lib/ completeness scan). Known cs≠ww divergences exposed by row migration are pinned in DATABYTEID_DIVERGED (3 entries) on the graduation discipline.

Carrier endgame

The declarative corpus and the native @test owners are the permanent test surface — the go/test/ analogy. The native C carrier fleet is the pre-Go-1.5 artifact and shrinks toward exactly two terminal classes:

C-bootstrap observers. The six in-process units watch the C frontend from inside its own process and are irreplaceable while cmd/ is the live frontend. The six bootstrap gates are the same class's proof column: they compare the wwstage tools against the C stage to a fixed point. At the eventual selfhost flip both freeze into a bootstrap smoke gate (Go 1.5 deleted its C toolchain and that toolchain's tests; it did not port them).
Host ABI/platform gates. Behavior owned by the host linker, loader, or ABI (996_dyn_ww today). These observe the platform, not the compiler, and stay native exactly as long as the claim is about the platform.

Everything else gets a ww owner. Byte and artifact observations — assembly-pattern greps, symbol tables, frame layouts, .wwi round-trips, driver-leg comparisons — are subprocess plus file IO plus string search, which test/package/package_test.ww already performs natively (runcommand + in-language assertions). A C carrier whose assertions fit that shape is ported and retired in the same commit, assertions preserved or strengthened, with no compatibility execution path left behind. A carrier that is merely historical is deleted outright; git history is the archive.

Public targets

Target	Composition
`test`	Six in-process units plus one compile-only C/WW compiler-fixture smoke case
`test-compiler`	Complete fixture corpus plus residual compiler/integration carriers
`test-package`	Package planning, grouping, routing, and package runtime only
`test-lang`	Language-owned `@test` behavior
`test-library`	The coordinator's `lib/...` walk plus three import-free standalone-source C/WW compilation checks
`test-commit`	Unit + compiler + package + language + library behavior
`test-byteid`	Compiler-output identity gates
`test-bootstrap`	Fixed-point bootstrap plus the 950/991–995 native gates
`test-platform`	Host-dependent dynamic-link gate
`test-wwfixture`	Fixture CLI/process/protocol integration boundary
`test-all`	Commit + byte-ID + bootstrap + platform + test-infrastructure checks

test-commit deliberately excludes byte identity, bootstrap, and platform work. test is intentionally smaller than the old target and is the ordinary developer feedback gate. Its purpose is a short, direct path from a compiler edit to useful evidence, not compliance with an arbitrary wall-clock cutoff.

Make and fixture scheduling have separate, explicit owners. The Makefile does not detect CPU count or add -j to MAKEFLAGS; the caller selects Make parallelism with the standard make -jN option. JOBS ?= 1 controls only the wwfixture -j N value passed by test-compiler. A normal fast gate can use both layers deliberately:

make -j4 JOBS=4 test-commit

For deterministic failure reproduction, make both layers serial explicitly:

make -j1 JOBS=1 test-commit

Serial execution is a debugging mode, not a correctness requirement. JOBS is not inferred from MAKEFLAGS, and there is no jobserver adapter or second scheduler hidden in Make.

nocc remains the separate, explicit reproduction route from a checked-in stage-0 snapshot. It is not an implicit prerequisite of ordinary tests or of test-bootstrap, because it has an external stage-0 precondition.

Compiler fixtures

Each fixture is one directory with one case.ww. Its first line is exactly one of:

//ww:error "required diagnostic fragment"
//ww:error c "C-stage fragment" ww "WW-stage fragment"
//ww:compile
//ww:run
//ww:run-exit N

error requires normal nonzero frontend termination and the declared stderr fragment. The labeled form routes distinct fragments to the C and WW cells; labels are fixed-order and neither fragment is treated as a shared fallback.
compile requires frontend exit zero and produces no executable.
run builds and requires normal program exit zero.
run-exit N builds and requires normal program exit N.

Every fixture is run against the C and WW frontends. Signals, launch failures, timeouts, build failures, and runtime exits are distinct outcomes. test-compiler passes -j $(JOBS) and therefore uses one fixture slot by default. The direct wwfixture CLI retains its own four-slot default; pass -j N when its concurrency must be explicit. Its existing os.exec.start/poll loop supervises the independent processes. Each cell has its own working directory; filesystem fixtures use that directory or an existing temp.named path rather than a shared fixed pathname.

The old TESTS list and all explicit per-wrapper Make rules are gone. A surviving C carrier is registered only by its source file and built through one generic pattern rule. Four arena/frontend units share a static rule; the codegen unit links the existing private cgen and text-emitter objects directly.

test/wwfixture/integration.sh remains the direct black-box owner for behavior that exists only at the command/process/protocol boundary: filtering and list output, phase and outcome classification, diagnostic routing, malformed-corpus and identity-drift rejection, signal/timeout/interruption cleanup, publication failure, and strict result-stream decoding. Semantic fixtures cannot prove those observations about their coordinator. test/wwfixture/process/main.ww owns the lower-level os.exec primitives, not the CLI policy layered over them, so it is complementary rather than a duplicate owner.

The six unit sources have zero active system, popen, fork, or exec calls. Before direct conversion they contained two popen call sites and a full test-unit run launched one ww -V plus twelve w6c processes. 400_w6c now checks the same twelve assembly fragments in process, grouped under ten unique sources. The CLI version assertion lives with C/WW-driver parity in the existing 949_driver_flagargs integration carrier; 000_smoke retains its arena-growth and nonempty-version-constant assertions.

Package and language behavior

ww test delegates directory package requests to the native package coordinator. The coordinator owns request-pattern expansion, package grouping, same-package and external-package action selection, one canonical directory-owned product, filtering, result aggregation, and its internal temporary workspace. Test sources are exclusively *_test.ww (Go's _test.go contract): a line-leading @test declaration in any other source is rejected loudly — by both driver stages at directory enumeration and by the coordinator at source classification. A local spelling containing ... (for example ww test lib/..., Go's ./... form) is recognized by both driver stages before path resolution and walks from the directory prefix before the first wildcard. Recursive children beginning . or _ and exact testdata subtrees are pruned; child directory symlinks are not followed. A wildcard cannot consume a non-terminal exact vendor element, while an explicitly vendor-rooted pattern remains legal. Every selected test-bearing directory becomes exactly one package run, binary, and result even when both internal and external actions exist. Selected source-bearing directories without selected test files report exactly ? <package> [no test files]\n after ordinary production validation, with no support, generated main, link, binary, captured runtime result, or process. The coordinator first launches one driver plan for the complete command-global package/action union. After that shared build completes, -j N schedules up to N successful selected test binaries concurrently under os.exec start/poll supervision (no threads); emission stays strictly in group order, so the byte stream is identical at every -j level, and -j 1 — the default — matches the former sequential run loop exactly. Measured on the 31-package lib/... walk: 7.0s sequential, 2.4s at -j 4.

List mode uses that same product process and initialization boundary but starts no per-test child. The shared language harness emits only selected qualified test names, one per line in descriptor order. A valid filter selecting no tests therefore produces no harness list bytes, warning, or accounting while the process returns success; the coordinator continues with the package's normal unsuffixed ok result. Ordinary non-list zero-selection execution instead writes the exact line testing: warning: no tests to run through harness stderr, retains its accounting and successful status, and gives the successful package ok result the suffix [no tests to run]. Marker recognition is line-delimited, at capture byte zero or after a newline; an arbitrary substring in user output does not annotate the result. Concurrent products keep independent empty or nonempty captures, markers, and suffixes, and persistent work never caches a test result. A directly invoked retained binary exposes the warning on stderr and accounting on stdout because no coordinator combines its descriptors or emits a package result.

Directory test binaries are always linked under the coordinator's temporary product root. The retained-output option has the exact registered name o and accepts -o VALUE, --o VALUE, -o=VALUE, and --o=VALUE. An equals form splits only at its first =, preserving an empty value and any later = bytes. Repeated occurrences are last-value-wins. A final empty value is no effective explicit output: it requests no retained copy for a running test, while -c falls back to the ordinary <import-leaf>.test name. Concatenated -oVALUE and --oVALUE spell unknown flags and reject before loading, product construction, tools, execution, or publication. Known test flags, including these exact output forms, remain recognized before or after package operands. -c independently requests a caller-visible executable copy and suppresses execution. A nonempty effective -o independently requests a copy and still runs the temporary binary unless -c is also present. Without an effective explicit output, -c publishes <import-leaf>.test in the invocation directory; an output ending in / or naming an existing directory receives one such name per selected package and missing parents are created. A single non-directory output is legal for one package only. Multi-package non-directory output and duplicate visible binary names reject before tools; exact /dev/null discards every copy and permits duplicate names. The private test link still runs unless -c suppresses it; the raw single-file compatibility route also links into driver-owned private storage instead of using /dev/null as an artifact or .sepwork stem. A package with no selected test source validates production, reports the same [no test files] result, publishes nothing, and does not create an otherwise unneeded output directory. Successful -c output is silent apart from that no-test-files report.

The package suite also pins the adjacent ww build output-directory branch. After normal loading, an existing directory (including a symlink to one) or an explicit spelling ending in / receives each command under its requested import-leaf name (or local directory-leaf fallback) even for exactly one selected root; a raw .ww command uses its source basename. A missing trailing-slash hierarchy is created from 0777, filtered by umask, by the request transaction. A directory or raw no-main-only selection and an import failure run no tools and do not create output. Independently selected non-main siblings in a mixed request load but have no action; command dependencies still build. Package/import rejection precedes no-main, derived destination length, duplicate command basename, and implicit default-directory preflight. If the synthesized default command basename already exists as a directory, load/graph validation precedes a zero-tool rejection and preserves that directory; a non-main package has no corresponding default output. Persistent work reuse, invalidation, failures, interruption, and concurrent drivers retain the same action and rollback rules; output form does not enter package/import/action identity. This build branch starts no runtime process, while direct execution of its published binary remains an artifact check rather than part of ww build. Assembly-only -S still selects command actions and rejects a no-command directory request, but it performs no install-only destination validation or output-directory creation.

Package declaration checking also precedes test execution. In a package whose declared name is main, let main, const main, def main, and type main are rejected by both compiler stages as cannot declare main - must be func and are not installed into package scope. The generated directory-test main is an independent tool-owned action and cannot mask the malformed production or same-package declaration: the product emits no test body output, accounting, or ok result. This condition keys only on the source-declared package name; a package with another declared name may export main even when its dotted path or physical leaf is main, and WW's supported argument/result-bearing function entries remain valid. The native nonfunction_main_declarations_reject observer covers all four declaration kinds, direct compiler ownership, build/test diagnostics, command-import precedence, declared-name/import identity, valid artifact bytes, cold cleanup, and warm work/output rollback and reuse across both stages.

The retained executable is byte-identical to the temporary runnable and has executable mode 0777 filtered by the caller's umask, but it is never the path executed by the coordinator. Compile-only publication participates in the driver's one request-wide transaction: producer, linker, staging, or installation failure preserves every prior destination and removes stages and newly created output prefixes. A running retained request instead withholds the public path from that build transaction. The coordinator runs the private binary, then invokes the selected stage driver's guarded install action only after a successful process result. Failure, signal, timeout, interruption, or child-start failure therefore preserves any prior retained binary and creates no new one. Successful products install independently after their runs. The language runtime owns individual @test functions.

Every build or retained-test public install follows Go 1.26.5's late destination safety rule. After applicable producers (and, for running tests, after the successful run), ordinary stat rejects a directory and rejects a nonempty regular non-object file. Empty reservations, recognized prior outputs, and non-directory non-regular paths remain replaceable. Recognition uses Go's archive/ELF/Mach-O/PE/Plan 9/WASM/XCOFF magic plus WW's narrow //ww:module interface prefix. Package/import rejection keeps its earlier diagnostic precedence; exact /dev/null and -S never enter the guard.

Every actually executed directory product gives its single generated binary the product's canonical absolute physical source directory as child cwd. A fresh per-run environment removes inherited exact uppercase PWD entries and appends one PWD with that same directory, matching Go 1.26.5's observable AppendPWD plus last-value-wins os/exec behavior. The coordinator's cwd and environment do not change, and overlapping -j N children share no writable environment vector. Relative ordinary data, testdata, and writes resolve in the tested directory. Production and test-only dependency initializers run in that product process and see its directory; separately testing the dependency creates a separate process using the dependency directory. Equivalent direct, recursive, redundant, absolute, and root-symlink spellings converge before this runtime field is assigned.

Every test binary actually executed by either driver stage also receives one effective uppercase PATH beginning with the canonical absolute directory of the selected WW driver. Missing and empty caller values produce only that directory; a nonempty first effective caller value follows it after :. Normal duplicate PATH= entries collapse to the one child value. Directory products allocate this environment independently per concurrent child; the raw single-file route performs the same PATH transformation while retaining its caller cwd, PWD, stdin, and split streams. Running retained tests transform the private run and install only after success. Compile-only, assembly-only, no-test, rejected, and build requests start no test process and therefore have no test-PATH state.

For a directory product, the rest of the child vector is a Go-like snapshot of the caller environment rather than the build-tool environment: the first case-sensitive occurrence of every normal key=value survives, later normal duplicates and raw empty entries do not, and nonempty malformed entries remain in order. Only the effective PATH and package PWD are then appended. Caller LC_ALL, TMPDIR, empty values, case-distinct keys, and arbitrary variables therefore reach dependency initialization and selected tests. Build-plan compiler, assembler, archiver, linker, and scratch environments keep their existing pinned locale and request-private temporary directory.

Every such product also receives an independently opened null device as fd 0. Caller terminal, pipe, and file bytes remain with the coordinator; serial and parallel products observe immediate EOF rather than consuming a shared input offset. Production and test-only dependency initialization, filters, list mode, no-match execution, failure, and timeout use that same process boundary.

The same product maps fd 1 and fd 2 to one open product-local capture. This matches Go's test-command use of one writer for exec.Cmd.Stdout and exec.Cmd.Stderr: writes are not drained into two files and regrouped later. The coordinator emits the completed combined bytes and any product run-status line on stdout. Before its existing directory-product ok or run-status FAIL trailer, it emits exactly one newline when that capture is nonempty and does not already end in newline. Empty and newline-terminated captures gain no byte. Raw single-file and later direct retained-binary execution have no coordinator trailer, so their bytes remain untouched. Each parallel product has a distinct capture, while canonical group-order emission remains byte-stable across -j values. Build-plan stdout and stderr stay distinct, and loader/build diagnostics stay on stderr.

After canonical result emission, an ordinary test command with any explicit directory, recursive, dotted-directory, or raw-file target emits exactly one final standalone FAIL\n when attributable setup, build, or test execution failed. It follows later successful package output and applies to filters, list mode, signals, timeouts, executable-start failures, and running retained tests. The bare implicit-current-directory form, -c, -S, build requests, CLI usage/shape or output preflight, publication-only, capture-only, cleanup-only, and direct retained-binary execution do not receive the command marker. Directory coordination and both raw-file drivers own only this final presentation state; test captures, result trailers, diagnostics, scheduling, rollback, artifact bytes, and persisted action identity remain unchanged.

The physical directory remains distinct from exact dotted package identity and from production, internal, external, recompiled, support, and generated-main action identity. Only product execution uses it. Compiler, assembler, archiver, linker, support, and generated-main commands retain their build-plan cwd and environment. Captured directory build plans begin with null stdin, which their inherited-stdio tool descendants retain. ww build, ww test -c, -c -o, and no-selected-test products start no test child. A published test binary run directly inherits the user's cwd/environment/stdin. The raw single-file compatibility path inherits caller cwd/stdin and all unrelated environment entries, with only the test-PATH transformation above; neither route embeds a package directory or input behavior in the binary.

Before package grouping, both directory drivers and the shared recursive coordinator apply Go 1.26.5's filename OS/architecture rule for WW's fixed linux/amd64 target. The basename stem ends at its first dot; a final _test token is removed for matching; a recognized OS/architecture pair takes precedence over a recognized final single token; and unknown or misplaced tokens remain ordinary. Wrong-target production, same-package test, and external-test files therefore create no imports, variants, generated-main inputs, runtime tests, artifacts, or persistent invalidation. Names are byte-sorted before selected-source validation, so diagnostics do not depend on directory entry order. A recursive pattern omits a directory with no eligible source, while an explicit wrong-target-only build rejects it as having no WW package source. There is no source-level build-expression or user-tag mode.

Within every eligible production, same-package test, external-test, or test-only source, the package clause is followed by one contiguous import section and then ordinary top-level declarations. A later import is a parser error, imports must appear before other declarations. The imports-only load pass diagnoses the first import in each separated late section before graph or producer construction; the full parser gives direct compiler input the same result. Consequently an invalid test source creates no variant, generated main, test binary, runtime process, accounting, retained output, or new persistent generation. The directory command retains its existing attributable final FAIL\n. Wrong-target files remain excluded before this rule and contribute no diagnostic. This ordering state is per source and never package, import, variant, action, artifact, publication, or persistence identity.

Every eligible source also owns an independent source-start position. An exact UTF-8 BOM (EF BB BF) at byte offset zero is ignored before package-clause and import parsing, so production, same-package, external-package, and test-only sources all count the marker's three bytes and begin the following token at logical line 1 column 4. The same code point anywhere else, including a comment or literal, is a source error before variant actions, generated main, producers, test runtime, accounting, or retained publication. The shared coordinator applies the offset-zero rule while classifying package clauses; the selected stage driver scans the complete source and preserves the per-file rule while composing package units. Wrong-target files remain excluded before either operation. The marker is source representation only and never package/import/action/artifact/publication/persistence identity; no test result is cached.

After filename eligibility and before package-clause classification, each raw source byte 00 in a selected production, same-package test, external-package test, or test-only source is one positioned invalid NUL character error. Comments and literal text do not hide it. The shared coordinator owns that early package/test rejection and reports every raw NUL in the invalid physical source. On complete direct frontend inputs, the C and WW source decoders diagnose each raw NUL and omit it from their logical character streams before token recovery. Thus a NUL cannot split an escape, identifier, number or suffix, operator, comment delimiter, or EOF boundary into a stage-dependent second error. Test loading maintains a stage-equal source-diagnostic stream before import discovery, graph actions, producers, test execution, accounting, result output, or retained publication. An invalid selected test request emits only the existing attributable FAIL\n; it does not construct a variant, generated main, test child, or new persistent generation. Wrong-target exclusion remains first, so an excluded file with a raw NUL has no diagnostic or persistence effect. A \\x00 escape remains a valid literal value. This rule is limited to raw U+0000; malformed UTF-8 and the independent per-source BOM rule retain their existing, separate contracts.

After filename and test-role eligibility and before package-clause classification, every malformed UTF-8 byte in a selected production, same-package test, external-package test, or test-only source is one positioned invalid UTF-8 encoding error. The shared coordinator scans selected physical files in existing byte-sorted order and reports all malformed bytes in the first invalid file before package grouping, import discovery, or delegated tools. The C and WW source decoders consume each malformed byte, omit it from their logical streams, and resume, so the byte cannot manufacture or alter a package token, import spelling, binding, or edge or produce stage-specific fallback recovery. Correctly encoded U+FFFD and other valid non-ASCII text in WW's permitted comment and literal contexts remain valid. A malformed multi-byte spelling is diagnosed byte by byte according to width-one UTF-8 decodes. Raw NUL and per-source BOM diagnostics remain independent and interleave with malformed-byte diagnostics in physical byte order.

Malformed selected source rejects before package/test variant construction, generated main, compiler, assembler, archiver, linker, runtime, accounting, retention, or publication. The existing attributable explicit test request still owns its final FAIL\n; no package ok line or test-result cache is created. Cold rejection leaves no unit, .wwi, assembly, object, archive, binary, capture, status, stage, or transaction residue. Warm rejection commits nothing and preserves the prior generation and public output byte for byte; restoring the valid bytes follows ordinary exact-content reuse. Producer, runtime, publication-only, and cleanup failures are unchanged because source rejection precedes those phases. Validation is source/request-local, adds no process or signal boundary, cannot contaminate an overlapping valid request, and leaves interruption and owned-process cleanup with their established owners. Wrong-target and ordinary-build-excluded test sources remain unread by this semantic preflight.

malformed_utf8_is_rejected_in_every_selected_source in test/package/package_test.ww, together with focused C and WW lexer coverage, is the focused proof owner for valid encodings and encoded U+FFFD; stray leads and continuations; overlong, surrogate, out-of-range, truncated, repeated, and token-boundary cases; BOM/NUL interaction; root and dependency builds; source versus import-resolution precedence; all test roles; wrong-target exclusion; cold and warm rejection; exact restoration/reuse; parallel isolation; diagnostic and valid-artifact stage parity; and absence of residue. Concrete post-change measurements and hashes are recorded only after focused and full validation. The completed BOM, raw-NUL, and exact output-option slices are not reopened, and canonical dotted package/import identity remains unchanged.

After that eligibility boundary and the coordinator's required package-clause classification and production @test validation parses, the delegated loader performs selected-basename Go 1.26.5 simple-fold preflight before its graph import scan. During graph loading, effective canonical imports register only after contextual local/vendor expansion; exact identity reuse is accepted, but two distinct fold-equivalent identities reject the entire shared plan. Selected basenames register in one canonical-directory scope. A production build registers production names only. A test plan shares the filename scope across production, internal-test, and external-test actions, including same-only, external-only, and mixed directories, without combining their source units. Hidden, underscore-prefixed, wrong-platform, and otherwise ineligible files never register. Package variants, product-scoped recompiled copies, support, and generated main remain action wiring over exact identities and do not manufacture package collisions. The fold preflight does not replace or precede those coordinator-owned source-validation parses.

Separate compilation is the only driver build path; no compatibility mode switch remains.

test/package/package_test.ww owns the focused dual-stage single_root_build_output_directory observer. It covers existing and missing directory spellings; trailing and non-trailing forms; literal, logical, default-dot, symlink, raw-file, and implicit-default routes; build/test adjacency; mixed and repeated roots; command-only publication, no-main and implicit-collision rejection, non-main default behavior; import diagnostic precedence over no-main, derived-path, duplicate-destination, and recursive implicit-collision checks; -S -w command selection with long, duplicate, implicit, and missing-directory publication preflight omitted; cold, warm, and invalidated persistence; compiler, assembler, and linker action traces; compiler/linker failure and linker signal; prior-state and directory-creation rollback; concurrent isolation; executable modes and runtime exit values; .new/transaction cleanup; and Cstage/WWstage diagnostic, binary, assembly, and persistent semantic-artifact byte parity. General output permissions, occupied stages, multi-product transactions, null discard, and test runtime timeout remain with their existing observers.

test/package/package_test.ww also owns the dual-stage declared_name_identity_and_file_import_scope and explicit_import_alias_binding_modes observers. They generate temporary directory trees proving that canonical import identity, physical directory, declared package name, optional source alias, and effective source-file qualifier remain distinct. The alias matrix covers default and explicit-only qualification, bare function/type/def/const/variable rejection with exact unused-before-undefined diagnostics, same-file duplicate bindings, accepted same-path imports under distinct bindings, alias reuse across files, sibling scope isolation, package-declaration collisions, unused aliases, blank side-effect imports, repeated blank/default/explicit combinations, and blank no-binding/no-unused behavior. It also pins canonical .wwi spelling, archive/action ownership, one canonical direct edge for repeated occurrences, stage-equal diagnostics and binaries, and clean rejection state. The declared-name observer proves imports are file-scoped while dependency edges are the package-wide sorted union; command and production/internal/external/directory-main actions retain canonical action ownership while one canonical directory owns their product; external self-imports bind the augmented internal action; vendor expansion changes identity but not the effective qualifier; compiler argv contains only direct .wwi inputs; and its named rejected actions leave neither committed nor staged action artifacts or a published binary. It also forces a request transaction to reject after dependency work has staged, then proves that both stages restore every prior artifact, unit, tool record, stamp, and product rather than accepting a mixed warm generation. Together with the existing directory, recursive, vendor, exact-argv, command, and persistent-workdir observers, the package suite proves archive-only link argv and exact warm/rejection-state behavior without duplicating those broader mechanisms in this observer.

The focused dual-stage effective_init_imports_never_enter_binding_recovery observer is the acceptance owner for imports whose effective file-local qualifier is init. Its validation matrix is required to cover an explicit alias and an implicit qualifier obtained from package init; unused and used occurrences; repeated rejected occurrences; a package-scope declaration named init; later init.Name recovery; missing-target precedence; and valid blank and non-init explicit-alias controls. It must exercise ordinary root and imported builds plus production-test, same-package-test, external-test, and test-only source roles where applicable. Each resolved rejected occurrence must produce one core diagnostic at the alias/path spec token and no unused, duplicate, or declaration-collision noise from that binding; a later selector remains an independent undefined-name error. The resolved dotted occurrence must remain in dependency graph and action provenance while the rejected qualifier stays absent from file scope.

That observer's lifecycle proof requires fixed-path Cstage/WWstage status, stdout, and stderr parity; cold absence of public and retained products; warm preservation of the complete committed work-file manifest and public product; imported-dependency rejection and cold empty-workdir rollback; valid control artifact-byte parity; and absence of active .new, .install, .wwtxn.*, adjacent .sepwork, capture, scratch, and tool-stage transaction residue. Producer/runtime failure, request concurrency, interruption, descendant cleanup, and test-process topology retain their established owners because the selected checker rejection adds no producer, runtime, process, signal, timeout, shared-state, or publication boundary. The observer passed in both stages, including imported-dependency rejection, all four test-source roles, cold and warm rollback, normalized diagnostic parity, valid artifact-byte parity, and residue checks. The complete ordered test, test-commit, test-byteid, test-bootstrap, test-platform, and test-all gates also passed. Build workdir format remains 18, test workdir format remains 19, semantic storage remains 3, and no test-result cache is introduced.

The same package owner contains the focused directory_test_execution_working_directory observer. It constructs independent temporary directories from an unrelated caller cwd and compares Cstage/WWstage output for production/internal/external/combined and every test-only shape; production and test-only dependency initialization; recompiled external self-import; duplicate inherited PWD; absent, empty, nonempty, and duplicate inherited PATH; ordinary data, testdata, relative writes, and deliberately nonempty caller stdin; direct/recursive/redundant/absolute/symlink roots; reversed request and creation order; serial and parallel products; filters, list, and no-match execution; failure and timeout; build/no-test and compile-only paths; running retained success and failed-run rollback; direct published binaries and raw single-file execution; build-tool cwd/argv/environment/stdin; persistent data-only reuse and artifact bytes; input-open failure before capture creation; source-class rejection with empty workdirs; and deterministic post-build child-chdir failure isolated from a successful sibling. It pads the inherited environment beyond former fixed observer sizes, requires one appended product PWD, one toolchain-first child PATH, requires EOF for captured actions and caller data for inherited- stdio routes, alternates fd-1/fd-2 write syscalls through every executing variant and initializer, requires one ordered stdout stream on success, assertion failure, signal, timeout, and setup failure, requires direct/raw inherited routes to keep their streams separate, and sweeps the persistent workdir for staged residue. Command-global bounded-memory failure remains independently owned by allocation_failure_is_command_global.

The focused directory_test_trailer_starts_on_new_line observer generates empty, already terminated, and unterminated combined captures. Across both stages it covers stdout success, stderr failure, signal termination, concurrent products, filtering, listing, running retention, retained artifact bytes, and the raw/manual routes that have no coordinator trailer. It requires complete concurrent stdout/stderr parity and forbids both an adjacent trailer and a doubled separator.

The same package owner contains the focused platform_filename_source_selection observer. It independently generates the suffix matrix, production/internal/external sources, wrong-target import and malformed sentinels, wrong-target-only directories, and reverse-created diagnostic files. Across cold and persistent Cstage/WWstage roots it compares normalized compiler/assembler/linker argv, units, .wwi, assembly, objects, archives, generated-main archives, binaries, runtime/test output, direct and recursive behavior, reversed roots, ignored-edit reuse, selected private-change propagation, and request rollback after a dependency has staged and the root compiler fails. Wrong-target test files cannot add a target to the directory main or create one for an otherwise no-test directory. The bounded-memory package-initialization observer now pressures the combined directory topology itself: production, augmented internal test, external self-import, a transitive recompile-for-test clone, support, and the two-target generated main. Each stage searches its own bounded address-space ceiling, while the accepted failure must have the same ww: out of memory diagnostic, invoke no compiler, assembler, or linker, and leave an empty caller work directory with no output, status, or staging path.

The same observer now owns selected-basename folding: ASCII production, internal, external, same-only, external-only, mixed, and cross-classification collisions; printable Unicode Kelvin folding; malformed UTF-8 replacement-rune keys with exact \xNN diagnostics; accepted composed/decomposed neighbors; ordinary-build isolation from test collisions; ignored folded pairs; reversed creation order; and direct/recursive collision-diagnostic parity. It proves zero compiler/assembler/linker calls and an empty workdir, including no test-support/main artifact, on cold rejection. A committed warm package then gains and loses a colliding file; the failed middle request preserves every unit, .wwi, assembly, object, archive, binary, tool record, and stamp, and removal restores exact reuse. package_graph_diagnostics_are_stable provides the corresponding direct, transitive, reversed-root, recursive, same-directory, exact-repeat, and warm import proof. vendor_directory_import_resolution proves folding uses the expanded vendor identity under reversed imports and product descriptors, even when both spellings reach one physical directory, and that multi-product rejection publishes no output, status, tool state, or staging residue.

test/sep/sepinit_test.ww is the single focused package-initialization owner. It generates all source trees temporarily and runs independent cold/persistent Cstage and WWstage legs. Its matrix covers blank-only reachability; dependency, diamond, and independent ready-task order; runtime call/allocation and aggregate package lets; multiple init declarations and invalid forms; direct/qualified init invisibility; initialization cycles and diagnostic order; command, internal-test, external-test, support, and one directory-generated-main ownership; canonical task and dispatcher symbols; deterministic one- and two-member archives; .wwi exclusion; byte-identical artifacts/binaries; init-only invalidation; warm invalid-init rollback; direct/recursive variant equivalence; exact production, support, ptest-before-pxtest-before-main task ordering, exactly-once package/dependency initialization in one dispatcher; complete normalized rejection diagnostics; dangling staging/rollback no-follow rejection; and checked compiler-output failure/non-regular-destination rollback; and bounded-memory Cstage/WWstage allocation-failure parity across the complete combined package-test graph, under independently discovered ceilings supplied by the repository-built sep-limitexec helper.

ww build and an explicit single-file ww test -o <stem> publish <stem>.sepwork as a caller-owned artifact directory. The driver acquires it with one fresh mkdir and refuses an existing path; it never clears a collision. A caller keeps only the exact artifacts it observes and removes that exact tree on every later success or failure. Directory-package test plans instead keep their cold semantic-action scratch and runnable binary inside the coordinator's temporary root; a compile-only retained executable escapes through the build transaction, while a running retained executable uses the post-run install action above. ww run and no-output single-file ww test use driver-owned scratch instead; both driver stages place that scratch and their temporary executable beneath one freshly acquired directory, remove both after every build result, and make cleanup failure fail the command. Make recipes build driver-produced tools in invocation-owned directories and apply the same exact cleanup rule.

ww build -w DIR and single-file ww test -w DIR replace that scratch with a caller-owned persistent package-artifact workdir: for these direct routes the directory must already exist, is never cleaned by the driver, and holds one committed unit, .wwi, .s, .o, and dep .a per package plus byte copies of the invoking driver, compiler, and assembler and a small mode stamp. A package is reused only when its freshly composed owner unit byte-equals the committed unit, no recompiled direct dependency emitted changed export bytes, and every applicable recorded executable byte-equals the live executable — content identity only, no mtimes, no hashes, every decision reproducible with cmp against plain files. The driver identity covers graph construction, owner-unit composition, direct-export argument construction, archive creation, and commit algorithms that neither unit bytes nor compiler identity can name. Recompiled artifacts, dispatcher artifacts, products, statuses, tool copies, and the stamp land at staged .new names. After every requested product stages successfully, one rollback-capable transaction installs the whole generation; any producer or installation failure preserves the prior committed bytes and removes remaining stages. An interrupted or rejected request therefore cannot create false reuse or a mixed generation; a successful executable link still reruns when required. One workdir serves one invocation at a time and may hold the complete command-global action universe for all selected roots and isolated variants; semantic identity, rather than request shape, controls reuse. Both driver stages implement the identical contract. Test-mode persistent workdirs use format 19. This is build staleness in the Make/mk/Go sense, not a result cache: real directory test products always run, while no-selected-test directories do not create a process. The byte-identity and bootstrap gates keep building on fresh scratch. make clean reclaims every workdir under out/.

Directory-product cwd, PWD, and test PATH are request-time child-process metadata, not persisted action inputs or results. A runtime directory-entry failure therefore does not erase a successfully committed compilation generation. Changing only ordinary fixture data causes no unit/export/assembly/object/archive/main/binary change and no producer work beyond the same established warm final relink, while the next always-run product observes the new bytes. Build workdir format remains 18, test workdir format remains 19, and semantic storage remains 3.

Fold keys and first-spelling tables are request-only loader state and are never stored in a workdir. They do not alter successful unit bytes, action/storage keys, or tool records, so build format 18, test format 19, and semantic storage format 3 remain unchanged. Structural fold rejection occurs before staging and cannot mutate a committed generation; a later request with the collision removed evaluates the original exact-key reuse contract.

On a package or tree target, ww build/test -w DIR forwards that exact caller path as the one command-global semantic-action store; neither pattern spelling nor package-group name derives a persistent subdirectory. The delegated driver may create a missing store only after graph and request preflight, and rolls back newly created empty prefixes if later setup fails. Every reuse decision stays with the driver's content-identity contract above. Test -c may use the same store: the workdir owns semantic actions and the invocation/output path owns the separately staged retained executable. The test-library route uses the same command-global store contract: one ww test -j $(JOBS) -w out/wwbuild/wwtest-lib lib/... line owns library behavior (measured: 2.3s cold, 1.0s warm at -j 4, vs 4.7s for the retired 59-target per-file fan-out).

test/lang currently uses one package per source file, so its complete gate still performs independent package builds. That remaining source layout is not hidden behind caching or concurrency; it is outside the small test target.

Compiler-only byte identity

ww build -S and ww test -S -o <stem> run source discovery, dependency ordering, unit composition, and each required w6c -c invocation. They return after the complete .s/.wwi set exists. The producer loop does not invoke w6a, create per-package archives, or invoke w6l when -S is active.

test-lang-byteid runs the 158 selected language files once with C w6c and once with WW w6c, requires identical emitted .s filename sets including __root.s, and compares those bytes. Relative to the old two-leg full builds, this removes at least 3,476 assembler launches, 3,160 archive writes, and 316 linker launches from the comparison loop. A cold Make invocation may still build prerequisite compiler binaries; the compiler-only claim applies to the per-language-file comparison path.

test-data-byteid applies the same comparator to the declarative corpus: every non-error case.ww builds twice through the fixed cstage driver with only WW_W6C swapped, and every emitted per-package .s must be byte-identical. //ww:error fixtures have no .s; their both-stage reject parity is owned by the fixture corpus itself. Known cs/ww divergences are pinned in DATABYTEID_DIVERGED with the 989_lib_byteid discipline: a pinned fixture must still build on both stages and still differ, so a compiler fix fails the gate demanding graduation rather than silently widening coverage. The full sweep compares the 1,157 non-error fixtures in about a minute and is scratch-rooted under out/, not /tmp.

Byte identity is an explicit proof gate. It is not a prerequisite of test or test-commit.

Bootstrap, subprocesses, and CSP

Stage-2-through-stage-4 fixed-point proofs and the 950/991–995 self-host gates are reachable through test-bootstrap and test-all, never through test or test-commit. Cold ordinary targets may still build their C- and WW-stage tool prerequisites once; they do not iterate those tools to a fixed point. The C bootstrap source and the standalone nocc route remain intact.

The bootstrap recipe owns the fixed out/bootstrap tree. Make schedules that target once within one invocation, but two independent make bootstrap invocations are not safe to run concurrently and remain mutually exclusive.

lib/os/exec is the sole reusable WW subprocess mechanism. Fixture and package coordinators use its captured asynchronous path. An empty captured-command stdinpath opens the null device; an explicit path supplies controlled input, and either descriptor is installed before exec with checked setup reporting. Distinct output paths create independent exclusive captures; byte-equal output paths open once and duplicate that descriptor so fd 1 and fd 2 share the same open file description. The executor process proof covers both modes, including closed caller standard descriptors. The WW driver directly uses os.exec.runstdio for inherited-stdio, inherited-environment, leader-only compiler, assembler, linker, cleanup, run, and single-file-test calls. The local WW procrun implementation is deleted. The C bootstrap retains its C process implementation because it cannot consume a WW standard-library module.

WW has tokens and an opaque type for future CSP/channel work, but no mature production channel operations, task runtime, or scheduler. No channel, goroutine, thread, worker-runtime, or CSP library was added. Coordinators stay single-threaded; OS process polling does not require language-level threads.

Retired mechanisms

The following are deleted, not adapted:

test/run, including its phase classifier, xargs scheduler, atomic private records, timeout-text classifier, result collector, and last-green cache;
test/run_test.sh, the synthetic shell tests for that protocol;
every explicit TESTS registration and all 329 explicit wrapper rules;
the unregistered test/runww.ww corpus runner;
frozen duplicate compiler-corpus code under internal/wwtest, test/wwtest, and test/compiler;
the redundant standalone smoke, test-run, and test-harness routes; and
library-launcher wrappers whose only assertion was an existing @test source's exit status, including the declarative 900_stdlib.c launcher.

There is no test cache, daemon, scanner, generated manifest, database, new framework, compatibility API, concurrency runtime, dependency, or changed timeout policy in this architecture.

Open driver work

None; directory-package -c and -o now have the applicable Go 1.26.5 retention, naming, fan-out, execution, and publication behavior.

Validation policy

Use test-unit as the inner loop for lexer, parser, checker, and narrow codegen changes. Run test for the ordinary local compiler check, followed by the focused owner for the changed behavior. Use test-commit for ordinary pre-commit behavior; use make -j4 JOBS=4 test-commit when parallel feedback is desired, and make -j1 JOBS=1 ... to reproduce failures deterministically. Run test-byteid and test-bootstrap only when those proof categories are intended. test-all is the exhaustive CI/release composition and should not be launched casually.

53 KiB Raw Blame History Unescape Escape