From 22ee8f5ae695ef1a03b8ebb7eb7eb2dd877205b9 Mon Sep 17 00:00:00 2001 From: Hojun-Cho Date: Thu, 13 Aug 2026 00:54:16 +0900 Subject: [PATCH] docs: pin canonical package model evidence --- docs/build-system.md | 262 +++++++++++++++++++++++++++---------------- 1 file changed, 165 insertions(+), 97 deletions(-) diff --git a/docs/build-system.md b/docs/build-system.md index fdd3c381..07d56df2 100644 --- a/docs/build-system.md +++ b/docs/build-system.md @@ -2878,39 +2878,59 @@ policy. The command owns source selection, package loading, compiler inputs, archive construction, generated-main construction, and linking for every package variant: -- The ordinary production action selects the directory's byte-sorted - non-test files and is reused wherever that canonical package is imported. -- `same-test` is a distinct internal production-plus-test action. It selects - the byte-sorted production files followed by the byte-sorted matching - `package p` test files, so tests can use private production declarations. -- `external-test` selects only matching `package p_test` files. Its `import p` - is a direct edge to the canonical production action for that directory. - That action compiles with module qualifier `p`, selects every production - file, emits compiler export data and an archive, and exposes no private - declaration to the external root. If the action is first reached through - the external product, its collision-proof artifact key is derived from the - owning root, such as `__ww-test-001-external-production`; a normal import of - the same `p` and canonical directory reuses that action rather than creating - a second compilation. -- Each selected internal or external variant gets a distinct generated-main - package action. Its generated owner-only unit declares `package main` and - imports exactly the selected variant and test-support package. It consumes - exactly those direct `.wwi` artifacts, emits its own `.wwi/.o/.a`, and alone - receives compiler `-T --entry`. +- The ordinary production action selects the directory's byte-sorted non-test + files. A no-test requested product uses this action directly, and every + ordinary, internal-test, or external-test import of that package reuses it. +- The internal production-plus-test variant selects the byte-sorted production + files followed by byte-sorted matching `package p` test files. It is distinct + from production so its private production declarations and test declarations + remain isolated to that test product. +- The external variant selects only matching `package p_test` files. Its + `import p` is a direct edge to the same ordinary production action used by + ordinary products and transitive imports; there is no external-production + role or artifact. +- Generated main is a separate package action whose owner-only generated unit + declares `package main` and imports exactly the selected variant and test + support. It consumes those direct `.wwi` files, emits its own `.wwi/.o/.a`, + and alone receives compiler `-T --entry`. -The command loads all variants into one command-scoped package universe. -Directory variants retain injective artifact keys such as -`__ww-test-000-same` or `__ww-test-003-external`; their generated mains use -matching `__ww-test-NNN-main` keys and distinct internal package identities. -Hyphens keep artifact keys illegal as WW import identities. Repeated requests -for the same canonical directory and variant reuse one compile action, and -imports of the same canonical production directory intern to one production -action across every selected test product. A deterministic dependency-first -traversal of the complete union therefore invokes the compiler and archiver -once per package variant, even when many products share it. Generated-main -actions remain product-specific and are linked separately. The shared plan is -deliberately package-test-specific: it is not a generalized scheduler, action -schema, cache, or protocol. +The authoritative directory-action identity is the triple **(canonical +filesystem directory, canonical ordinary import path, semantic variant)**. +The semantic variants are production, internal production-plus-test, and +external `_test`; generated main and the reserved test-support alias are +explicit non-directory action classes. Source package name validates the leaf +of the bound import path but is not itself a second identity. Requested-root +state, discovery role, product ordinal, output path, persistent artifact key, +and discovery order never enter the triple. + +A literal directory root may enter the interner before its full import spelling +is known. It is provisionally interned by canonical directory and variant, +binds immediately if a source import reaches it, and otherwise binds after its +sources establish the validated manifest-free package name. All binding is +finished before generated-main construction or compilation. One bound import +path mapping to two directories and one directory acquiring two incompatible +ordinary import paths are both command-global deterministic errors before any +compiler, assembler, archiver, or linker ambiguity. The same check spans +variants: an external action with ordinary base `p` cannot hide a different +directory's production `p` behind its derived compiler path `p_test`. + +Artifact publication follows the semantic action instead of product order: +production uses `p`, internal uses `p-internal-test`, external uses +`p_test-external-test`, and their generated mains append `-main`. Generated +package identities are likewise variant-derived, for example +`__wwtestmain.p.internal.main` and +`__wwtestmain.p_test.external.main`. Equivalent products therefore reuse an +already-interned directory action and generated-main action and publish through +the same persistent-workdir slots regardless of discovery or product order. +The narrow raw single-file compatibility path alone retains `__root`. + +A deterministic dependency-first traversal of the complete command union +invokes the compiler, assembler, and in-driver deterministic archiver once per +interned action. This is compile-time interning, not linker-argument +deduplication. Each product is then linked separately from its generated-main +or executable root archive and the complete reachable archive closure. The +shared plan remains package-test-specific; it is not a generalized scheduler, +action schema, cache, or protocol. Each selected directory retains the ordinary entry-directory-first resolution context from the local package slice: its directory, explicit `-I` roots in @@ -2955,17 +2975,14 @@ tests. Explicit raw single-file `ww test FILE` fixtures retain the narrow fused canonical directory package; that path still consumes support as a direct export and emits a root `.wwi/.a`. -The reserved support action and an ordinary source-imported toolchain `test` -action may coexist in the command universe because their compiler qualifiers -are `__wwtest` and `test`. An external-first colocated production action uses -an owning-root artifact key so it remains collision-proof if the command also -loads a different physical package bearing the same compiler qualifier. -Same-qualifier, same-canonical normal and external-production actions always -coalesce; different logical identities for one ordinary physical directory -remain an error. These are the -only narrow action-role exceptions: each product closure is checked to contain -at most one importable action for a compiler module qualifier, so unrelated -roles can never introduce duplicate linked package symbols. +The reserved support action and an ordinary source-imported package `test` may +coexist only because the former is explicitly rebound to the compiler-only +qualifier `__wwtest`. This is the sole directory-alias exception and cannot be +created by a source import. All ordinary production actions, including an +external test's colocated production dependency, use canonical directory +identity and the global bidirectional import-path checks above. There is no +role-based tolerance for duplicate ordinary import identities and no late +product-closure ambiguity to resolve. The selected internal/external variants and a production action reached by their imports or test-runtime closure are the sanctioned graph nodes that may @@ -3001,21 +3018,19 @@ have freshly committed units, whereas failed or no-longer-requested actions cannot be reused. A retry therefore recompiles the failed action without discarding unchanged canonical packages built for independent products. -This ownership split follows Go 1.26.5's separation of production, -same-package-test, external-test, and generated test-main inputs in -[`cmd/go/internal/load/test.go`](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go), -while retaining WW's compiler-owned dispatcher. Go's loader returns the same -cached package pointer for repeated imports, and `PackageList` walks arbitrary -multiple roots with pointer-based deduplication in -[`cmd/go/internal/load/pkg.go`](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/pkg.go). -Its command-scoped builder deduplicates compile actions by package identity. WW -borrows that one-universe, multi-root ownership boundary without borrowing Go's -cache machinery. Direct -compile dependencies and the separately expanded link closure follow the -boundary in -[`cmd/go/internal/work/action.go`](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go), -with compiler and linker import configurations emitted separately in -[`cmd/go/internal/work/exec.go`](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go). +Warm reuse compares the staged owner-only unit, committed artifacts, and the +actual bytes of each direct dependency export. A changed shared dependency is +compiled once; its direct importers are reconsidered once; and propagation +stops as soon as a regenerated importer export is byte-identical. Stable +semantic artifact keys make that behavior independent of which ordinary or +test product first discovered the action. + +This ownership split follows the pinned Go 1.26.5 implementation cited in +section 11.12: its loader reuses canonical cached package objects, its work +builder interns actions by mode and package, and its test loader reuses ordinary +production while constructing only the required internal, external, and +generated-main variants. WW borrows that command-global ownership boundary +without adding Go's build cache, import configuration format, or module system. ### 11.8 Implemented exact package-tool invocation slice @@ -3226,49 +3241,102 @@ vectors across two clean Cstage builds and two clean WWstage builds. The existing directory-package variant regression checks separate production, internal, external, and generated-main actions, exact generated-main direct variant/support exports, -the external-production action, and archive-only link closures. Both stages -compile and run those actions with owner-only units and byte-identical -artifacts. +canonical production reuse across ordinary and test products, and archive-only +link closures. It also reverses equivalent product descriptors, proves a +selected ordinary root imported by another selected product compiles once, +and changes a shared direct export in persistent workdirs to prove propagation +through direct importers stops at the first byte-identical regenerated export. +Both stages compile and run those actions with owner-only units and +byte-identical artifacts. The pinned official Go 1.26.5 tag (commit `c19862e5f8415b4f24b189d065ed739517c548ba`) supplies the design boundary: -- Directory identity starts with `ImportDir`'s named directory - ([`go/build/build.go`, lines 521–524](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/go/build/build.go#521)); directory reads are name-sorted - ([lines 108–111](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/go/build/build.go#108)), lookup selects directory candidates - ([lines 725–809](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/go/build/build.go#725)), and the selected directory is enumerated - ([lines 859–900](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/go/build/build.go#859)). The sorted iteration filters files and appends each source to exactly one ordinary/internal/external-test bucket - ([lines 895–1036](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/go/build/build.go#895)). -- The loader records direct imports - ([`load/pkg.go`, lines 220–225](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/pkg.go#220)), guarantees repeated cache loads return the same package pointer - ([lines 633–636](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/pkg.go#633)), keys reuse by canonical import path - ([lines 757–768](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/pkg.go#757)), and pointer-deduplicates dependency-first package lists - ([lines 2776–2795](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/pkg.go#2776)). -- Build actions are interned by mode and package identity - ([`work/action.go`, lines 437–447](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go#437)); compilation depends only on direct imports - ([lines 628–659](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go#628)), and even package `main` receives that normal interned archive-producing compile action - ([lines 641–647](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go#641)). A link action places that main compile action at its first dependency and expands transitive dependencies - ([lines 918–957](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go#918), - [lines 1034–1068](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/action.go#1034)). The executor derives compiler import inputs from those direct actions - ([`work/exec.go`, lines 864–884](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go#864)), passes them separately from source files - ([lines 928–930](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go#928)), packs remaining objects into the package archive and records it as the built result - ([lines 1017–1033](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go#1017)), then supplies the root archive and all expanded package inputs to linking - ([lines 1592–1624](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go#1592), - [lines 1635–1647](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/work/exec.go#1635)). -- Tests preserve four package roles - ([`load/test.go`, lines 85–102](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#85)): internal production-plus-test - ([lines 175–225](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#175)), external test - ([lines 228–265](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#228)), and generated main - ([lines 272–293](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#272)). Its support imports are loaded for that generated package - ([lines 307–332](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#307)), with the selected internal/external variants attached as distinct direct imports at - [lines 351–373](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/go/internal/load/test.go#351). -- Unified export writing re-links, re-exports, and prunes facts - ([`noder/unified.go`, lines 152–165](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/noder/unified.go#152)), finalizes self-contained data - ([lines 463–470](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/noder/unified.go#463)), selects export roots, and sorts declaration and body indices before serialization - ([lines 495–570](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/noder/unified.go#495)). Compiler import lookup opens the separately mapped artifact - ([`noder/import.go`, lines 61–101](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/noder/import.go#61)) and decodes its exports independently of source parsing - ([lines 170–225](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/noder/import.go#170)); `ReadPackage` consumes that package decoder - ([`importer/ureader.go`, lines 28–62](https://go.googlesource.com/go/+/refs/tags/go1.26.5/src/cmd/compile/internal/importer/ureader.go#28)). +- `go/build` represents one selected directory package with its import path, + package name, ordinary files, internal-test files, external-test files, and + their imports ([`build.go`, lines 436–493](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L436-L493)). + Its directory reader is required to return name-sorted entries + ([lines 108–111](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L108-L111)), + import lookup selects one directory in search order + ([lines 725–767](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L725-L767)), + and the selected directory alone is scanned + ([lines 859–913](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L859-L913)). + The sorted scan assigns each accepted source to that package's ordinary, + internal-test, or external-test list + ([lines 948–1036](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L948-L1036)). +- `cmd/go/internal/load` expands source imports before recording their + canonical paths + ([`pkg.go`, lines 658–669](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L658-L669)). + It resolves canonical path and directory before consulting the package-data + cache ([lines 833–842](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L833-L842), + [lines 863–911](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L863-L911)), + and the command-global package cache returns the existing package pointer for + a later root or import of the resolved identity + ([lines 633–647](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L633-L647), + [lines 757–775](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L757-L775)). + The package's parsed import list becomes its direct package dependencies, + rather than a transitive flattening + ([lines 433–440](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L433-L440), + [lines 2024–2047](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L2024-L2047)). +- `cmd/go/internal/work` keys its action cache by operation mode plus package + pointer ([`action.go`, lines 202–206](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L202-L206)) + and returns the already-interned action for that key + ([lines 437–447](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L437-L447)). + A compile action depends on only `p.Internal.Imports` + ([lines 628–658](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L628-L658)); + an executable link asks for that same cached root compile action + ([lines 919–957](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L919-L957)) + and separately expands the complete transitive link closure + ([lines 1034–1068](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L1034-L1068)). +- The work executor derives compiler package mappings from those direct build + dependencies + ([`exec.go`, lines 864–884](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L864-L884)), + compiles the package's own source list to `_pkg_.a` + ([lines 928–935](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L928-L935)), + packs and publishes that package archive + ([lines 1017–1033](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L1017-L1033)), + and links the compiled main archive with mappings for every dependency + already expanded onto the link action + ([lines 1592–1624](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L1592-L1624), + [lines 1635–1647](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L1635-L1647)). +- Go's test loader explicitly models production, internal + production-plus-test, external `_test`, and generated main, and states that + `ptest == p` when production can be reused + ([`test.go`, lines 85–102](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L85-L102)). + Test imports use the ordinary load cache and compare canonical `ImportPath` + ([lines 118–161](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L118-L161)); + the internal copy is created only when needed + ([lines 175–226](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L175-L226)), + while external and generated-main packages remain distinct + ([lines 228–293](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L228-L293)). + Generated main receives its direct support and selected-variant imports + ([lines 307–358](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L307-L358)), + and copy-on-write rewriting preserves the original package pointers/actions + for unaffected importers + ([lines 421–472](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L421-L472)). +- Unified export production begins from the local package, re-exports required + dependency data, and prunes unnecessary detail + ([`unified.go`, lines 147–168](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/unified.go#L147-L168)). + It type-checks the package's parsed sources and writes deterministically + ordered public/private roots + ([lines 314–362](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/unified.go#L314-L362)), + then finalizes self-contained export data with sorted relocated declaration + and body indexes and a fingerprint + ([lines 463–570](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/unified.go#L463-L570)). +- Compiler import handling canonicalizes each source import and rejects self + import ([`import.go`, lines 125–167](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/import.go#L125-L167)), + then independently opens and decodes each direct package archive/export + ([lines 170–225](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/import.go#L170-L225)). + The complete unified section and linker fingerprint are read from that + selected package file + ([lines 229–296](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/noder/import.go#L229-L296)). + `ReadPackage` reconstructs a package from its public export root + ([`ureader.go`, lines 28–62](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/importer/ureader.go#L28-L62)), + interns embedded package descriptors by canonical path and restores their + import lists + ([lines 152–196](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/importer/ureader.go#L152-L196)), + and reconstructs declarations from relocated export records + ([lines 391–468](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/importer/ureader.go#L391-L468)). WW adopts those practical ownership and action semantics while retaining its small direct CLI representation and existing self-contained `.wwi` encoding.