test: prove package universes beyond 256 actions
This commit is contained in:
@@ -2673,18 +2673,20 @@ not a self-contained functional toolchain outside the build tree.
|
||||
|
||||
### 11.3 Scaling, invalidation, and hidden inputs
|
||||
|
||||
The driver stores fixed `deps[256]` arrays and performs linear graph lookup.
|
||||
Every package unit reads one interface per transitive dependency, so total
|
||||
interface reads and copied interface text are quadratic on deep/dense graphs even
|
||||
on a warm build. Package compiler/assembler work within one driver is serial;
|
||||
Make gains parallelism only by launching independent top-level driver builds.
|
||||
The driver performs deterministic linear action interning but now grows every
|
||||
package dependency vector dynamically (section 11.14). Every package compiler
|
||||
reads exactly one interface per direct dependency; transitive dependencies enter
|
||||
only the executable archive closure. Package compiler/assembler work within one
|
||||
driver remains serial; Make gains parallelism only by launching independent
|
||||
top-level driver builds.
|
||||
|
||||
The observed invalidation rules are:
|
||||
|
||||
- a private change in a directory dependency rebuilds that package and the
|
||||
unconditional final link, but not importers;
|
||||
- an exported change changes its `.wwi` and rebuilds the whole reverse-transitive
|
||||
ancestor cone, even when an intermediate package's own interface is unchanged;
|
||||
- an exported change rebuilds direct importers and continues through an ancestor
|
||||
only while the regenerated direct-dependency export bytes change, stopping at
|
||||
the first byte-identical regenerated interface;
|
||||
- a private change in a folded file import rebuilds its entire owner;
|
||||
- a link-only option reruns the always-executed link but not package compiles;
|
||||
- changing copied compiler or assembler bytes rebuilds every package; and
|
||||
@@ -3554,9 +3556,10 @@ allocations. Cstage's assembler no longer copies a line, operand, `TEXT`, or
|
||||
`DATA` symbol through 256-byte arrays, and the C checker no longer resolves a
|
||||
qualified type through a 128-byte prefix buffer. `PATH_MAX` remains only at
|
||||
actual host pathname and syscall boundaries; the 255-byte constant remains
|
||||
only as the conservative internal basename component bound. `SEP_MAXPKG`,
|
||||
`SEP_MAXPRODUCT`, and `SEP_MAXCONTEXT` remain action-count limits, not byte
|
||||
limits on semantic identity.
|
||||
only as the conservative internal basename component bound. The former
|
||||
`SEP_MAXPKG`, `SEP_MAXPRODUCT`, and `SEP_MAXCONTEXT` action-count limits are
|
||||
removed by the dynamically sized package-universe implementation in the next
|
||||
section.
|
||||
|
||||
The existing native observers exercise the new boundary with ordinary dotted
|
||||
identities over 255 bytes and punctuation-heavy reversible local identities
|
||||
@@ -3624,6 +3627,190 @@ build IDs, importcfg, module machinery, or scheduler:
|
||||
[`test.go`, lines 175–226](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L175-L226),
|
||||
[`test.go`, lines 228–293](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L228-L293)).
|
||||
|
||||
### 11.14 Implemented dynamically sized command-global package universe
|
||||
|
||||
Package, dependency, resolution-context, selected-product, traversal, support,
|
||||
order, and closure storage no longer has an arbitrary 256-element boundary.
|
||||
This is a storage correction, not a new build abstraction: source imports still
|
||||
form one command-global canonical package graph; each semantic package variant
|
||||
still has one action; each compiler still receives exactly its direct exports;
|
||||
and each executable linker still receives its complete reachable archive
|
||||
closure.
|
||||
|
||||
The Cstage representation is exact and deliberately small:
|
||||
|
||||
- `sepgraph.pkg` is a dynamically allocated `struct seppkg *` with logical
|
||||
count `n` and capacity `pkgcap`;
|
||||
- `sepgraph.context` is a dynamically allocated `struct sepcontext *` with
|
||||
logical count `ncontext` and capacity `contextcap`;
|
||||
- each `seppkg.deps` is a dynamically allocated `int *` with `ndeps` and
|
||||
`depcap`;
|
||||
- each `seppkg.context_state` is a lazily extended, zero-filled
|
||||
`unsigned char *` with `context_cap`;
|
||||
- parsed `sepproduct` values are a dynamically allocated vector, and each
|
||||
product stores its support-action index directly; and
|
||||
- package-load frames and topological-DFS frames are temporary dynamic vectors,
|
||||
replacing recursion proportional to graph depth.
|
||||
|
||||
The WWstage representation is isomorphic. `sepgraph.pkg: []seppkg` and
|
||||
`sepgraph.context: []sepcontext` use allocated slice length as capacity and keep
|
||||
separate `n`/`ncontext` logical counts. Every `seppkg` owns a dynamically grown
|
||||
`deps: []i32` with `ndeps` and a lazily zero-extended `contextstate: []u8`.
|
||||
Products, load frames, and topological frames use typed dynamically allocated
|
||||
slices. `internal/wwpackage` continues to construct one union command for all
|
||||
selected directory-test groups, but now checks the complete
|
||||
`12 + 6*products + 2*includes` builder argument count before allocating or
|
||||
starting the driver.
|
||||
|
||||
All graph and product growth starts at capacity 8 and doubles until it covers
|
||||
the requested element count. Cstage clamps before `INT_MAX`, checks the element
|
||||
count against `SIZE_MAX / sizeof(element)`, and publishes a `realloc` result only
|
||||
after success. WWstage checks against the same signed 32-bit count boundary,
|
||||
allocates a replacement typed slice, copies the live prefix, and publishes it
|
||||
only after success. Context-state growth copies old bytes and explicitly zeros
|
||||
the new tail. The shared deterministic failures are `ww: package graph is too
|
||||
large` for an unrepresentable count and `ww: out of memory` for failed storage;
|
||||
compiler and linker argument-count arithmetic is checked before allocation and
|
||||
before any affected tool invocation. Each driver records allocation/size
|
||||
failure during graph discovery and propagates it as a command-fatal load
|
||||
result, rather than treating it as one product's semantic failure and starting
|
||||
tools for a sibling root. The coordinator uses fallible dynamic storage for
|
||||
discovered paths, source/folder/group/plan vectors, process handles, tool
|
||||
environments, and complete builder/run argument vectors; it reports an
|
||||
oversized product set or allocation failure before the corresponding
|
||||
`exec.start` and cleans an already-created request temporary tree. Host pathname,
|
||||
filesystem-component, process-argument, and available-memory boundaries remain
|
||||
real host constraints; none is used as a disguised package-count maximum.
|
||||
|
||||
Vector growth never changes semantic references. Dependency edges, resolution
|
||||
contexts, selected-product roots and variant roots, generated-main/support
|
||||
edges, load/topological frames, order entries, and closure membership are all
|
||||
stable `int`/`i32` indices. Code reserves a graph slot before taking an element
|
||||
pointer and never carries an element pointer across a graph reserve. Capacity,
|
||||
addresses, request order, product order, output names, and workdir location
|
||||
therefore cannot enter action identity, sorting, diagnostics, storage locators,
|
||||
or artifact bytes. Dependency lists retain byte-sorted insertion and duplicate
|
||||
elimination. The iterative loader retains mark-before-child and post-child
|
||||
command-import validation; the iterative tri-color DFS retains deterministic
|
||||
postorder and the complete live path for cycle diagnostics.
|
||||
|
||||
No fixed package, product, context, action, support-map, traversal, order, or
|
||||
closure cardinality remains in either driver. The unrelated `SEP_MAXLFLAGS ==
|
||||
32` limit is retained solely for the existing `-L`/`-l` command-line interface;
|
||||
it neither indexes nor bounds package actions. Compiler and linker tools already
|
||||
allocate their import/input tables from `argc`; their genuine remaining process
|
||||
boundary is the host's executable-argument limit.
|
||||
|
||||
The native package observers generate rather than commit large fixture trees.
|
||||
The extended `long_shared_link_closure_is_complete` builds and runs a chain of
|
||||
300 ordinary directory packages under independent cold Cstage and WWstage work
|
||||
roots. Its command root directly imports all 300 packages and repeats one import,
|
||||
proving an action beyond index 256 compiles, the root receives exactly 300
|
||||
sorted/deduplicated direct `.wwi` inputs, every ordinary action receives only its
|
||||
one direct export, every `.unit.ww` contains only its two byte-sorted owner
|
||||
sources, and the linker receives the root plus all 300 archives exactly once and
|
||||
no `.wwi`. A second command root imports only `p000`; it reuses all 300 ordinary
|
||||
actions and its exact linker line still contains the root followed by the full
|
||||
`p000` through `p299` transitive archive chain and runtime archive, proving that
|
||||
closure construction—not the wide root's direct imports—crosses the old boundary.
|
||||
The observer compares every unit, export, assembly, object, archive, and binary
|
||||
across stages. A second equivalent persistent request invokes no compiler or
|
||||
assembler. Changing `p257`'s export recompiles exactly `p257`, direct importer
|
||||
`p256`, and the wide root, and stops before `p255` after `p256` regenerates a
|
||||
byte-identical export. Closing the chain at `p299 -> p000` produces the complete
|
||||
stage-identical 300-node cycle diagnostic with empty compiler, assembler, and
|
||||
linker traces.
|
||||
|
||||
`dynamic_package_universe_crosses_former_boundary` first selects 309 package-test
|
||||
products through 257 directory spellings in one direct request per stage.
|
||||
Fifty-two real directories each produce ordinary production, internal
|
||||
production-plus-test, external `_test`, and two generated-main actions; the
|
||||
shared support closure brings that command-global universe to 270 actions. Two
|
||||
hundred five explicit symlink spellings of `p000` add distinct valid products
|
||||
and resolution contexts while reusing its one canonical internal action. The
|
||||
observer reverses product and variant order between stages, proves one compile
|
||||
per action, checks all 309 binaries exist, runs boundary products, validates
|
||||
variant-owned units, and compares every action artifact plus representative
|
||||
binaries byte-for-byte.
|
||||
|
||||
The same observer then exercises the public `ww test <tree>/...` coordinator
|
||||
path. The generated tree has 257 real directories: the 52 test-bearing
|
||||
directories above plus 205 production-only directories. The coordinator forms
|
||||
and passes 309 products and 258 contexts (including support) in one driver
|
||||
request. Reusing the direct request's persistent workdir preserves all 270
|
||||
existing actions and adds exactly 410 production/generated-main actions, for 680
|
||||
distinct actions. Both stages run every product, produce byte-identical ordered
|
||||
coordinator output and all 680 action artifacts, and perform no compilation on a
|
||||
second equivalent public request. It checks all 104 test-bearing result labels
|
||||
and all 205 no-test result labels, while a builder-boundary observer records
|
||||
exactly one invocation containing all 309 descriptors for each cold and warm
|
||||
public request. Existing focused observers continue to prove
|
||||
dependency-first/root-first canonical reuse, root-only/combined artifact
|
||||
identity, exact reordered-product trace bytes, and persistent request-directory
|
||||
stability.
|
||||
|
||||
This representation follows the semantic separation and scalable action
|
||||
construction in the pinned official Go 1.26.5 source, identified by
|
||||
[`VERSION`, lines 1–2](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/VERSION#L1-L2),
|
||||
at commit `c19862e5f8415b4f24b189d065ed739517c548ba`:
|
||||
|
||||
- Go's loader states that repeated package lookup returns the same pointer
|
||||
([`pkg.go`, lines 633–636](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L633-L636)),
|
||||
resolves canonical path and directory before package-data lookup
|
||||
([lines 863–911](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L863-L911)),
|
||||
and reuses the package cached under the resolved `ImportPath`
|
||||
([lines 757–768](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L757-L768)).
|
||||
- A Go builder has one command-global action cache, while each action's
|
||||
dependencies are a dynamically accumulated slice
|
||||
([`action.go`, lines 38–45](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L38-L45),
|
||||
[lines 84–89](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L84-L89)).
|
||||
The cache key is operation plus canonical package pointer and returns the
|
||||
existing action
|
||||
([lines 202–206](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L202-L206),
|
||||
[lines 437–447](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L437-L447)).
|
||||
- Go constructs an archive compile action and dynamically appends actions only
|
||||
for the package's direct imports
|
||||
([lines 628–659](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L628-L659)).
|
||||
It separately interns a link action rooted in that cached compile action
|
||||
([lines 919–958](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L919-L958))
|
||||
and dynamically expands the complete transitive link closure
|
||||
([lines 1034–1068](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/action.go#L1034-L1068)).
|
||||
Requested package actions are likewise accumulated with `append`
|
||||
([`build.go`, lines 519–534](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/build.go#L519-L534),
|
||||
[lines 551–558](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/build.go#L551-L558)).
|
||||
- During execution, Go builds one package from its own source list
|
||||
([`exec.go`, lines 721–790](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L721-L790),
|
||||
[lines 928–935](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L928-L935)),
|
||||
maps its direct action dependencies into compiler inputs
|
||||
([lines 864–884](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L864-L884)),
|
||||
and links the root archive with mappings for the complete link-action closure
|
||||
([lines 1592–1647](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/exec.go#L1592-L1647)).
|
||||
- `go/build` keeps directory, import identity, declared name, and ordinary,
|
||||
internal-test, and external-test file lists separate
|
||||
([`build.go`, lines 436–493](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L436-L493));
|
||||
`ImportDir` explicitly processes the named directory
|
||||
([lines 521–525](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L521-L525)),
|
||||
reads precisely that directory
|
||||
([lines 859–900](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L859-L900)),
|
||||
and assigns accepted files to the separate package-owned lists
|
||||
([lines 948–1039](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L948-L1039)).
|
||||
- Go's test loader explicitly returns generated main, internal production-plus-
|
||||
test, and external-test packages, reusing production when valid
|
||||
([`test.go`, lines 85–102](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L85-L102));
|
||||
constructs the internal, external, and generated-main variants separately
|
||||
([lines 175–293](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L175-L293));
|
||||
dynamically appends, sorts, and deduplicates generated-main imports
|
||||
([lines 315–376](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L315-L376));
|
||||
and uses copy-on-write test variants while preserving unaffected package
|
||||
objects
|
||||
([lines 421–474](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L421-L474)).
|
||||
|
||||
WW adopts those package/action distinctions and scalable dependency
|
||||
accumulation, but not Go's build IDs, module system, importcfg, cache/CAS,
|
||||
preloader, parallel action scheduler, or network behavior. Normal local WW
|
||||
builds and tests remain offline, manifest-free, registry-free, database-free,
|
||||
CAS-free, and network-free.
|
||||
|
||||
## 12. Candidate architectures and hard-gate decision
|
||||
|
||||
Five candidates were developed as coherent systems, not as feature bins.
|
||||
|
||||
Reference in New Issue
Block a user