Hojun-Cho/ww
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: prove recursive local package patterns

Browse Source
This commit is contained in:
Hojun-Cho
2026-08-13 23:42:54 +09:00
parent d5fdc27c0a
commit 9a2812347c
2 changed files with 1631 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

329
docs/build-system.md
View File

@@ -2806,8 +2806,11 @@ imports-only parser, unions duplicate imports, byte-sorts direct edges, interns
canonical directory actions, and reports self-imports and stable cycle chains
before compilation.
A directory package consists of its immediate regular non-symlink `.ww` files,
excluding `*_test.ww`, in byte-sorted filename order. Every selected file must
A directory package consists of its immediate `.ww` entries whose basenames do
not begin `.` or `_`: regular files and symlinks targeting regular files are
included under the entry name, while symlinks targeting directories are
ignored. The production variant excludes `*_test.ww`; each variant retains
byte-sorted filename order. Every selected file must
declare the same package. An ordinary importable directory's declaration must
equal the final component of its import path. A selected command directory
instead declares `package main` to validate command kind while retaining its
@@ -2925,11 +2928,13 @@ importer under the existing propagation rule.
### 11.7 Implemented directory package-test slice
Directory tests now enter that same local package loader and build path. The
supported manifest-free commands are `ww test DIR`, `ww test DIR/...`, and
their existing `-run`, `-filter`, `-list`, `-timeout-ms`, `-j`, `-c`, and `-w`
forms. `ww test -c -o test.bin DIR` names the result when the selected directory
has one test variant; the coordinator rejects one output name for a multi-variant
or recursive request. Explicit `ww test FILE` retains its compatibility path.
supported manifest-free forms include `ww build DIR...`, `ww build DIR/...`,
`ww test DIR...`, and `ww test DIR/...`, with overlapping direct and recursive
roots. Test retains its existing `-run`, `-filter`, `-list`, `-timeout-ms`,
`-j`, `-c`, and `-w` forms. `ww test -c -o test.bin DIR` names the result when
the selected directory has one test variant; the coordinator rejects one output
name for a multi-variant or recursive test request. Explicit `ww test FILE`
retains its compatibility path.
The test coordinator still discovers requested directories, enumerates the
test package names, selects variants, executes independent binaries, and emits
@@ -2978,9 +2983,8 @@ invocation, each still-unbound directory is finalized by this exact algorithm:
1. For every import-resolution context that reached the directory, walk that
context's roots in its normal forward precedence: the selected package's
directory, explicit `-I` roots in command order, then `WW_SRCLIB` or the
selected toolchain source root. Recursive package-test requests privately
insert their symlink-resolved discovery root before user `-I` roots, so
descendants retain their complete relative identity.
selected toolchain source root. A request pattern never becomes an import
root and therefore cannot shorten, replace, or donate package identity.
2. Canonicalize each candidate root and require the package directory to be a
strict descendant. Every relative path component must be a non-keyword WW
identifier. Convert separators to dots, then resolve that relative spelling
@@ -3019,11 +3023,13 @@ its derived `_test` compiler path.
The derivation and diagnostics are implemented symmetrically in
`cmd/ww/main.c` and `selfhost/cmd/ww/main.ww`. The package coordinator in
`internal/wwpackage/package.ww` supplies the canonical recursive discovery root,
preserves an explicitly resolved logical request identity, and keys a persistent
request workdir only by the canonical discovery directory. `w6c` and `wcc`
continue to consume and validate the finalized dotted identity; neither tool
performs directory lookup or introduces a package registry.
`internal/wwpackage/package.ww` supplies canonical selected directories and
variant descriptors, preserves an explicitly resolved identity only for one
direct request, and forwards a caller's `-w` semantic-action store unchanged.
It never derives identity or persistent layout from a pattern traversal prefix
and does not add that prefix to import search. `w6c` and `wcc` continue to
consume and validate the finalized dotted identity; neither tool performs
directory lookup or introduces a package registry.
Artifact publication follows the semantic action instead of product order:
production uses the full finalized ordinary identity, internal appends
@@ -3368,7 +3374,7 @@ discovery each compile the shared production action once; recursive `a/foo`
and `b/foo` directories declaring the same `package foo` publish distinct
`a.foo` and `b.foo` variants; and two outside-root `package foo` directories
coexist in one command under distinct reversible local identities. Equivalent
recursive spellings reuse the same persistent request directory without new
recursive spellings reuse the same persistent semantic-action store without new
compilation, while source imports of the reserved local namespace reject before
tool invocation. The exact-argv command root declares `package main` but keeps
its non-`main` canonical identity in the unit, export, archive, compiler argv,
@@ -3594,23 +3600,18 @@ existing complete-path owner check before tools. This preserves collision
checking without serializing machine-specific canonical directories into units
or exports and without a sidecar, registry, or new metadata protocol.
The package coordinator similarly preserves the reversible existing
`d_<escaped-canonical-request-directory>` component while it fits. A longer
canonical request directory uses:
```text
d_<lowercase-sha256("ww-request-workdir-v1:" || canonical-directory)>
```
This key selects only the driver workdir container; it is never a package
identity or action address. Equivalent logical, relative, absolute, `.` and
explicit root-symlink requests canonicalize before this derivation. Inside the
container the driver independently derives every action locator from its full
semantic tuple and validates the committed semantic owner, so even a
coordinator-key collision cannot silently reuse an unrelated package action or
invoke a tool with a shortened identity. Explicit root symlinks are followed
and canonicalized; source-file symlinks and symlinked recursive children retain
their existing rejection/skipping rules.
The package coordinator derives no persistent container from a request
directory or pattern. `-w DIR` names the driver's semantic-action store itself,
and the driver independently derives every action locator in that store from
the full semantic tuple above and validates each committed semantic owner.
Equivalent direct, logical, relative, absolute, dotted, recursive, duplicate,
and explicit-root-symlink requests that select the same canonical actions can
therefore reuse the same slots; request shape cannot split or alias persistent
state. For a delegated request only, a missing caller `-w` directory is created
after graph, identity, visibility, cycle, closure, and output preflight. An
explicit root symlink is followed and canonicalized; a source entry symlink to
a regular file is followed under the entry name, a source-shaped symlink to a
directory is ignored, and symlinked recursive children are not traversed.
The old `SEP_IMPORT_PATH_MAX` and all corresponding 255-byte WWstage import,
variant, local-identity, and generated-main checks are removed. Package names,
@@ -3778,7 +3779,7 @@ actions and its exact linker line still contains the root followed by the full
`p000` through `p299` transitive archive chain and runtime archive, proving that
closure construction—not the wide root's direct imports—crosses the old boundary.
The observer compares every unit, export, assembly, object, archive, and binary
across stages. A second equivalent persistent request invokes no compiler or
across stages. A second equivalent request against the same persistent store invokes no compiler or
assembler. Changing `p257`'s export recompiles exactly `p257`, direct importer
`p256`, and the wide root, and stops before `p255` after `p256` regenerates a
byte-identical export. Closing the chain at `p299 -> p000` produces the complete
@@ -3801,7 +3802,7 @@ The same observer then exercises the public `ww test <tree>/...` coordinator
path. The generated tree has 257 real directories: the 52 test-bearing
directories above plus 205 production-only directories. The coordinator forms
and passes 309 products and 258 contexts (including support) in one driver
request. Reusing the direct request's persistent workdir preserves all 270
request. Reusing the direct request's persistent semantic-action store preserves all 270
existing actions and adds exactly 410 production/generated-main actions, for 680
distinct actions. Both stages run every product, produce byte-identical ordered
coordinator output and all 680 action artifacts, and perform no compilation on a
@@ -3810,8 +3811,8 @@ and all 205 no-test result labels, while a builder-boundary observer records
exactly one invocation containing all 309 descriptors for each cold and warm
public request. Existing focused observers continue to prove
dependency-first/root-first canonical reuse, root-only/combined artifact
identity, exact reordered-product trace bytes, and persistent request-directory
stability.
identity, exact reordered-product trace bytes, and request-shape-independent
persistent action reuse.
This representation follows the semantic separation and scalable action
construction in the pinned official Go 1.26.5 source, identified by
@@ -4141,9 +4142,10 @@ Generated-main-to-variant, generated-main-to-support, and coordinator product
edges remain synthetic and receive no map or retroactive source legality.
`cmd/wwtest` remains a dispatcher. `internal/wwpackage` still discovers and
classifies source groups and submits one command-global union; it performs no
vendor resolution. Its only related state change is transactional cleanup: if
it created a cold hashed request workdir and the driver rejects before a tool
stamp exists, it removes that empty request directory.
vendor resolution. It creates only its removable command-temporary coordination
tree. The private driver creates a missing delegated `-w` directory and a
requested output directory only after all semantic graph and output preflight
succeeds; a rejected request therefore leaves neither directory behind.
Resolution, identity collision checks, contextual legality, dependency-failure
propagation, cycles, command kind, publication paths, and action closures all
@@ -4208,6 +4210,253 @@ direct exports/import maps, archive-only link closures, runtime output,
stage-equal normalized argv/artifacts/binaries, warm no-op production, and cold
and warm rejection-state preservation.
### 11.17 Implemented manifest-free recursive local package-pattern selection
Build and test now share one local request selector. A positional spelling with
no `...` is one explicit directory root. In a spelling containing `...`, each
occurrence in a valid UTF-8 spelling has Go's regular-expression wildcard
semantics; an invalid UTF-8 pattern matches nothing, and a final `/...` also
matches the directory before that suffix. WW applies the local matcher to its
existing manifest-free `DIR/...` interface as well as `./...`, `../...`, and
absolute spellings. It does not interpret the non-filesystem portion as a
module or registry path. Multiple direct roots and patterns may be mixed in one
command.
For build, `--` before the first positional ends option parsing and every
following argument is a package selector, including a spelling that begins
with `-`. Once the first positional has already ended flag parsing, a later
`--` is itself another package argument.
Pattern expansion is request processing only. For each pattern, traversal starts
at the directory prefix before the first `...` and is bounded to that physical
tree. The selector never obtains a traversal or source-root boundary from
another product. It produces canonical physical directory roots; only imports
parsed from their real source files add dependency edges. The request spelling,
wildcard prefix, match membership, output name, request order, and discovery
order never become a package import identity or action key and never enter an
import search path.
The recursive eligibility rules are:
- Directory entries are read completely and byte-sorted before processing.
Every recursively encountered directory whose basename begins `.` or `_`, or
equals `testdata`, is pruned with its subtree. An explicit literal selection
bypasses these traversal exclusions, so those directories remain legal direct
roots.
- A directory owns only its immediate `.ww` directory entries. Basenames
beginning `.` or `_` are ignored. As in `go/build`, a source symlink whose
target is a regular file is read under the symlink entry's byte-sorted name;
a source-shaped symlink to a directory is ignored. A subdirectory never
donates sources to its parent.
- A directory with at least one production or `*_test.ww` source is eligible.
A recursively encountered source-empty directory is silently skipped. A
direct source-empty root is an error. A malformed source-bearing directory is
retained as a root and fails during ordinary package-clause or driver loading;
malformed files in excluded or source-empty trees do not poison the request.
- The traversal does not prune `vendor`. Instead a wildcard cannot consume a
non-terminal exact path component named `vendor`. Thus `DIR/...` may select a
code-bearing terminal `DIR/vendor` but not `DIR/vendor/x`.
`DIR/vendor/...` explicitly selects that vendor root and its descendants until
another non-terminal `vendor` becomes a barrier. `vendorx` is ordinary.
The explicit traversal root is opened after following a directory symlink, as
in Go. It may therefore name a target outside the lexical spelling, but the
target becomes the canonical traversal boundary. Directory symlinks encountered
below that root are never followed, so they cannot escape, create cycles, or
change selection. A cyclic explicit root is rejected while canonicalizing it.
Relative, absolute, dotted, and explicit-root-symlink spellings that reach one
physical package collapse to one canonical root. This physical interning is
WW's stronger command-global identity rule; it deliberately avoids Go's few
GOPATH cases in which different lexical import paths can retain distinct package
objects.
Raw requested spellings are byte-sorted first and each is then lexically cleaned
before traversal. Matched source paths are canonicalized, sorted by canonical
directory and filename, and deduplicated. Products are then byte-sorted by
canonical directory and variant. Duplicate patterns, overlapping patterns, and
canonical aliases therefore select one root/product and reuse one action. Go
itself processes patterns in argv order and suppresses later package objects; WW
performs the stronger final canonical sort required by its
request-order-independent command universe. Reversing request or product order
does not change roots, diagnostics, normalized tool arguments, artifacts, or
runtime output.
An unmatched pattern emits:
```text
ww: warning: "PATTERN" matched no packages
```
`PATTERN` is quoted with the pinned `strconv.Quote` rules, including
deterministic escapes for quotes, backslashes, controls, non-printing Unicode,
and malformed UTF-8 bytes. Warnings are emitted in the
sorted request order. With no remaining roots,
ordinary build without `-o` succeeds as an empty build, while test reports
`ww test: no packages to test`. A build with a non-directory `-o` reports no
packages to build; a directory `-o` reports no main packages to build. A
non-directory `-o` still requires exactly one production root. An existing
directory or spelling ending in `/` receives each selected command under its
canonical directory basename; non-main selected roots receive no named output.
Two commands with the same destination basename are rejected rather than
overwriting one another. All selection, canonicalization, package-clause,
duplicate-output, and unusable-request diagnostics precede producer execution.
Raw `-o` and `-w` spellings and every derived directory-fan-out output, cold
scratch name, persistent tool-record name, and package artifact are bounded and
validated symmetrically before tools; a raw spelling that fits but whose suffix
or command basename does not fit is rejected with the same Cstage/WWstage
diagnostic and no filesystem publication.
For a delegated multi-root or recursive `-S` build, `-w` is required so the
assembly outputs have caller-owned persistent destinations instead of vanishing
with the coordinator's temporary plan.
Build and test begin with the same eligible canonical directory set. Recursive
build removes a source-bearing root that has only test files; an explicitly
selected test-only directory remains an unusable build root and fails. Build
creates one production product per remaining directory. Test retains test-only
directories and, after selection, constructs the already specified isolated
production/no-test, internal production-plus-test, external `_test`, support,
and generated-main actions. Pattern expansion does not create those variants
and cannot make one variant visible to another.
Vendor selection remains distinct from vendor import resolution. Selecting a
directory below `vendor`, literally or through an explicitly vendor-rooted
pattern, keeps its complete canonical local identity; it is never shortened to
the suffix after `vendor`. A generic recursive pattern does not expose vendored
descendants as ordinary short command roots. Independently, an allowed real
source import still searches nearest-first below local `vendor`, creates the
expanded identity described in section 11.16, supplies the required
`--import-map`, and performs spelling and visibility checks for that importer
even when the canonical action already exists.
The ownership split is exact:
- `cmd/ww/main.c` and `selfhost/cmd/ww/main.ww` recognize recursive and
multi-root build/test requests symmetrically and delegate them. Their private
build route consumes production product descriptors in one command-global
package universe. It records library-root completion without linking and
links each command product independently. Recursive build forwards `-S`,
`-L`, and `-l` through the same private product route; assembly-only products
receive completion markers only after their producer pass succeeds. Both
stages allocate the same bounded delegation argv and inherit the existing
environment directly, so delegation adds no WWstage-only environment-copy
allocation or failure point.
- `internal/wwpackage/package.ww` owns pattern cleaning and matching, bounded
directory traversal, source-bearing eligibility, canonical root/product
sorting and deduplication, build-versus-test product classification, output
coordination, and deterministic reporting. It does not resolve a source
import and does not prepend a pattern root to `-I`.
- `cmd/wwtest` remains a dispatcher. The Cstage and WWstage drivers own package
enumeration, canonical identity, contextual import resolution, graph loading,
variants, compilation, archive construction, linking, publication, and
persistent reuse.
Consequently the existing action and tool contracts remain unchanged after
root selection. Every action unit contains only the owner's byte-sorted source
files. A compiler receives exactly the sorted, deduplicated `.wwi` exports of
direct source dependencies and any required vendor import-map binding. Each
executable link receives its root archive and complete reachable archive
closure, never a `.wwi`. Canonical duplicate roots reuse the same action; the
pattern text and the product receiving an output do not affect semantic or
persistent identity.
Selection and package-clause validation finish before the coordinator creates
its removable temporary plan, and the coordinator never creates persistent
state. Driver graph, identity, visibility, cycle, closure, and output validation
finish before a missing delegated work or output directory is created and
before scratch, voucher, status, tool-state, publication, or producer mutation.
If later setup of another requested directory fails, both drivers remove every
empty path prefix created by that setup while preserving all pre-existing caller
directories. A cold rejected pattern leaves no partial request state. The same
rejection against an existing caller work root leaves its marker and every
committed artifact, voucher, and tool record byte-unchanged and invokes no
producer. A valid equivalent direct or recursive warm request remains a
package-production no-op; changed exports continue to propagate through direct
dependencies only.
This slice follows only official Go 1.26.5 source at commit
`c19862e5f8415b4f24b189d065ed739517c548ba`:
- local literal versus wildcard handling, the prefix before the first `...`,
explicit root-symlink following, recursive exclusions, source-empty omission,
and malformed-directory retention are in
[`cmd/go/internal/search/search.go`, lines 276–418](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/search/search.go#L276-L418),
and its quoted unmatched-pattern warning is at
[lines 424–429](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/search/search.go#L424-L429);
- wildcard syntax, the empty match for trailing `/...`, and the non-terminal
`vendor` barrier are in
[`cmd/internal/pkgpattern/pkgpattern.go`, lines 32–106](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/internal/pkgpattern/pkgpattern.go#L32-L106),
including invalid-UTF-8 rejection at lines 75–76,
with non-terminal vendor-element replacement implemented at
[lines 125–137](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/internal/pkgpattern/pkgpattern.go#L125-L137);
- manifest-free request expansion, canonical package loading, first-occurrence
deduplication, and pattern membership are in
[`cmd/go/internal/load/pkg.go`, lines 2922–2965](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L2922-L2965),
while source imports alone recurse through package loading at
[lines 2024–2047](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/pkg.go#L2024-L2047);
- build uses that common matcher, implements single versus directory `-o`, and
omits wildcard-selected test-only roots at
[`cmd/go/internal/work/build.go`, lines 459–559](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/build.go#L459-L559)
and
[lines 731–745](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/work/build.go#L731-L745);
- test uses the same package request set, reports an empty set, and constructs
isolated test variants only afterward at
[`cmd/go/internal/test/test.go`, lines 684–719](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/test/test.go#L684-L719)
and
[lines 1133–1226](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/test/test.go#L1133-L1226);
- immediate directory ownership and hidden/underscore source-name exclusion
are in
[`go/build/build.go`, lines 859–914](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L859-L914)
(including regular-file source symlink following and symlink-to-directory
omission at lines 886–900),
parse-error retention and production/test classification are in
[lines 931–1036](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L931-L1036),
and `NoGoError` eligibility is in
[lines 1076–1082](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L1076-L1082); and
- child symlinks are skipped by `Lstat`-based directory walking and entries are
traversed in byte order at
[`cmd/go/internal/fsys/walk.go`, lines 14–59](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/fsys/walk.go#L14-L59)
and
[`os/dir.go`, lines 109–125](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/os/dir.go#L109-L125).
Build subcommand option termination is delegated by the Go command at
[`cmd/go/main.go`, lines 312–321](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/main.go#L312-L321),
with `--` termination implemented by
[`flag/flag.go`, lines 1074–1089](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/flag/flag.go#L1074-L1089),
and the parse loop stops at the first positional at
[lines 1153–1176](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/flag/flag.go#L1153-L1176).
Unmatched-pattern quoting uses the pinned `strconv.Quote` decoder and escape
rules in
[`strconv/quote.go`, lines 28–123](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/strconv/quote.go#L28-L123),
including the exported `Quote` entry at lines 117–123, and the `IsPrint`
algorithm at
[`strconv/quote.go`, lines 515–559](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/strconv/quote.go#L515-L559),
with its generated tables in
[`strconv/isprint.go`, lines 8–733](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/strconv/isprint.go#L8-L733).
No module cutoff, `go.mod`, module vendor mode, `vendor/modules.txt`, importcfg,
build ID, registry, database, CAS, or network behavior is copied.
The native observer `recursive_tree_discovery` generates every tree
temporarily and runs Cstage and WWstage from independent cold roots. It covers
ordinary and test-only roots, hidden/underscore/testdata exclusions and direct
exceptions, terminal and explicit vendor patterns, an imported expanded vendor
dependency at both package-local and ancestor vendor directories,
overlapping/reversed/duplicate patterns independently in each stage, root,
child-directory, regular-file, and directory-target source symlinks,
canonical aliases, source-empty and malformed directories, build/test variant
selection with an asserted common production-root set, middle-position and
valid-UTF-8 wildcard edge cases, byte-sorted quoted unmatched diagnostics,
multi-command directory output and duplicate-destination rejection, recursive
assembly/link-flag forwarding, build `--` termination, raw-versus-derived
output/scratch/work-path boundaries with zero producer calls, owner-only units,
exact direct exports and import maps, archive-only links, normalized stage-equal tool argv and
artifacts/binaries/output, warm no-op package production, and cold/warm pre-tool
rejection-state preservation. The existing
`vendor_directory_import_resolution` observer supplies the exact compiler
`--import-map` argv proof for imported vendor dependencies, and the existing
diamond and long-closure observers independently prove sorted/deduplicated
direct `.wwi` cardinality and archive-only reachable link closures.
## 12. Candidate architectures and hard-gate decision
Five candidates were developed as coherent systems, not as feature bins.

1422
test/package/package_test.ww
View File

File diff suppressed because it is too large Load Diff