The ww compiler frontend was split across packages lex (lex+tok), ww
(ast+sym+typ), and parse — mirroring Hare's ref/hare/hare/{ast,lex,parse}.
That split's only payoff is third-party reuse, which ww has zero of: the
frontend is consumed by exactly one client, the wcc backend. The split's
cost is a wide cross-package export surface — every fn over a sibling
package's type must export it, and under separate compilation that
re-triggers check_exported_type, plus a phantom `import tok;` (tok lives
in package lex). Consolidate into ONE package lib/ww/syntax/, modelled on
Go's cmd/compile/internal/syntax. The 9 files move in (package syntax);
the intra-frontend mutual references become same-package; wcc and the
tool mains import syntax. No cstage C change (the C frontend mangles from
the source package clause). Internal data shapes (AST kinds, token model,
lexer/parser state) still mirror ref/hare/hare per rule 6/12 — only the
module decomposition collapses; the stdlib is untouched.
USER-approved (#74); spec .ai/rob-frontend-reorg.md (drew2 fidelity-
confirmed). Rule-6 carve-out documented in CLAUDE.md. Dissolves the tok
phantom import; collapses the intra-frontend export sprawl. Byte-id
rebaseline (lex.X/parse.X/ww.X -> syntax.X); cs==ww held. The residual
syntax->wcc export surface (10 types) + the unqualified-ref question are
separate follow-ups (#72/#75).
build_one/buildone gain an explicit objstem: 'ww build -o X' derives
main.{combined.ww,s,o} beside X; 'ww run' uses its per-pid /tmp stem;
default no--o stays next-to-source (load-bearing for the make regen of
tracked combined.ww and the freshness gate). Realizes the redirect TODO
in 995_self_rebuild.c. Kills the concurrent-test torn-read race on
selfhost/cmd/<tool>/main.* (the 991 transient). 993/995/949 pass -o
into their per-pid workdirs; 949's '-o /dev/null' relied on the old
driver ignoring -o (audited: the only live driver case). Atomic
combined write (os.rename) deferred to its own commit - lib/os lacks
rename and that addition is an import-floor regen.
test/run fans out test/wcc/*.c except 950 and 990-997 across $(nproc)
xargs workers, then runs the byte-id gates sequentially in phase 2.
Each worker writes <prefix>.status into a mktemp results dir; the
collector emits them in glob order for deterministic output. Phase 2
sequentiality sidesteps the race introduced by ww_ww writing
intermediates next to every traversed source (#15) — concurrent reads
of selfhost/cmd/<tool>/main.{combined.ww,s,o} would see partial bytes
(#16).
`make test-unit` (UNIT=1) skips phase 2 entirely for a 3.8s inner-loop
check; `make test` runs the full 132 in ~8:30. test-unit goal aligns
with the selfhost-bootstrap project goal: 990-997 + 950 are the
toolchain-rebuild gates, fast feedback is for unit work below them.
990-995 wrap every ww/w6c/w6a/w6l/wwdump invocation in `timeout 180`;
995's hardcoded /tmp/ww_d_hello.ww is now pid-keyed. 995 forks its 5
ww_ww builds concurrently (waitpid for collection) so its solo wall
drops from 3:15 to 1:32. Phase-2 split (parallel readers + sequential
writers) deferred to #17, post #15.
Wall: 9:59 → 8:32 (full) / 3.8s (test-unit, ~158× from baseline).
ww_ww build writes its intermediates (.combined.ww/.s/.o/binary)
next to the source path. When 995 invoked `ww_ww build /abs/path/
selfhost/cmd/wwdump/main.ww` in parallel with 991/992/994 (which
read selfhost/cmd/<tool>/main.s as a fixture), the build mid-wrote
main.s and the fixture-reading test saw a partial 520B view of the
1.4MB file → "C 1422712 vs ww 520 bytes" assembler-byte-diff fail.
The race was latent pre-31594e4 too: parallel test/run + serial
995 always finished 995 last (3min wall), so 991 was already done
by the time 995's wwdump build wrote main.s. Internal-parallel 995
finishes in 1m47, opening the window.
Fix: cp the source to <workdir>/build.ww before invoking ww_ww,
build from build.ww. Driver writes intermediates as build.{combined.ww,
s,o} and binary `build` in workdir; canonical selfhost/cmd/<tool>/
main.* are never touched by 995. Imports still resolve via -I, but
those paths are read-only fixtures.
Verified 3 consecutive `make test` runs 132/132 (4:18, 3:22, 3:16).
Was sequential `for (i = 0; i < 5; i++) rebuild_one(...)` at ~3:15
wall. Split into spawn_build (fork+system ww_ww build) phase and
waitpid+slurp_eq collect phase. The five tool builds now run
concurrently; wall drops from sum(builds) to max(builds).
Per-tool workdir already pid-keyed (/tmp/wwsr_<pid>_<tool>); add
build.err per workdir so concurrent diagnostics don't merge —
collect phase replays the err file on failure. 180s timeout still
inside the cd && ... && ww_ww build string.
All five tools' failure status collected before reporting, so a
multi-divergence run names every tool (not just the first).
make test wall: 4m31s → 3m17s on 8 cores. 132/132 green.
test/run fans test/wcc/*.c across $(nproc) workers via xargs -0 -n2 -P
$JOBS; each worker writes <prefix>.status and an optional <prefix>.fail
sentinel into a mktemp results dir. The driver collects in glob order so
output stays deterministic across runs.
Wraps every ww/w6c/w6a/w6l/wwdump invocation in 990-995 with
`timeout 180`, bounding orphan compilers under 8-way contention. The
direct runwait(exe) calls for fresh-built test binaries get the same
treatment via a "timeout 180 %s" snprintf hop. Motivation: pid 2101 was
a w6c_ww that ran 42h on /tmp/wcas_asm_1326_15.ww until we killed it
during this session; timeout makes that impossible.
993's /tmp/ww_d_hello.ww is now pid-keyed (snprintf %d getpid); without
it concurrent runs (or future shards of 993 itself) would race on the
staging file.
Wall: 9m59s → 4m31s on 8 cores. Same 132/132 status.
Replace the cmd/ww + selfhost driver's file-walk import resolver
with true directory enumeration. `import encoding.utf8;` now finds
the lib/encoding/utf8/ directory and concatenates every *.ww file
in it (excluding *test.ww and the driver's *.combined.ww artifacts)
in byte-wise sorted order, instead of just finding the single
lib/encoding/utf8/utf8.ww file. Mirrors Hare's
hare/module/srcs.ha:183 _findsrcs minus tag handling.
Lookup order in both stages: (1) <dir>/<dot-as-slash>/ as directory
→ enumerate. (2) <dir>/<dot-as-slash>.ww as file. The legacy
<dir>/<name>/<name>.ww shape from #18's retained divergence is
dropped per rule-9 Hare-fidelity — Hare has no foo/foo.ha fallback;
a module IS the directory.
Symmetric across cstage (cmd/ww/main.c via opendir+qsort+stat) and
wwstage (selfhost/cmd/ww/main.ww via existing lib/os.getdents64 +
os.stat — no new lib/os surface needed; the rundirtests() walker
in main.ww from #18 was the model). Bootstrap ww2.s==ww3.s==ww4.s
byte-identical post-change.
Bundling justification (rule 11): strict-same-package validation is
bundled because the failure mode is dir-enum's own (a non-dir-enum
compilation unit cannot trigger mismatch across enumerated files).
The natural enforcement site is the driver — the parser can't
distinguish dir-enum concat from file-walk concat. Both stages
peek each file's first `package <name>;` line in expand_dir /
expanddir and exit(1) on mismatch with a precise error pointing
at the offending file. Hare's hare/module/srcs.ha:131 has the
same constraint via its README gate. Other half of #23 (strict
missing-package error tightening — 63 inline-source test wrappers
blocker) stays deferred per its filing.
Parser side (cmd/wcc/parse.c parseuse + lib/ww/parse/decl.ww
parseuse): n->str now carries only the LEAF identifier from a
dotted import. With the driver translating the full dotted path
to a directory walk, the checker only needs the package bareword
(last component) for the N_USE → decl disambiguation walk in
check.c's src_imports / decl_mod. Mirrors Hare's
`use encoding::utf8;` → `utf8::name` semantics
(ref/hare/hare/ast/import.ha:7).
Migration: lib/ww/sym.ww drops `import typ; import ast;`;
lib/ww/parse/parse.ww drops `import expr; import stmt; import
decl;`; lib/ww/lex/lex.ww drops `import tok;` — all sibling
imports auto-resolve via the new dir-enum when callers import the
package directory. lib/strings/, lib/encoding/utf8/utf8test.ww
migrate `import utf8;` → `import encoding.utf8;`. Makefile drops
-I lib/encoding/utf8 stopgap from wwdump_ww + w6c_ww. Seven test
wrappers (700_e2e, 966_strings_run, 970_fmt_run, 971_log_run,
972_fnmatch_run, 982_getopt_run, 990_selfhost) and 995_self_rebuild
drop the -I lib/encoding/utf8 runtime stopgap.
Tests: new 737_direnum C wrapper + test/wcc/data/direnum/ fixtures
pin (a) cross-pkg multi-file dir-enum build at runtime (both stages
must succeed) and (b) strict-same-package mismatch error (both
stages must surface "differs from" + exit non-zero). 738_module_decl
gains row 6 pinning the n_use->str leaf-only storage post-parser
change.
Retained workaround at selfhost/cmd/ww/main.ww expanddir loop:
`names[i][k]` nested-deref-then-index split into
`let nm: *u8 = names[i]; nm[k]` because wwstage cgen miscompiles
the chained form (treats inner u8 element as 8B sizeof *u8 instead
of 1B sizeof u8: extra MOVQ $8 + IMULQ on the inner index, MOVQ
instead of MOVZBQ load). Inline rule-8 WHY comment cites task #24
(wwstage cgen chained-index inner element size on **T). Two-step
form routes through the bare-pointer index path which both stages
handle byte-identically.
Class A wwstage cgen UNDER (chained-index inner element size on
**T) surfaced first time the codebase exercises the **T[i][k]
shape via enumeratedir() — corpus-coverage-blind landmine pattern,
same family as the trio (#27/#28/#31) from STATUS-5.
112/112 ok. ww2 == ww3 == ww4 byte-id holds.
Plan 9-style w-prefix on the per-arch tools, disambiguating from the
real Plan 9 6c/6a/6l in ref/plan9front/:
cmd/wwc/ → cmd/wcc/ libwwc.a → libwcc.a
cmd/6{c,a,l} → cmd/w6{c,a,l} binary names too
test/wwc/ → test/wcc/ 6 test files w/ w6 prefix
selfhost/cmd mirror in lockstep
bootstrap/amd64/{w6c,w6a,w6l} snapshot binaries (gitignored)
WW_6{C,A,L} → WW_W6{C,A,L} env-var overrides
Plan 9 source-tree refs ("Plan 9 6c shape", ref/plan9front/, etc.)
preserved. Hare-style driver, both C and ww sides:
ww test [path] discover *_test.ww in a directory module, run
each; single-file mode for `ww test foo.ww`
Module-by-name `ww build foo` resolves to foo.ww or foo/foo.ww
via search path (cwd : -I dirs : $WW_LIB)
Default-to-cwd `ww build` / `ww test` build the cwd module
Run pass-through `ww run path arg1 arg2` reaches the program
lib/os: getcwd (79) and getdents64 (217) syscalls power `.` resolution
and directory enumeration on the ww side.
Makefile: wwstage tool deps now include lib/os/os.ww (+ lib/strconv
for wwdump_ww) so lib/* edits force their rebuild instead of leaving
stale binaries — surfaced when test 995 first failed against a stale
w6c_ww built before the lib/os additions.
Test 993 byte-identical parity gate (C-side ww vs ww-side ww_ww on a
build corpus) stays green; all 19 tests pass.
