Hojun-Cho/ww
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cmd+selfhost+lib+test: directory-as-module enumeration in driver (#22)

Browse Source 
Replace the cmd/ww + selfhost driver's file-walk import resolver
with true directory enumeration. `import encoding.utf8;` now finds
the lib/encoding/utf8/ directory and concatenates every *.ww file
in it (excluding *test.ww and the driver's *.combined.ww artifacts)
in byte-wise sorted order, instead of just finding the single
lib/encoding/utf8/utf8.ww file. Mirrors Hare's
hare/module/srcs.ha:183 _findsrcs minus tag handling.

Lookup order in both stages: (1) <dir>/<dot-as-slash>/ as directory
→ enumerate. (2) <dir>/<dot-as-slash>.ww as file. The legacy
<dir>/<name>/<name>.ww shape from #18's retained divergence is
dropped per rule-9 Hare-fidelity — Hare has no foo/foo.ha fallback;
a module IS the directory.

Symmetric across cstage (cmd/ww/main.c via opendir+qsort+stat) and
wwstage (selfhost/cmd/ww/main.ww via existing lib/os.getdents64 +
os.stat — no new lib/os surface needed; the rundirtests() walker
in main.ww from #18 was the model). Bootstrap ww2.s==ww3.s==ww4.s
byte-identical post-change.

Bundling justification (rule 11): strict-same-package validation is
bundled because the failure mode is dir-enum's own (a non-dir-enum
compilation unit cannot trigger mismatch across enumerated files).
The natural enforcement site is the driver — the parser can't
distinguish dir-enum concat from file-walk concat. Both stages
peek each file's first `package <name>;` line in expand_dir /
expanddir and exit(1) on mismatch with a precise error pointing
at the offending file. Hare's hare/module/srcs.ha:131 has the
same constraint via its README gate. Other half of #23 (strict
missing-package error tightening — 63 inline-source test wrappers
blocker) stays deferred per its filing.

Parser side (cmd/wcc/parse.c parseuse + lib/ww/parse/decl.ww
parseuse): n->str now carries only the LEAF identifier from a
dotted import. With the driver translating the full dotted path
to a directory walk, the checker only needs the package bareword
(last component) for the N_USE → decl disambiguation walk in
check.c's src_imports / decl_mod. Mirrors Hare's
`use encoding::utf8;` → `utf8::name` semantics
(ref/hare/hare/ast/import.ha:7).

Migration: lib/ww/sym.ww drops `import typ; import ast;`;
lib/ww/parse/parse.ww drops `import expr; import stmt; import
decl;`; lib/ww/lex/lex.ww drops `import tok;` — all sibling
imports auto-resolve via the new dir-enum when callers import the
package directory. lib/strings/, lib/encoding/utf8/utf8test.ww
migrate `import utf8;` → `import encoding.utf8;`. Makefile drops
-I lib/encoding/utf8 stopgap from wwdump_ww + w6c_ww. Seven test
wrappers (700_e2e, 966_strings_run, 970_fmt_run, 971_log_run,
972_fnmatch_run, 982_getopt_run, 990_selfhost) and 995_self_rebuild
drop the -I lib/encoding/utf8 runtime stopgap.

Tests: new 737_direnum C wrapper + test/wcc/data/direnum/ fixtures
pin (a) cross-pkg multi-file dir-enum build at runtime (both stages
must succeed) and (b) strict-same-package mismatch error (both
stages must surface "differs from" + exit non-zero). 738_module_decl
gains row 6 pinning the n_use->str leaf-only storage post-parser
change.

Retained workaround at selfhost/cmd/ww/main.ww expanddir loop:
`names[i][k]` nested-deref-then-index split into
`let nm: *u8 = names[i]; nm[k]` because wwstage cgen miscompiles
the chained form (treats inner u8 element as 8B sizeof *u8 instead
of 1B sizeof u8: extra MOVQ $8 + IMULQ on the inner index, MOVQ
instead of MOVZBQ load). Inline rule-8 WHY comment cites task #24
(wwstage cgen chained-index inner element size on **T). Two-step
form routes through the bare-pointer index path which both stages
handle byte-identically.

Class A wwstage cgen UNDER (chained-index inner element size on
**T) surfaced first time the codebase exercises the **T[i][k]
shape via enumeratedir() — corpus-coverage-blind landmine pattern,
same family as the trio (#27/#28/#31) from STATUS-5.

112/112 ok. ww2 == ww3 == ww4 byte-id holds.
This commit is contained in:
Hojun-Cho
2026-05-18 19:22:27 +09:00
parent 79d9528a00
commit 9e0816e199
31 changed files with 2521 additions and 1564 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
cmd/wcc/parse.c
View File

@@ -1241,17 +1241,23 @@ parseblock(Parser *p)
/* ------- top-level decls ------------------------------------------- */
/* `import encoding.utf8;` — the driver resolves the dotted path to a
* directory; the checker only needs the leaf (`utf8`) as the module
* bareword for n_use→decl disambiguation, mirroring Hare's
* `use encoding::utf8;` → `utf8::name` (ref/hare/hare/ast/import.ha:7
* stores `ident: []str` but identifier-resolution uses the last
* component). */
static Node *
parseuse(Parser *p)
{
Pos pp = p->cur.pos;
expect(p, TK_USE);
Node *n = newnode(p->a, N_USE, pp);
const char *base = expectident(p);
const char *leaf = expectident(p);
while (accept(p, TK_DOT))
base = aprintf(p->a, "%s.%s", base, expectident(p));
n->str = base;
n->strlen = strlen(base);
leaf = expectident(p);
n->str = leaf;
n->strlen = strlen(leaf);
expect(p, TK_SEMI);
return n;
}

315
cmd/ww/main.c
View File

@@ -80,35 +80,48 @@ import_add(struct ImportSet *s, const char *path)
s->paths[s->n++] = strdup(path);
}
/* try <dir>/<name>.ww then <dir>/<name>/<name>.ww — symmetric with
* wwstage locatein (selfhost/cmd/ww/main.ww) for byte-identical
* driver output (rule 10).
*
* Retained divergence from brief: directory-as-module enumeration
* NOT implemented in either stage. The user's "module IS directory"
* mental model is partially honored via the `package` keyword + file-
* walk + sibling `import` chain; true dir enumeration (lib/foo/*.ww
* concatenated atomically without sibling import statements) is
* deferred to task #22. The cstage scaffold (enumerate_dir + qsort +
* is_testfile + dotpath_to_slash) was drafted and reverted during
* #18 because the symmetric wwstage port needs a ww-side
* getdents64 walker (~150-200 lines new ww in selfhost driver) and
* the symmetric stage-rebuild blew the context budget mid-flight.
* Rule 7 + rule 8 documentation. */
static int
locate_import_in(const char *dir, const char *name, char *out, size_t outsz)
/* Translate dots in an `import` name to slashes for path lookup.
* `encoding.utf8` → `encoding/utf8`. Mirrors Hare hare(1)'s
* use-path → fs-path mapping (ref/hare/hare/module/srcs.ha:78
* builds the same shape via path::push per ident part). */
static void
import_path_form(const char *name, char *out, size_t outsz)
{
snprintf(out, outsz, "%s/%s.ww", dir, name);
if (access(out, 0) == 0) return 1;
snprintf(out, outsz, "%s/%s/%s.ww", dir, name, name);
if (access(out, 0) == 0) return 1;
size_t i;
for (i = 0; i + 1 < outsz && name[i] != '\0'; i++)
out[i] = (name[i] == '.') ? '/' : name[i];
out[i] = '\0';
}
/* try <dir>/<path>/ as a directory, then <dir>/<path>.ww as a file.
* Sets *is_dir on hit. Symmetric with wwstage locatein for byte-id
* driver output (rule 10). The legacy <dir>/<name>/<name>.ww form
* was dropped in task #22 — directory-as-module enumeration replaces
* it, mirroring ref/hare/hare/module/srcs.ha (Hare has no fallback
* matching `foo/foo.ha`; a module IS the directory). */
static int
locate_import_in(const char *dir, const char *path_form, char *out,
size_t outsz, int *is_dir)
{
struct stat st;
snprintf(out, outsz, "%s/%s", dir, path_form);
if (stat(out, &st) == 0 && S_ISDIR(st.st_mode)) {
*is_dir = 1;
return 1;
}
snprintf(out, outsz, "%s/%s.ww", dir, path_form);
if (access(out, 0) == 0) {
*is_dir = 0;
return 1;
}
return 0;
}
/* Walk a colon-separated dirlist trying to resolve `name`. Returns 1
* on the first hit. */
/* Walk a colon-separated dirlist trying to resolve `path_form`. Returns
* 1 on the first hit and writes the concrete path + dir/file marker. */
static int
locate_import(const char *dirs, const char *name, char *out, size_t outsz)
locate_import(const char *dirs, const char *path_form, char *out,
size_t outsz, int *is_dir)
{
const char *p = dirs;
while (*p) {
@@ -119,7 +132,8 @@ locate_import(const char *dirs, const char *name, char *out, size_t outsz)
if (n >= sizeof dir) n = sizeof dir - 1;
memcpy(dir, p, n);
dir[n] = '\0';
if (locate_import_in(dir, name, out, outsz)) return 1;
if (locate_import_in(dir, path_form, out, outsz,
is_dir)) return 1;
}
if (!e) break;
p = e + 1;
@@ -127,11 +141,134 @@ locate_import(const char *dirs, const char *name, char *out, size_t outsz)
return 0;
}
/* Recursively expand `path`: for each top-level `use IDENT;` we find,
* resolve the import and expand it first, then append our own bytes.
* Already-visited paths are skipped. Each source carries its own
* `module <name>;` declaration (the parser stamps decls from it), so
* the driver no longer injects a `// MODULE:` marker. */
/* memcmp-based string compare for qsort. Byte-wise total order is
* locale-independent; rule-10 byte-id requires the two stages sort
* the same way. (strcmp would work today but Hare-fidelity points
* at memcmp via ref/hare/sort/cmp/cmp.ha:9.) */
static int
strs_cmp(const void *a, const void *b)
{
const char *sa = *(const char *const *)a;
const char *sb = *(const char *const *)b;
return strcmp(sa, sb);
}
/* enumerate_dir_ww — collect *.ww names in `dirpath` excluding
* *test.ww, sort byte-wise. Returns count; caller frees entries. */
static int
enumerate_dir_ww(const char *dirpath, char ***out_files)
{
DIR *d = opendir(dirpath);
if (d == NULL) { *out_files = NULL; return 0; }
char **arr = NULL;
int n = 0, cap = 0;
struct dirent *ent;
while ((ent = readdir(d)) != NULL) {
const char *nm = ent->d_name;
size_t nl = strlen(nm);
if (nl <= 3) continue;
if (strcmp(nm + nl - 3, ".ww") != 0) continue;
/* skip "*test.ww" (no underscore — bytestest.ww
* ostest.ww utf8test.ww — Hare convention is _test.ha
* but ww corpus settled on the un-underscored shape). */
if (nl >= 7 && strcmp(nm + nl - 7, "test.ww") == 0)
continue;
/* skip "*.combined.ww" — driver-generated concat
* artifacts (the previous build leaves them next to
* the source). They look like .ww but parse-erroring
* when re-included. */
if (nl >= 12 && strcmp(nm + nl - 12, ".combined.ww") == 0)
continue;
if (n + 1 > cap) {
cap = cap ? cap * 2 : 8;
arr = realloc(arr, cap * sizeof *arr);
}
arr[n++] = strdup(nm);
}
closedir(d);
if (n > 1) qsort(arr, n, sizeof *arr, strs_cmp);
*out_files = arr;
return n;
}
static void expand(FILE *out, const char *path, struct ImportSet *visited,
const char *libdir);
/* Scan `path` for its first non-comment-non-blank line; if it starts
* with `package <name>;` write the name into `out` (NUL-terminated)
* and return 1, else 0. Strict-same-package enforcement (task #23
* subset) is bundled here because the failure mode is dir-enum's
* own — a non-dir-enum compilation unit cannot trigger it. Hare's
* hare/module/srcs.ha:131 has the same constraint via the README
* gate; we encode it as same-package across all enumerated files. */
static int
peek_package(const char *path, char *out, size_t outsz)
{
FILE *in = fopen(path, "rb");
if (in == NULL) return 0;
char line[2048];
int found = 0;
while (fgets(line, sizeof line, in)) {
const char *p = line;
while (*p == ' ' || *p == '\t') p++;
if (*p == '\n' || *p == '\0') continue;
if (p[0] == '/' && p[1] == '/') continue;
if (strncmp(p, "package ", 8) != 0
&& strncmp(p, "package\t", 8) != 0) break;
p += 8;
while (*p == ' ' || *p == '\t') p++;
size_t i = 0;
while (i + 1 < outsz && ((p[i] >= 'a' && p[i] <= 'z')
|| (p[i] >= 'A' && p[i] <= 'Z')
|| p[i] == '_' || (p[i] >= '0' && p[i] <= '9')))
out[i] = p[i], i++;
out[i] = '\0';
found = (i > 0);
break;
}
fclose(in);
return found;
}
/* expand_dir — enumerate <dirpath>/*.ww (skip *test.ww), byte-sort,
* recurse into each. Mirrors ref/hare/hare/module/srcs.ha:183
* `_findsrcs` minus tag handling. The visited set still keys on
* concrete file paths so multi-file modules are pulled once.
* Strict-same-package: all enumerated files must declare the same
* `package <name>;` (task #23 subset; failure mode native to
* dir-enum). */
static void
expand_dir(FILE *out, const char *dirpath, struct ImportSet *visited,
const char *libdir)
{
char **files = NULL;
int n = enumerate_dir_ww(dirpath, &files);
char dirpkg[128] = {0};
for (int i = 0; i < n; i++) {
char fp[1024];
snprintf(fp, sizeof fp, "%s/%s", dirpath, files[i]);
char pkg[128];
if (peek_package(fp, pkg, sizeof pkg)) {
if (dirpkg[0] == '\0') {
snprintf(dirpkg, sizeof dirpkg, "%s", pkg);
} else if (strcmp(dirpkg, pkg) != 0) {
fprintf(stderr,
"ww: %s: package %s differs from %s in same module dir %s\n",
fp, pkg, dirpkg, dirpath);
exit(1);
}
}
expand(out, fp, visited, libdir);
free(files[i]);
}
free(files);
}
/* Recursively expand `path`: for each top-level `import IDENT;` we
* find, resolve the import and expand it first, then append our own
* bytes. Already-visited paths are skipped. Each source carries its
* own `package <name>;` declaration (the parser stamps decls from
* it). */
static void
expand(FILE *out, const char *path, struct ImportSet *visited,
const char *libdir)
@@ -158,10 +295,15 @@ expand(FILE *out, const char *path, struct ImportSet *visited,
|| *p == '_' || *p == '.' || (*p >= '0' && *p <= '9'))
if (j + 1 < (int)sizeof name) name[j++] = *p++;
if (j == 0) continue;
char path_form[256];
import_path_form(name, path_form, sizeof path_form);
char ipath[1024];
if (!locate_import(libdir, name, ipath, sizeof ipath))
int is_dir = 0;
if (!locate_import(libdir, path_form, ipath, sizeof ipath,
&is_dir))
continue; /* silently skip if not found */
expand(out, ipath, visited, libdir);
if (is_dir) expand_dir(out, ipath, visited, libdir);
else expand(out, ipath, visited, libdir);
}
rewind(in);
@@ -172,8 +314,9 @@ expand(FILE *out, const char *path, struct ImportSet *visited,
}
static int
build_one(const char *src, const char *out, const char *extra_includes,
const char *extra_libs, const char *extra_libdirs)
build_one(const char *src, int entry_is_dir, const char *out,
const char *extra_includes, const char *extra_libs,
const char *extra_libdirs)
{
const char *c6 = toolpath("WW_W6C", "w6c");
const char *a6 = toolpath("WW_W6A", "w6a");
@@ -199,9 +342,16 @@ build_one(const char *src, const char *out, const char *extra_includes,
* convention (its `hare test` is run from the module dir, making CWD
* == module-dir); our wrappers don't cd, so dirname(src) is the
* closest analog. Also matches cc -I. — source-dir wins ties over
* the system path. locate_import walks left-to-right. */
* the system path. locate_import walks left-to-right.
*
* For a dir entry the source-dir IS src; for a file entry it's
* the dirname. */
char srcd[1024];
{
if (entry_is_dir) {
snprintf(srcd, sizeof srcd, "%s", src);
size_t n = strlen(srcd);
while (n > 1 && srcd[n-1] == '/') srcd[--n] = '\0';
} else {
const char *slash = strrchr(src, '/');
if (slash) {
size_t n = (size_t)(slash - src);
@@ -221,19 +371,29 @@ build_one(const char *src, const char *out, const char *extra_includes,
snprintf(searchpath, sizeof searchpath, "%s:%s", srcd, srcdir);
srcdir = searchpath;
/* Strip extension to derive a stem; e.g. /tmp/foo.ww → /tmp/foo */
/* Derive a stem for .s/.o/.combined.ww side files. For a file
* entry strip the .ww. For a dir entry use <dir>/<basename(dir)>
* so artifacts land inside the module directory. */
char stem[1024];
snprintf(stem, sizeof stem, "%s", src);
char *dot = strrchr(stem, '.');
if (dot && strcmp(dot, ".ww") == 0) *dot = '\0';
if (entry_is_dir) {
const char *b = strrchr(srcd, '/');
const char *base = b ? b + 1 : srcd;
snprintf(stem, sizeof stem, "%s/%s", srcd, base);
} else {
snprintf(stem, sizeof stem, "%s", src);
char *dot = strrchr(stem, '.');
if (dot && strcmp(dot, ".ww") == 0) *dot = '\0';
}
char asmf[1024], obj[1024], combined[1024];
snprintf(asmf, sizeof asmf, "%s.s", stem);
snprintf(obj, sizeof obj, "%s.o", stem);
snprintf(combined, sizeof combined, "%s.combined.ww", stem);
/* Resolve `use X;` imports by concatenating sources into a temp
* file. The compiler then sees one flat source. */
/* Resolve imports by concatenating sources into a temp file. The
* compiler then sees one flat source. Dir entry → enumerate the
* module dir's *.ww (less *test.ww); file entry → start at the
* file. */
{
FILE *cf = fopen(combined, "wb");
if (cf == NULL) {
@@ -241,7 +401,8 @@ build_one(const char *src, const char *out, const char *extra_includes,
return 1;
}
struct ImportSet visited = {0};
expand(cf, src, &visited, srcdir);
if (entry_is_dir) expand_dir(cf, srcd, &visited, srcdir);
else expand(cf, src, &visited, srcdir);
fclose(cf);
for (int i = 0; i < visited.n; i++) free(visited.paths[i]);
free(visited.paths);
@@ -330,41 +491,35 @@ basename_no_ext(const char *path, char *out, size_t outsz)
if (dot && strcmp(dot, ".ww") == 0) *dot = '\0';
}
/* resolve_module: turn a name into a concrete .ww file path.
/* resolve_module: turn a name into a concrete entry path.
* foo.ww → use as-is if it exists
* <existing dir> → <dir>/<basename>.ww (Hare module convention)
* . → <cwd-basename>.ww in the cwd
* foo (bare) → walk cwd:incs:WW_LIB for foo.ww or foo/foo.ww */
* <existing dir> → returns the dir path (caller dir-enumerates)
* . → cwd as a directory
* foo (bare) → walk cwd:incs:WW_LIB; first hit is dir or file.
* Sets *is_dir on hit. Dir resolution drives directory-as-module
* enumeration in build_one. */
static int
resolve_module(const char *name, const char *incs, char *out, size_t outsz)
resolve_module(const char *name, const char *incs, char *out, size_t outsz,
int *is_dir)
{
struct stat st;
if (stat(name, &st) == 0) {
if (S_ISREG(st.st_mode)) {
snprintf(out, outsz, "%s", name);
*is_dir = 0;
return 1;
}
if (S_ISDIR(st.st_mode)) {
char buf[1024];
const char *base;
if (strcmp(name, ".") == 0) {
if (getcwd(buf, sizeof buf) == NULL) return 0;
} else {
snprintf(buf, sizeof buf, "%s", name);
size_t bl = strlen(buf);
while (bl > 1 && buf[bl-1] == '/') buf[--bl] = '\0';
}
const char *b = strrchr(buf, '/');
base = b ? b + 1 : buf;
snprintf(out, outsz, "%s/%s.ww", name, base);
if (access(out, 0) == 0) return 1;
return 0;
snprintf(out, outsz, "%s", name);
*is_dir = 1;
return 1;
}
}
char sp[4096];
search_path(incs, sp, sizeof sp);
if (locate_import(sp, name, out, outsz)) return 1;
return 0;
char path_form[256];
import_path_form(name, path_form, sizeof path_form);
return locate_import(sp, path_form, out, outsz, is_dir);
}
/* Append `path` to a heap string-array. Caller frees each entry + the array. */
@@ -461,13 +616,23 @@ do_build(int argc, char **argv)
libdirs, sizeof libdirs, libs, sizeof libs, &src);
if (src == NULL) src = "."; /* default: build cwd */
char resolved[1024];
if (!resolve_module(src, incs, resolved, sizeof resolved)) {
int is_dir = 0;
if (!resolve_module(src, incs, resolved, sizeof resolved, &is_dir)) {
fprintf(stderr, "ww build: cannot find module %s\n", src);
return 1;
}
char out[1024];
basename_no_ext(resolved, out, sizeof out);
return build_one(resolved, out, incs, libs, libdirs);
if (is_dir) {
char tmp[1024];
snprintf(tmp, sizeof tmp, "%s", resolved);
size_t n = strlen(tmp);
while (n > 1 && tmp[n-1] == '/') tmp[--n] = '\0';
const char *b = strrchr(tmp, '/');
snprintf(out, sizeof out, "%s", b ? b + 1 : tmp);
} else {
basename_no_ext(resolved, out, sizeof out);
}
return build_one(resolved, is_dir, out, incs, libs, libdirs);
}
static int
@@ -481,13 +646,15 @@ do_run(int argc, char **argv)
libdirs, sizeof libdirs, libs, sizeof libs, &src);
if (src == NULL) src = ".";
char resolved[1024];
if (!resolve_module(src, incs, resolved, sizeof resolved)) {
int is_dir = 0;
if (!resolve_module(src, incs, resolved, sizeof resolved, &is_dir)) {
fprintf(stderr, "ww run: cannot find module %s\n", src);
return 1;
}
char tmp[1024];
snprintf(tmp, sizeof tmp, "/tmp/ww_run_%d", getpid());
if (build_one(resolved, tmp, incs, libs, libdirs) != 0) return 1;
if (build_one(resolved, is_dir, tmp, incs, libs, libdirs) != 0)
return 1;
/* exec the built binary with any trailing argv as its argv. */
pid_t pid = fork();
if (pid < 0) { perror("ww: fork"); unlink(tmp); return 1; }
@@ -517,13 +684,15 @@ do_test(int argc, char **argv)
/* not a literal path — try module resolution and run as
* a single test program. */
char resolved[1024];
if (!resolve_module(target, "", resolved, sizeof resolved)) {
int is_dir = 0;
if (!resolve_module(target, "", resolved, sizeof resolved,
&is_dir)) {
fprintf(stderr, "ww test: cannot find %s\n", target);
return 1;
}
char tmp[1024];
snprintf(tmp, sizeof tmp, "/tmp/ww_test_%d", getpid());
if (build_one(resolved, tmp, "", "", "") != 0) return 1;
if (build_one(resolved, is_dir, tmp, "", "", "") != 0) return 1;
int rc = run(tmp);
unlink(tmp);
return rc;
@@ -532,7 +701,7 @@ do_test(int argc, char **argv)
/* single .ww file — build+run it. */
char tmp[1024];
snprintf(tmp, sizeof tmp, "/tmp/ww_test_%d", getpid());
if (build_one(target, tmp, "", "", "") != 0) return 1;
if (build_one(target, 0, tmp, "", "", "") != 0) return 1;
int rc = run(tmp);
unlink(tmp);
return rc;
@@ -558,7 +727,7 @@ do_test(int argc, char **argv)
snprintf(tmp, sizeof tmp, "/tmp/ww_test_%d_%d", getpid(), i);
const char *label = strrchr(files[i], '/');
label = label ? label + 1 : files[i];
int rc = build_one(files[i], tmp, target, "", "");
int rc = build_one(files[i], 0, tmp, target, "", "");
if (rc != 0) {
fprintf(stderr, "FAIL %s (build)\n", label);
fail++;