wcc,ww,os: atomic pkgcache store via temp+rename, both stages (#104)

The out/.pkgcache content-keyed store copied each artifact IN-PLACE
(cp -f / copyfile) to the fixed paths P.wwi/P.o/P.key. Key-last gave
crash-consistency but NOT concurrent-read safety: two same-stage builds
of a shared lib pkg (rt/time/os) target one out/.pkgcache/<pkg>/P.{wwi,o};
once an early finisher writes P.key, a later build's cache_lookup copies
P.wwi/P.o while a mid-finisher is still mid-write -> torn read -> corrupt
link / cs!=ww. The key is content-only, so it is purely the non-atomic
write.

Fix (Go-build-cache pattern, both stages in lock-step, rule 10): write
each artifact to a per-pid same-dir temp (P.wwi.tmp.<pid> etc.) then
rename() into place. Same dir => rename is atomic (cross-fs is not);
per-pid temp => concurrent writers don't clobber each other mid-copy;
content-keyed => last-writer-wins is byte-identical. Key renamed LAST so
a reader that sees the new key always finds complete artifacts. On any
mid-store error the per-pid temps are unlinked so a failed store leaves
no litter (cstage goto cleanup; wwstage cachermtmp helper).

  cstage cmd/ww/main.c cache_store: libc rename(2) + getpid().
  wwstage selfhost/cmd/ww/main.ww cachestore: new os.rename + cachetmp.
  lib/os/os.ww: add rename(2) (RENAME=82), ref/hare/os/os.ha:17 -- returns
    raw i32 errno like sibling remove/mkdir/rmdir (ww's os is the flat
    syscall floor, no fs:: layer); a second pathbuf2 slot holds newpath
    since kpath's single pathbuf can't carry both paths.

cache_lookup is unchanged: it reads cache->private scratch, and an atomic
source is never torn.

The torn-read race is closed BY CONSTRUCTION; a deterministic behavioral
regression-guard isn't feasible through the product build path (content-
keying => concurrent COLD builds all MISS+STORE, never HIT-read a mid-store
entry; a warm cache is never re-stored). The deferred white-box guard is
TASK #105. A WHY-comment at both fix sites records this.

Tests: 989_sepbuild_run KEEPS its private per-pid WW_PKGCACHE -- the
comment is corrected: the pin is NOT a torn-read mask (closed by
construction) but cold-compile isolation for the test's INTERMEDIATE
(.s/.unit.ww) byte-id compare, which a cache HIT legitimately skips
producing. The former 989_pkgcache_atomic_run is renamed to
989_pkgcache_concurrent_run and HONESTLY relabeled: it is a concurrent
shared-cache build-correctness smoke (N concurrent --sep builds sharing
one cache -> every binary byte-identical to an isolated reference + correct
run, both stages), NOT a torn-read/atomicity proof (a review revert-
experiment proved the original claim vacuous). Shrunk to 4 concurrent
builds x 1 batch x both stages. COLD/dev-only, off every byte-id/bootstrap
gate.

selfhost/cmd/ww/main.combined.ww remains stale (its writer was deleted at
the M4 E3-C1 flip; #90 deletes the file) -- not regenerated.

make test: all 445 passed; make sizelint clean; 990-997 byte-id hold.

Makefile

@@ -570,6 +570,7 @@ TESTS	= $(BIN)/test_smoke $(BIN)/test_lex $(BIN)/test_parse $(BIN)/test_check \
 	  $(BIN)/test_sepcycle_dup \
 	  $(BIN)/test_separchive_run \
 	  $(BIN)/test_pkgcache_run \
+	  $(BIN)/test_pkgcache_concurrent_run \
 	  $(BIN)/test_c6soak_run \
 	  $(BIN)/test_septest_run \
 	  $(BIN)/test_coloimport_sep \
@@ -3243,6 +3244,17 @@ $(BIN)/test_pkgcache_run: test/wcc/989_pkgcache_run.c $(BIN)/ww $(BIN)/ww_ww \
 		$(BIN)/w6l $(BIN)/w6l_ww $(LIB)/libwwrt.a | $(BIN)
 	$(CC) $(CFLAGS) -o $@ $<
 
+# 989_pkgcache_concurrent_run — #104 smoke: concurrent --sep builds sharing one
+# out/.pkgcache all produce correct, byte-identical binaries (not a torn-read
+# atomicity guard — that is closed by construction, #105). N concurrent builds
+# share one cache + one lib pkg; each binary must be byte-identical to an
+# isolated reference + run correctly. COLD/dev-only (off every byte-id/bootstrap
+# gate). Needs both driver + compiler + asm + linker stages + libwwrt.a.
+$(BIN)/test_pkgcache_concurrent_run: test/wcc/989_pkgcache_concurrent_run.c $(BIN)/ww $(BIN)/ww_ww \
+		$(BIN)/w6c $(BIN)/w6c_ww $(BIN)/w6a $(BIN)/w6a_ww \
+		$(BIN)/w6l $(BIN)/w6l_ww $(LIB)/libwwrt.a | $(BIN)
+	$(CC) $(CFLAGS) -o $@ $<
+
 # 989_c6soak_run — M3-tail commit-6 dual-path soak gate, phase-1 leg (#46
 # c6). Builds real lib chains (encoding.utf8 dotted + strings/bytes same-leaf
 # `contains`) BOTH combined and --sep, asserts combined==sep run-equivalence

cmd/ww/main.c

@@ -809,28 +809,56 @@ cache_lookup(struct sepgraph *g, int pi, const char *manifest,
 	return 1;
 }
 
-/* On MISS, persist the freshly compiled artifacts then the manifest. The key
- * is written LAST so a crash mid-store never leaves a key whose artifacts are
- * absent/partial (the next run simply re-misses). */
+/* On MISS, persist the freshly compiled artifacts then the manifest. Each is
+ * copied/written to a per-pid same-dir temp then rename()d into place: rename
+ * is atomic within one filesystem (cross-fs is not), so a concurrent
+ * cache_lookup never observes a half-written P.wwi/P.o/P.key (#104). The
+ * per-pid temp name keeps two concurrent writers from clobbering each other
+ * mid-copy; content-keyed ⇒ last-writer-wins is byte-identical. The key is
+ * renamed LAST so a reader that sees the new key always finds complete
+ * artifacts, and a crash mid-store never leaves a key without them.
+ *
+ * The torn-read race is thus closed BY CONSTRUCTION. A deterministic
+ * behavioral regression-guard isn't feasible through the product build path:
+ * content-keying means concurrent COLD builds all MISS at lookup and STORE —
+ * none HIT-reads a mid-store entry — and a warm cache is never re-stored, so
+ * "store concurrent with a HIT-read of the same entry" can't be forced. The
+ * deferred white-box guard is TASK #105; 989_pkgcache_concurrent_run smokes
+ * that concurrent shared-cache builds stay correct. On any mid-store error the
+ * per-pid temps are unlinked so a failed store leaves no litter. */
 static void
 cache_store(struct sepgraph *g, int pi, const char *manifest,
     const char *wwi, const char *obj)
 {
 	char dir[1024], keyp[1100], cwwi[1100], cobj[1100], cmd[4096];
+	char twwi[1200], tobj[1200], tkey[1200];
+	int pid = (int)getpid();
+	FILE *f;
 	pkgcache_dir(g, pi, dir, sizeof dir);
 	snprintf(cmd, sizeof cmd, "mkdir -p '%s'", dir);
 	if (run(cmd) != 0) return;
 	snprintf(cwwi, sizeof cwwi, "%s/P.wwi", dir);
 	snprintf(cobj, sizeof cobj, "%s/P.o", dir);
 	snprintf(keyp, sizeof keyp, "%s/P.key", dir);
-	snprintf(cmd, sizeof cmd, "cp -f '%s' '%s'", wwi, cwwi);
-	if (run(cmd) != 0) return;
-	snprintf(cmd, sizeof cmd, "cp -f '%s' '%s'", obj, cobj);
-	if (run(cmd) != 0) return;
-	FILE *f = fopen(keyp, "wb");
-	if (f == NULL) return;
+	snprintf(twwi, sizeof twwi, "%s/P.wwi.tmp.%d", dir, pid);
+	snprintf(tobj, sizeof tobj, "%s/P.o.tmp.%d", dir, pid);
+	snprintf(tkey, sizeof tkey, "%s/P.key.tmp.%d", dir, pid);
+	snprintf(cmd, sizeof cmd, "cp -f '%s' '%s'", wwi, twwi);
+	if (run(cmd) != 0) goto cleanup;
+	snprintf(cmd, sizeof cmd, "cp -f '%s' '%s'", obj, tobj);
+	if (run(cmd) != 0) goto cleanup;
+	f = fopen(tkey, "wb");
+	if (f == NULL) goto cleanup;
 	fputs(manifest, f);
 	fclose(f);
+	if (rename(twwi, cwwi) != 0) goto cleanup;
+	if (rename(tobj, cobj) != 0) goto cleanup;
+	if (rename(tkey, keyp) != 0) goto cleanup;
+	return;
+cleanup:
+	unlink(twwi);
+	unlink(tobj);
+	unlink(tkey);
 }
 
 /* build_one_sep — the --sep orchestration: discover_deps, reverse_topo,

lib/os/os.ww

@@ -37,6 +37,7 @@ type nr = enum i64 {
 	EXECVE      = 59,
 	EXIT        = 60,
 	WAIT4       = 61,
+	RENAME      = 82,
 	MKDIR       = 83,
 	RMDIR       = 84,
 	UNLINK      = 87,
@@ -89,6 +90,10 @@ export fn exit(code: i32) void = {
 // .ai/probe_tagged_return_pointer_payload.ww.
 export def PATH_MAX: i32 = 4096;
 let pathbuf: [4096]u8;
+// Second path slot: [[rename]] needs both old+new NUL-terminated at once,
+// which the single [[pathbuf]] kpath slot can't hold (see kpath's
+// non-reentrancy note).
+let pathbuf2: [4096]u8;
 
 // ref/hare/sys/+linux/types.ha:886-888. ww folds `sys` into `os`, so the
 // std fd NUMBERS live here (the sys role). Typed i32, NOT io.file as in
@@ -301,6 +306,23 @@ export fn rmdir(path: str) i32 = {
 	return syscall1(nr.RMDIR, p: i64): i32;
 };
 
+// rename — rename(2). Atomic when oldpath and newpath are on the same
+// filesystem; cross-fs is not. Returns 0 on success, negative errno
+// otherwise. Mirrors Hare's os::rename (ref/hare/os/os.ha:17), but
+// returns the raw i32 errno like sibling remove/mkdir/rmdir rather than
+// Hare's (void | fs::error): ww's os is the flat syscall floor, with no
+// fs:: error layer. newpath lands in the second [[pathbuf2]] slot since
+// kpath's single [[pathbuf]] can't hold both paths at once.
+export fn rename(oldpath: str, newpath: str) i32 = {
+	let p: *u8 = kpath(oldpath);
+	if (p == nil: *u8) { return -36i32; };
+	if (newpath.len + 1 >= PATH_MAX) { return -36i32; };
+	let i: i32 = 0;
+	for (i < newpath.len) { pathbuf2[i] = newpath[i]; i += 1; };
+	pathbuf2[newpath.len] = 0u8;
+	return syscall2(nr.RENAME, p: i64, (&pathbuf2[0]): i64): i32;
+};
+
 // mkdirs — recursive mkdir. Creates `path` and any non-existent
 // parent directories with the given mode. EEXIST is silently
 // accepted (matches Hare's `errors::exists` skip in os::mkdirs);

selfhost/cmd/ww/main.ww

@@ -1266,8 +1266,65 @@ fn cachelookup(cacheroot: *u8, g: *sepgraph, pi: i32, manifest: *u8,
 	return 1;
 };
 
-// cachestore — on MISS persist the artifacts then the key (key LAST: a crash
-// mid-store never leaves a key without its artifacts; the next run re-misses).
+// cachetmp — build "<dir>/<name>.tmp.<pid>" (NUL-terminated) into a fresh
+// buffer. #104: a per-pid unique same-dir temp for the atomic cachestore
+// rename. pid is folded in decimal manually (no strconv import here, twin
+// of makeruntmp).
+fn cachetmp(dir: *u8, name: str, pid: i32) *u8 = {
+	let buf: []u8 = alloc([], (os.PATH_MAX: u64))!;
+	buf.len = os.PATH_MAX;
+	let off: u64 = cstrinto(buf.ptr, 0u64, dir);
+	off = byteinto(buf.ptr, off, 47u8);	// '/'
+	off = strinto(buf.ptr, off, name);
+	off = strinto(buf.ptr, off, ".tmp.");
+	let dig: [16]u8;
+	let n: i32 = 0;
+	if (pid <= 0) {
+		dig[n] = 48u8;	// '0'
+		n += 1;
+	} else {
+		let v: i32 = pid;
+		for (v > 0) {
+			dig[n] = ((v % 10) + 48): u8;
+			n += 1;
+			v = v / 10;
+		};
+	};
+	let k: i32 = n - 1;
+	for (k >= 0) {
+		buf.ptr[off] = dig[k];
+		off += 1u64;
+		k -= 1;
+	};
+	cstrseal(buf.ptr, off);
+	return buf.ptr;
+};
+
+// cachermtmp — remove the per-pid cachestore temps on a mid-store error so a
+// failed store leaves no litter (#104). Twin of cstage cache_store's cleanup.
+fn cachermtmp(a: *u8, b: *u8, c: *u8) void = {
+	os.remove(pathstr(a));
+	os.remove(pathstr(b));
+	os.remove(pathstr(c));
+};
+
+// cachestore — on MISS persist the artifacts then the key. Each is copied/
+// written to a per-pid same-dir temp then os.rename'd into place: rename is
+// atomic within one filesystem (cross-fs is not), so a concurrent cachelookup
+// never observes a half-written P.wwi/P.o/P.key (#104). The per-pid temp name
+// keeps two concurrent writers from clobbering mid-copy; content-keyed ⇒
+// last-writer-wins is byte-identical. The key is renamed LAST so a reader that
+// sees the new key always finds complete artifacts, and a crash mid-store
+// never leaves a key without them. Twin of cstage cache_store.
+//
+// The torn-read race is thus closed BY CONSTRUCTION. A deterministic
+// behavioral regression-guard isn't feasible through the product build path:
+// content-keying means concurrent COLD builds all MISS at lookup and STORE —
+// none HIT-reads a mid-store entry — and a warm cache is never re-stored, so
+// "store concurrent with a HIT-read of the same entry" can't be forced. The
+// deferred white-box guard is TASK #105; 989_pkgcache_concurrent_run smokes
+// that concurrent shared-cache builds stay correct. On any mid-store error the
+// per-pid temps are removed so a failed store leaves no litter.
 fn cachestore(cacheroot: *u8, g: *sepgraph, pi: i32, manifest: *u8,
     mlen: u64, wwi: *u8, objf: *u8) void = {
 	let dir: *u8 = pkgcachedir(cacheroot, g, pi);
@@ -1278,13 +1335,26 @@ fn cachestore(cacheroot: *u8, g: *sepgraph, pi: i32, manifest: *u8,
 	let cwwi: *u8 = joinpathlit(dir, "P.wwi");
 	let cobj: *u8 = joinpathlit(dir, "P.o");
 	let keyp: *u8 = joinpathlit(dir, "P.key");
-	if (copyfile(wwi, cwwi) != 0) { return; };
-	if (copyfile(objf, cobj) != 0) { return; };
-	let fd: i32 = os.open(pathstr(keyp),
+	let pid: i32 = os.getpid();
+	let twwi: *u8 = cachetmp(dir, "P.wwi", pid);
+	let tobj: *u8 = cachetmp(dir, "P.o", pid);
+	let tkey: *u8 = cachetmp(dir, "P.key", pid);
+	if (copyfile(wwi, twwi) != 0) { cachermtmp(twwi, tobj, tkey); return; };
+	if (copyfile(objf, tobj) != 0) { cachermtmp(twwi, tobj, tkey); return; };
+	let fd: i32 = os.open(pathstr(tkey),
 	    os.flag.WRONLY | os.flag.CREATE | os.flag.TRUNC, 420i32);	// 0o644
-	if (fd < 0) { return; };
+	if (fd < 0) { cachermtmp(twwi, tobj, tkey); return; };
 	os.writeall(fd, manifest, mlen);
 	os.close(fd);
+	if (os.rename(pathstr(twwi), pathstr(cwwi)) != 0) {
+		cachermtmp(twwi, tobj, tkey); return;
+	};
+	if (os.rename(pathstr(tobj), pathstr(cobj)) != 0) {
+		cachermtmp(twwi, tobj, tkey); return;
+	};
+	if (os.rename(pathstr(tkey), pathstr(keyp)) != 0) {
+		cachermtmp(twwi, tobj, tkey); return;
+	};
 };
 
 // buildonesep — the --sep orchestration: discover deps, reverse-topo,

test/wcc/989_pkgcache_concurrent_run.c

@@ -0,0 +1,227 @@
+/*
+ * 989_pkgcache_concurrent_run — concurrent shared-cache build-correctness
+ * smoke (#104). N distinct root programs that all `import shared` are built
+ * CONCURRENTLY into ONE shared WW_PKGCACHE; every resulting binary must be
+ * byte-IDENTICAL to a reference built in ISOLATION (private cache) and run to
+ * its expected exit. This guards that concurrent `ww build --sep` sharing one
+ * out/.pkgcache produces correct, deterministic binaries — a regression guard
+ * for the cache subsystem under contention (store crash, lock bug, wrong-key
+ * copy, etc.).
+ *
+ * It does NOT prove the temp+rename store is atomic against torn reads: with
+ * content-keying every concurrent cold build MISSES at lookup and STORES (it
+ * never HIT-reads a mid-store entry), so the torn-read window is not forced
+ * here. That race is closed by construction at the cache_store fix site; the
+ * deferred white-box guard is TASK #105.
+ *
+ * Both driver stages (rule 10): the cs and ww references are byte-identical.
+ * Light wwstage-driver test (CLAUDE.md rule 14): all outputs + caches live
+ * under a private /tmp tree, so it is parallel-safe and off every byte-id /
+ * bootstrap gate. Models 989_pkgcache_run conventions.
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/wait.h>
+#include <sys/stat.h>
+
+#define NPROG 4
+#define BASE_EXIT 7	/* shared.v() == 7; rootI returns 7 + I */
+
+static int
+runwait(const char *cmd)
+{
+	int rc = system(cmd);
+	if (rc == -1) return -1;
+	if (WIFEXITED(rc)) return WEXITSTATUS(rc);
+	return 1;
+}
+
+static const char *
+absbin(void)
+{
+	const char *b = getenv("BIN");
+	if (!b) b = "out/bin";
+	if (b[0] == '/') return b;
+	static char buf[2048];
+	char cwd[1024];
+	if (getcwd(cwd, sizeof cwd) == NULL) return NULL;
+	snprintf(buf, sizeof buf, "%s/%s", cwd, b);
+	return buf;
+}
+
+static int
+write_file(const char *path, const char *body)
+{
+	FILE *f = fopen(path, "wb");
+	if (!f) return -1;
+	fputs(body, f);
+	fclose(f);
+	return 0;
+}
+
+static int
+slurp(const char *path, char **outbuf, size_t *outlen)
+{
+	FILE *f = fopen(path, "rb");
+	if (!f) return -1;
+	fseek(f, 0, SEEK_END);
+	long n = ftell(f);
+	fseek(f, 0, SEEK_SET);
+	if (n < 0) { fclose(f); return -1; }
+	char *b = malloc((size_t)n + 1);
+	if (!b) { fclose(f); return -1; }
+	if (fread(b, 1, (size_t)n, f) != (size_t)n) { free(b); fclose(f); return -1; }
+	b[n] = '\0';
+	fclose(f);
+	*outbuf = b;
+	*outlen = (size_t)n;
+	return 0;
+}
+
+static int
+files_eq(const char *a, const char *b)
+{
+	char *ba = NULL, *bb = NULL;
+	size_t na = 0, nb = 0;
+	if (slurp(a, &ba, &na) < 0 || slurp(b, &bb, &nb) < 0) {
+		free(ba); free(bb);
+		return -1;
+	}
+	int eq = (na == nb && memcmp(ba, bb, na) == 0);
+	free(ba); free(bb);
+	return eq ? 0 : 1;
+}
+
+static const char *shared_src =
+	"package shared;\n"
+	"export fn v() i32 = { return 7; };\n";
+
+int
+main(void)
+{
+	const char *bin = absbin();
+	if (!bin) return 1;
+	int fail = 0;
+	char td[64], cmd[8192], batch[32768];
+	char sharedww[1024];
+
+	snprintf(td, sizeof td, "/tmp/wwconc_%d", getpid());
+	snprintf(cmd, sizeof cmd, "rm -rf %s", td);
+	runwait(cmd);
+	snprintf(cmd, sizeof cmd, "mkdir -p %s/shared", td);
+	runwait(cmd);
+
+	snprintf(sharedww, sizeof sharedww, "%s/shared/shared.ww", td);
+	if (write_file(sharedww, shared_src)) { fail++; goto out; }
+
+	/* N distinct roots, each importing the one shared pkg. Distinct return
+	 * (7 + I) so each program — and so each clean binary — is distinguishable,
+	 * while the contended cache entry (shared) is common to all. */
+	for (int i = 0; i < NPROG; i++) {
+		char rootww[1024], src[256];
+		snprintf(rootww, sizeof rootww, "%s/root%d.ww", td, i);
+		snprintf(src, sizeof src,
+		    "package main;\nimport shared;\n"
+		    "fn main() i32 = { return shared.v() + %d; };\n", i);
+		if (write_file(rootww, src)) { fail++; goto out; }
+	}
+
+	struct { const char *drv, *tag; } stg[] = {
+		{ "ww",    "cs" },
+		{ "ww_ww", "ww" },
+	};
+
+	for (int s = 0; s < 2; s++) {
+		const char *drv = stg[s].drv, *tag = stg[s].tag;
+		char shcache[1024];
+		snprintf(shcache, sizeof shcache, "%s/cache.%s", td, tag);
+
+		/* References: each rootI built ISOLATED (private cache, no contention)
+		 * = the clean baseline bytes a concurrent build must reproduce. */
+		for (int i = 0; i < NPROG; i++) {
+			char refcache[1024], refprog[1024], refscr[1024], rootww[1024];
+			snprintf(refcache, sizeof refcache, "%s/refc.%s.%d", td, tag, i);
+			snprintf(refprog, sizeof refprog, "%s/ref.%s.%d", td, tag, i);
+			snprintf(refscr, sizeof refscr, "%s/ref.%s.%d.sepwork", td, tag, i);
+			snprintf(rootww, sizeof rootww, "%s/root%d.ww", td, i);
+			snprintf(cmd, sizeof cmd,
+			    "rm -rf %s %s; WW_PKGCACHE='%s' timeout 240 %s/%s build --sep "
+			    "-o %s %s >/dev/null 2>&1",
+			    refscr, refcache, refcache, bin, drv, refprog, rootww);
+			if (runwait(cmd) != 0) {
+				fprintf(stderr, "concur FAIL[%s]: reference build %d failed\n",
+				    drv, i);
+				fail++;
+			}
+			if (runwait(refprog) != BASE_EXIT + i) {
+				fprintf(stderr, "concur FAIL[%s]: reference %d wrong exit\n",
+				    drv, i);
+				fail++;
+			}
+		}
+
+		/* Fresh shared cache so every build in the batch MISS-stores `shared`
+		 * concurrently → write contention on shared/P.{wwi,o,key}. Distinct
+		 * per-prog output so every binary can be verified, not just one. */
+		snprintf(cmd, sizeof cmd, "rm -rf %s", shcache);
+		runwait(cmd);
+
+		size_t off = 0;
+		off += (size_t)snprintf(batch + off, sizeof batch - off,
+		    "export WW_PKGCACHE='%s'; ", shcache);
+		for (int i = 0; i < NPROG; i++) {
+			off += (size_t)snprintf(batch + off, sizeof batch - off,
+			    "( rm -rf %s/c.%s.%d.sepwork; timeout 240 %s/%s build "
+			    "--sep -o %s/c.%s.%d %s/root%d.ww >/dev/null 2>&1 ) & ",
+			    td, tag, i, bin, drv,
+			    td, tag, i, td, i);
+		}
+		off += (size_t)snprintf(batch + off, sizeof batch - off, "wait");
+		if (off >= sizeof batch) {
+			fprintf(stderr, "concur FAIL: batch cmd truncated\n");
+			fail++; goto out;
+		}
+		runwait(batch);
+
+		for (int i = 0; i < NPROG; i++) {
+			char prog[1024], refprog[1024];
+			snprintf(prog, sizeof prog, "%s/c.%s.%d", td, tag, i);
+			snprintf(refprog, sizeof refprog, "%s/ref.%s.%d", td, tag, i);
+			if (runwait(prog) != BASE_EXIT + i) {
+				fprintf(stderr, "concur FAIL[%s]: concurrent prog i=%d "
+				    "wrong/failed exit\n", drv, i);
+				fail++;
+			}
+			if (files_eq(prog, refprog) != 0) {
+				fprintf(stderr, "concur FAIL[%s]: concurrent binary i=%d "
+				    "!= isolated reference\n", drv, i);
+				fail++;
+			}
+		}
+	}
+
+	/* rule 10: the cs and ww isolated references are byte-identical. */
+	for (int i = 0; i < NPROG; i++) {
+		char a[1024], b[1024];
+		snprintf(a, sizeof a, "%s/ref.cs.%d", td, i);
+		snprintf(b, sizeof b, "%s/ref.ww.%d", td, i);
+		if (files_eq(a, b) != 0) {
+			fprintf(stderr, "concur FAIL: cs != ww reference %d (rule 10)\n", i);
+			fail++;
+		}
+	}
+
+out:
+	snprintf(cmd, sizeof cmd, "rm -rf %s", td);
+	runwait(cmd);
+	if (fail) {
+		fprintf(stderr, "concur: %d check(s) failed\n", fail);
+		return 1;
+	}
+	printf("concur: %d concurrent --sep builds sharing one cache, both stages "
+	    "— every binary byte-identical to its isolated reference + correct run; "
+	    "cs==ww references (rule 10)\n", NPROG);
+	return 0;
+}

test/wcc/989_sepbuild_run.c

@@ -236,14 +236,15 @@ main(void)
 	runwait(cmd);
 	mkdir(td, 0755);
 
-	/* E3-C1: with the flip making sep the sole build path, every concurrent
-	 * test now writes the global out/.pkgcache; this test's cs/ww per-package
-	 * byte-id compare on the heavily-shared real lib pkgs (rt/time/os) then
-	 * races a sibling building the same pkg. Pin a private per-pid cache so
-	 * the cs and ww legs are isolated (system() builds inherit this env).
-	 * pkgcache writes are non-atomic (cp -f), the real torn-read bug; this
-	 * hermetic isolation is correct standalone, pending #104 (atomic
-	 * temp+rename, both stages). */
+	/* This test byte-id compares the per-package INTERMEDIATES (.s, .unit.ww)
+	 * between the cs and ww legs, which a cache HIT legitimately skips
+	 * producing (a HIT copies only .wwi/.o and bypasses compose+w6c). So the
+	 * compare requires every package to COLD-compile: pin a private per-pid
+	 * cache to force misses. This is orthogonal to #104 — the atomic temp+
+	 * rename closed the torn-read race (by construction; #105 is the deferred
+	 * white-box guard). Concurrent shared-cache build correctness is smoked by
+	 * 989_pkgcache_concurrent_run. Here the pin is for intermediate-compare
+	 * isolation, not a torn-read mask. */
 	char cachedir[80];
 	snprintf(cachedir, sizeof cachedir, "%s/pkgcache", td);
 	setenv("WW_PKGCACHE", cachedir, 1);