From f2720689404ff68e8daa5a3e906e2665be16f1e1 Mon Sep 17 00:00:00 2001 From: Hojun-Cho Date: Fri, 5 Jun 2026 03:01:32 +0900 Subject: [PATCH] test/run: missing test binary is a loud FAIL, not a silent SKIP MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A test/wcc/NNN_*.c with no Makefile-wired binary used to write a SKIP status, count toward "all N tests passed", and leave MAKE_TEST_EXIT=0. Two live instances: 953_arrlit_slice_run skipped under a green gate for weeks (committed unwired at bf1037d, wired in the previous commit), and a 938 hit the same hole mid-gate. The runner now treats a missing binary as FAIL naming the file and the target to add, touches the .fail marker, and so flips the summary + exit code. Closed by construction: make test builds every $(TESTS) target before the runner walks test/wcc/*.c, so the missing-binary path is reachable only by an unwired file — there is no legitimate missing-binary SKIP to preserve. Intentional skips keep their existing visible forms (UNIT-mode non-enumeration; in-test per-row skip messages), neither of which folds into the pass count. Count math: the summary N is unchanged (a SKIP already incremented ran); what changes is honesty — pre-fix "all N passed" could include silent skips, post-fix every counted test actually executed. A bare sh test/run without the make-built binaries now fails loud instead of green-skipping the whole suite. --- test/run | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/test/run b/test/run index 186d79b0..208199d4 100755 --- a/test/run +++ b/test/run @@ -20,7 +20,16 @@ if [ "$1" = "--one" ]; then short=${name#[0-9][0-9][0-9]_} bin=$BIN/test_$short if [ ! -x "$bin" ]; then - printf 'SKIP %s (no binary %s)\n' "$name" "$bin" > "$prefix.status" + # A missing binary means an unwired test/wcc file: `make test` + # builds every $(TESTS) target before this runs, so the only way + # to get here is a .c with no Makefile rule. That used to SKIP + # and still count toward "all N tests passed" — 953_arrlit_slice + # sat dark for weeks under a green gate. Fail loud instead. + { + printf 'FAIL %s (no binary %s)\n' "$name" "$bin" + printf 'unwired test: add $(BIN)/test_%s to TESTS + a build rule in Makefile\n' "$short" + } > "$prefix.status" + : > "$prefix.fail" exit 0 fi if "$bin" > "$prefix.out" 2> "$prefix.err"; then