ww source: reject malformed UTF-8

2026-08-22 03:14:53 +09:00
parent d61805262e
commit 4069eda942
10 changed files with 1146 additions and 30 deletions
--- a/docs/build-system.md
+++ b/docs/build-system.md
@@ -8828,6 +8828,211 @@ This is command parsing and output presentation only. No persisted-byte
 contract changes: build workdir format remains `18`, test workdir format
 remains `19`, and semantic storage format remains `3`.

+### 11.45 Implemented selected-source malformed UTF-8 rejection
+
+Every malformed UTF-8 byte in an eligible selected physical `.ww` source is
+rejected at its 1-based physical line and raw-byte column with exactly
+`invalid UTF-8 encoding`. The source decoder consumes that byte, omits it from
+the logical character stream, and resumes. Consequently, a malformed
+multi-byte spelling is diagnosed once for every byte that decodes as U+FFFD
+with width one, while a correctly encoded U+FFFD remains valid. Validation is
+source-wide: comment and literal contexts do not hide malformed bytes, and an
+invalid byte cannot split an identifier, number or suffix, operator, escape,
+comment delimiter, package keyword, or import spelling into a different token.
+
+This section adds only malformed-UTF-8 validation. It does not reopen the
+per-source leading-BOM contract in §11.42, raw-U+0000 rejection in §11.43, or
+exact output-option parsing in §11.44. BOM, raw NUL, and malformed UTF-8 remain
+independent positioned source conditions and are handled in raw-byte order.
+
+#### Pinned evidence, applicability, and fact classification
+
+The sole authority is official Go 1.26.5 at
+`c19862e5f8415b4f24b189d065ed739517c548ba`:
+
+- compiler reader `(*source).init`, `(*source).pos`/`error`, and
+  `(*source).nextch` establish the 1-based byte-positioned decoding boundary in
+  [`cmd/compile/internal/syntax/source.go`, lines 60–88 and 113–165](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/syntax/source.go#L60-L165);
+- `(*source).nextch` calls `utf8.DecodeRune` at lines 149–150, and a U+FFFD
+  result of width one reports exactly `invalid UTF-8 encoding`, consumes that
+  one byte, and resumes at lines 152–154;
+- compiler-scanner `TestScanErrors` asserts the positioned malformed byte and
+  truncated-`EF` regression in
+  [`cmd/compile/internal/syntax/scanner_test.go`, lines 587–600 and 658](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/syntax/scanner_test.go#L587-L658);
+- compiler testdata requires UTF-8 errors in interpreted and raw strings,
+  comments, identifiers, and ordinary source in
+  [`test/nul1.go`, lines 7–52](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/test/nul1.go#L7-L52);
+- the independent public scanner corroborates width-one malformed decoding in
+  [`go/scanner/scanner.go`, lines 63–108](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/scanner/scanner.go#L63-L108)
+  and its literal test at
+  [`go/scanner/scanner_test.go`, lines 810–811](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/scanner/scanner_test.go#L810-L811);
+- directory enumeration, filename eligibility, source reading, and test-role
+  classification are ordered by `Context.Import`, `Context.matchFile`, and
+  `Context.goodOSArchFile` in
+  [`go/build/build.go`, lines 859–953 and 1005–1036](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L859-L1036),
+  [`Context.matchFile`, lines 1438–1509](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L1438-L1509),
+  and
+  [`Context.goodOSArchFile`, lines 1980–2027](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/build/build.go#L1980-L2027),
+  while internal/external test variants consume those selected lists in
+  [`cmd/go/internal/load/test.go`, `TestPackagesAndErrors`, lines 85–102 and 175–240](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/go/internal/load/test.go#L85-L240).
+
+Those decoder branches, source-selection branches, and official assertions are
+**behavior directly implemented or asserted by pinned Go**. Applying the same
+per-selected-physical-source rule to `.ww` inputs, after WW's fixed-target
+filename/test-role eligibility and before its package/import interpretation,
+is **behavior derived from the pinned implementation**. It honestly applies to
+WW's declared UTF-8 source without adding modules, manifests, registries, lock
+files, caches, databases, CAS, network resolution, quoted/grouped/dot/general
+imports, or a source-level build language.
+
+Before this change, the following observations were **directly measured WW
+behavior**:
+
+- direct `w6c` and `w6c_ww` accepted comments containing stray continuation
+  `80`, lead `FF`, overlong `C0 80`, and surrogate `ED A0 80` bytes, exited
+  zero with empty streams, and emitted byte-identical 89-byte assembly with
+  SHA-256
+  `7385e3ce0107324edc94ee4377c5c2b0262c5b690939cb54f599ed09f9730db8`;
+- both direct compilers accepted raw `FF` in a string and emitted
+  byte-identical 236-byte assembly with SHA-256
+  `4dbe79617badb56d541526ca276a9867d0d717ad9aa0c540f629882f87e3f3ad`,
+  while valid Korean and accented controls remained accepted and
+  stage-identical;
+- both public build stages accepted malformed comments and literals, published
+  runnable binaries, and built and ran an imported dependency containing
+  malformed UTF-8; that dependency executable was 4,317 stage-identical bytes
+  with SHA-256
+  `f792da043743cc512c9d9a41deb4cb4af1e42d35e2716fcd5b57355fd7f566eb`;
+- malformed production, same-package, external-package, and test-only selected
+  sources ran successfully in both stages; one same-package compile-only binary
+  was 112,829 stage-identical bytes with SHA-256
+  `108150d071a3ab4264d021d2c3fdc7a94ff71a00153fdf14bfe8e26520cc5da2`;
+- a corrupted `pack<FF>age` produced a package-clause diagnostic plus
+  `unexpected character 0xff` in Cstage but a generic `unexpected character`
+  in WWstage, with 176-byte versus 171-byte stderr; corrupted imports reached
+  analogous fallback recovery rather than the pinned decoder diagnostic;
+- malformed source before `import nowhere;` allowed missing-import resolution
+  to win, while a malformed same-package test keyword reached the
+  coordinator's unpositioned `invalid or missing package clause`; and
+- malformed wrong-target and ordinary-build-excluded test files were ignored,
+  and reverse-created selected files were still diagnosed in byte-sorted name
+  order.
+
+No installed host Go result supplies any authority or measurement above.
+
+#### True ownership and complete four-axis result
+
+The semantic owners are the source-decoder twins in `cmd/wcc/lex.c` and
+`lib/ww/syntax/lex.ww`, plus the selected-physical-source preflight in
+`internal/wwpackage/package.ww`. The decoders enforce bytewise recovery for
+complete direct and composed compiler inputs. The shared coordinator enforces
+the same validation before its textual package-clause classifier and import
+discovery, so a coordinator fallback cannot outrank the physical-source error.
+The Cstage/WWstage package-unit composers only transport already admitted
+source bytes and are not additional owners.
+
+- **Go-like build:** each eligible selected root, library, or dependency source
+  is validated before its imports complete the graph or any compiler,
+  assembler, archiver, linker, install, publication, or runtime action starts.
+  A source error in a selected root precedes missing, self, cycle, `internal`,
+  vendor, and imported-command resolution. Wrong-target and test-only files
+  excluded from ordinary build are not semantic inputs and are not decoded.
+- **Go-like test:** production, same-package, external-package, and test-only
+  selected physical sources are validated before grouping or variant/product
+  construction. Rejection builds no support action or generated main, starts
+  no test process, emits no accounting or package `ok` line, and retains or
+  publishes no executable. An attributable explicit request keeps its existing
+  command-owned final `FAIL\n` presentation.
+- **Go-like package:** each selected physical file owns its positioned errors;
+  selected filenames retain byte-sorted order. Correctly encoded non-ASCII
+  content remains legal in WW's permitted comment/literal contexts. Declared
+  package names, source roles, package conflicts, command/test classification,
+  and variant boundaries do not change.
+- **Go-like import:** a malformed byte is filtered before it can manufacture,
+  split, or change an import occurrence, qualifier, or edge. Valid spelling,
+  aliases, file-scoped binding, local/vendor/internal resolution, visibility,
+  cycle detection, graph order, and initialization remain unchanged.
+
+Canonical dotted package and import identity remains exact and
+case-sensitive. Physical directory, declared name, alias, path leaf, filename,
+source bytes, artifact name, output path, and linker order remain loader,
+runtime, or presentation metadata only where already specified; none becomes
+package, graph, action, symbol, `.wwi`, publication, or persistence identity.
+
+#### Loading, graph, action, runtime, and diagnostics
+
+Filename and test-role eligibility occurs first. The shared preflight then
+scans eligible selected files in existing byte-sorted order, reporting every
+malformed byte in the first invalid physical file in position order before
+package-clause classification. Lines and columns advance by raw source bytes.
+A legal leading BOM still advances three columns, raw NUL keeps its own exact
+diagnostic, and the three source conditions interleave without one being
+reclassified as another.
+
+An invalid source completes no package node, import edge, test variant,
+support action, or generated-main action. No compiler, assembler, archiver,
+linker, installer, or runtime process is scheduled for that invalid request.
+Independent valid siblings and requests keep the established command-global
+planning and scheduling rules; validation adds no global state and cannot
+cancel or mutate them. Valid loading, graph identity, action order,
+initialization, runtime behavior, result order, and output selection are
+explicit non-effects.
+
+Diagnostics use exact text `invalid UTF-8 encoding` with path, 1-based line,
+and 1-based raw-byte column. Each width-one malformed decode is consumed and
+removed before token recovery, preventing a second stage-specific package,
+import, identifier, literal, operator, escape, or EOF interpretation. Complete
+direct frontend inputs report all malformed bytes. Public package/test loading
+uses the same sequential physical-source preflight and therefore preserves
+Cstage/WWstage diagnostic-byte parity and source-before-resolution precedence.
+Existing valid-input, BOM, NUL, package, import, and output-option diagnostics
+retain their owners and wording.
+
+#### Publication, persistence, artifacts, and failure lifecycle
+
+Cold malformed-source rejection creates no synthetic unit, `.wwi`, assembly,
+object, archive, executable, retained test binary, result status, capture, or
+published output. It leaves no `.new`, `.install`, `.wwtxn.*`, adjacent
+`.sepwork`, tool-stage transaction, or scratch residue. Invalid input has no
+artifact-byte comparison beyond identical absence.
+
+Warm rejection commits no generation and preserves every prior unit,
+interface, assembly, object, archive, tool record, stamp, executable, retained
+binary, and public output byte for byte. Because staging has not begun, the
+source branch requires no new rollback mechanism. Restoring the exact valid
+source follows ordinary content comparison and may reuse the prior committed
+generation. Valid Cstage and WWstage unit, interface, assembly, object,
+archive, generated-main, executable, and retained-test bytes keep their
+existing byte-identity contract.
+
+Producer failure, runtime failure, publication-only failure, and cleanup-only
+failure remain downstream owners and are not redefined; malformed-source
+rejection makes those phases unreachable for the invalid request. Validation
+state is source/request-local. Concurrent valid and invalid requests retain
+independent workdirs, outputs, captures, diagnostics, processes, and
+transactions. The change adds no process, wait, signal, timeout, cancellation,
+or interruption boundary, so existing process-group ownership, interruption,
+rollback, and cleanup remain unchanged. The recipe-owned fixed
+`out/bootstrap` tree is not transaction residue.
+
+#### Proof, twin parity, and formats
+
+Focused C and WW lexer proofs cover valid encodings and encoded U+FFFD;
+invalid leads and continuations; overlong, surrogate, out-of-range, truncated,
+and repeated malformed spellings; token boundaries; and BOM/NUL interaction.
+The WW-native `malformed_utf8_is_rejected_in_every_selected_source` observer
+owns direct compiler, root/dependency build, source/import precedence, every
+test source role, wrong-target selection, cold/warm rejection, exact
+restoration and reuse, valid sibling concurrency, artifact absence, residue
+cleanup, stage diagnostic parity, and valid-artifact byte parity. Concrete
+post-change byte counts, hashes, and gate results are recorded only after
+focused and full validation; they are not inferred from the implementation.
+
+No format bump. This changes invalid-source acceptance and diagnostics only;
+the valid persisted-byte contract is unchanged. Build workdir format remains
+`18`, test workdir format remains `19`, and semantic storage format remains
+`3`. No test-result cache is introduced.
+
 ## 12. Candidate architectures and hard-gate decision

 Five candidates were developed as coherent systems, not as feature bins.
--- a/docs/spec.md
+++ b/docs/spec.md
@@ -60,6 +60,19 @@ every other source position, including inside string and rune literals and
 comments. Apart from that marker rule, the lexer operates on bytes and
 non-ASCII bytes are legal only inside string and rune literals and comments.

+Every malformed UTF-8 byte in an eligible selected physical source produces
+one positioned `invalid UTF-8 encoding` error at its 1-based physical line and
+raw-byte column. The byte is consumed and omitted from the lexer's logical
+character stream before token recovery. A malformed multi-byte spelling is
+therefore diagnosed once for each byte that decodes as U+FFFD with width one;
+a correctly encoded U+FFFD is valid. Malformed bytes cannot split an
+identifier, number or suffix, operator, escape, comment delimiter, package
+keyword, or import spelling into different tokens. Filename and test-role
+eligibility precede validation, so an excluded physical file contributes no
+UTF-8 diagnostic. This rule is independent of the leading-BOM and raw-NUL
+rules below and does not make source bytes package, import, graph, action,
+artifact, publication, or persistence identity.
+
 Each raw byte `00` (U+0000) is invalid at every physical source position,
 including in comments and string or rune literal text. It produces one
 positioned `invalid NUL character` error at that byte's source position and
@@ -296,7 +309,10 @@ ImportPath    = ident { "." ident } .
  Production excludes selected `*_test.ww`; test variants classify only those
  selected test files. An excluded file contributes no declarations, imports,
  filename collision, package edge, action, export, artifact, initialization,
-  test, or persistent invalidation. After eligibility, two distinct selected
+  test, diagnostic, or persistent invalidation. After eligibility, each
+  selected physical source is validated for malformed UTF-8 and raw NUL before
+  package-clause or import interpretation, in the existing byte-sorted file
+  order. After that source preflight, two distinct selected
  basenames in one canonical directory that are equal under Go 1.26.5 Unicode
  simple folding are rejected after the coordinator's required package-clause
  classification and production `@test` validation parses, but before the
--- a/docs/test-system-v2.md
+++ b/docs/test-system-v2.md
@@ -459,6 +459,48 @@ remains a valid literal value. This rule is limited to raw U+0000; malformed
 UTF-8 and the independent per-source BOM rule retain their existing, separate
 contracts.

+After filename and test-role eligibility and before package-clause
+classification, every malformed UTF-8 byte in a selected production,
+same-package test, external-package test, or test-only source is one positioned
+`invalid UTF-8 encoding` error. The shared coordinator scans selected physical
+files in existing byte-sorted order and reports all malformed bytes in the
+first invalid file before package grouping, import discovery, or delegated
+tools. The C and WW source decoders consume each malformed byte, omit it from
+their logical streams, and resume, so the byte cannot manufacture or alter a
+package token, import spelling, binding, or edge or produce stage-specific
+fallback recovery. Correctly encoded U+FFFD and other valid non-ASCII text in
+WW's permitted comment and literal contexts remain valid. A malformed
+multi-byte spelling is diagnosed byte by byte according to width-one UTF-8
+decodes. Raw NUL and per-source BOM diagnostics remain independent and
+interleave with malformed-byte diagnostics in physical byte order.
+
+Malformed selected source rejects before package/test variant construction,
+generated main, compiler, assembler, archiver, linker, runtime, accounting,
+retention, or publication. The existing attributable explicit test request
+still owns its final `FAIL\n`; no package `ok` line or test-result cache is
+created. Cold rejection leaves no unit, `.wwi`, assembly, object, archive,
+binary, capture, status, stage, or transaction residue. Warm rejection commits
+nothing and preserves the prior generation and public output byte for byte;
+restoring the valid bytes follows ordinary exact-content reuse. Producer,
+runtime, publication-only, and cleanup failures are unchanged because source
+rejection precedes those phases. Validation is source/request-local, adds no
+process or signal boundary, cannot contaminate an overlapping valid request,
+and leaves interruption and owned-process cleanup with their established
+owners. Wrong-target and ordinary-build-excluded test sources remain unread by
+this semantic preflight.
+
+`malformed_utf8_is_rejected_in_every_selected_source` in
+`test/package/package_test.ww`, together with focused C and WW lexer coverage,
+is the focused proof owner for valid encodings and encoded U+FFFD; stray leads
+and continuations; overlong, surrogate, out-of-range, truncated, repeated, and
+token-boundary cases; BOM/NUL interaction; root and dependency builds; source
+versus import-resolution precedence; all test roles; wrong-target exclusion;
+cold and warm rejection; exact restoration/reuse; parallel isolation;
+diagnostic and valid-artifact stage parity; and absence of residue. Concrete
+post-change measurements and hashes are recorded only after focused and full
+validation. The completed BOM, raw-NUL, and exact output-option slices are not
+reopened, and canonical dotted package/import identity remains unchanged.
+
 After that eligibility boundary and the coordinator's required package-clause
 classification and production `@test` validation parses, the delegated loader
 performs selected-basename Go 1.26.5 simple-fold preflight before its graph