ww source: reject malformed UTF-8

This commit is contained in:
2026-08-22 03:14:53 +09:00
parent d61805262e
commit 4069eda942
10 changed files with 1146 additions and 30 deletions

View File

@@ -60,6 +60,19 @@ every other source position, including inside string and rune literals and
comments. Apart from that marker rule, the lexer operates on bytes and
non-ASCII bytes are legal only inside string and rune literals and comments.
Every malformed UTF-8 byte in an eligible selected physical source produces
one positioned `invalid UTF-8 encoding` error at its 1-based physical line and
raw-byte column. The byte is consumed and omitted from the lexer's logical
character stream before token recovery. A malformed multi-byte spelling is
therefore diagnosed once for each byte that decodes as U+FFFD with width one;
a correctly encoded U+FFFD is valid. Malformed bytes cannot split an
identifier, number or suffix, operator, escape, comment delimiter, package
keyword, or import spelling into different tokens. Filename and test-role
eligibility precede validation, so an excluded physical file contributes no
UTF-8 diagnostic. This rule is independent of the leading-BOM and raw-NUL
rules below and does not make source bytes package, import, graph, action,
artifact, publication, or persistence identity.
Each raw byte `00` (U+0000) is invalid at every physical source position,
including in comments and string or rune literal text. It produces one
positioned `invalid NUL character` error at that byte's source position and
@@ -296,7 +309,10 @@ ImportPath = ident { "." ident } .
Production excludes selected `*_test.ww`; test variants classify only those
selected test files. An excluded file contributes no declarations, imports,
filename collision, package edge, action, export, artifact, initialization,
test, or persistent invalidation. After eligibility, two distinct selected
test, diagnostic, or persistent invalidation. After eligibility, each
selected physical source is validated for malformed UTF-8 and raw NUL before
package-clause or import interpretation, in the existing byte-sorted file
order. After that source preflight, two distinct selected
basenames in one canonical directory that are equal under Go 1.26.5 Unicode
simple folding are rejected after the coordinator's required package-clause
classification and production `@test` validation parses, but before the