ww source: reject raw NUL characters

This commit is contained in:
2026-08-21 23:03:14 +09:00
parent 9381f8fb8e
commit 4b9968e4e6
10 changed files with 1304 additions and 320 deletions

View File

@@ -8545,6 +8545,119 @@ and diagnostic text is not a persisted-byte contract. Build workdir format
remains `18`, test workdir format remains `19`, and semantic storage format
remains `3`.
### 11.43 Implemented selected-source U+0000 rejection
Every selected physical `.ww` source rejects a raw byte `00` (U+0000) at its
physical line and byte column with exactly `invalid NUL character`. The rule is
source-wide: comments, interpreted-string text, rune text, and between-token
positions cannot turn a raw NUL into payload. An escape spelling such as
`\x00` remains a legal literal value because it is not byte `00` in
the source file. This section is only the raw-U+0000 rule: malformed UTF-8
and the independently implemented per-source BOM boundary in §11.42 are not
changed or broadened here.
#### Pinned evidence, applicability, and measured prior behavior
The sole authority is official Go 1.26.5 at
`c19862e5f8415b4f24b189d065ed739517c548ba`. Its compiler owner,
`(*source).nextch` in
[`cmd/compile/internal/syntax/source.go`, lines 113165](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/syntax/source.go#L113-L165),
detects ASCII zero at lines 121129, reports `invalid NUL character`, and
continues decoding. `TestScanErrors` pins the positioned diagnostic at
[`cmd/compile/internal/syntax/scanner_test.go`, lines 587600](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/cmd/compile/internal/syntax/scanner_test.go#L587-L600),
and compiler testdata [`test/nul1.go`, lines 752](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/test/nul1.go#L7-L52)
requires NUL errors in strings, raw strings, line/block comments, and ordinary
source. Go's independent public scanner has the same rule in
[`go/scanner/scanner.go`, lines 63108](https://github.com/golang/go/blob/c19862e5f8415b4f24b189d065ed739517c548ba/src/go/scanner/scanner.go#L63-L108)
and its tests at lines 790819. These are **behavior directly implemented or
asserted by pinned Go**.
Before this change, directly measured Cstage and WWstage package builds both
accepted a NUL in a line comment, wrote it into the committed synthetic unit,
published byte-identical archive/interface products, and allowed a dotted local
importer to link and run. The retained audit commands used `out/bin/ww` and
`out/bin/ww_ww` with `WW_SRCLIB=/home/kimchi/src/ww/lib` over
`/tmp/ww-pkgaudit.qtjcpO/src/nulcomment`; both exited zero with empty streams.
The same acceptance applied to raw NUL in strings and runes. Those observations
are **directly measured WW behavior**. Applying pinned Go's physical-source
rule independently to WW's selected files is **behavior derived from the
pinned implementation** and is applicable without importing Go's module,
manifest, registry, lock, cache, database, CAS, network, generalized-import,
or source-level build-expression model.
#### Ownership, timing, and four axes
The direct-frontend semantic owners are the logical source-decoder helpers in
`cmd/wcc/lex.c` and their exact twins in `lib/ww/syntax/lex.ww`: each raw zero
is consumed, reports the same positioned error, and is omitted before token
recovery. Filtered token spans preserve that decoder rule through identifiers,
numbers and suffixes, directives, escapes, operators, comments, and EOF. The
line-comment decoder consumes each body once before classifying an internal
module directive, so NUL filtering does not make generated path handling
superlinear. The
shared `internal/wwpackage/package.ww` coordinator owns public package/test
diagnostic precedence: its length-aware preflight reports every raw NUL in an
invalid physical source before its manual package-clause classifier. It is not
another identity policy. Driver source slurping and synthetic-unit composition
preserve byte lengths and are not semantic owners.
- **Go-like build:** after fixed-target filename selection, an invalid selected
root or dependency rejects during loading, before graph completion and before
compiler, assembler, archiver, linker, install, publication, or execution.
Source rejection in a root precedes resolution of that root's missing
imports. A wrong-target file is excluded before this rule and remains unread
by its semantic owners.
- **Go-like test:** selected production, same-package, external-package, and
test-only source each receive the rule before variant construction. Failure
emits the established attributable `FAIL\n` without a generated main, test
process, accounting, `ok` result, retained binary, or public test product.
- **Go-like package:** each selected physical file owns its diagnostic and
position. Declared package name, source role, package conflict handling,
command/test family, physical directory, and exact canonical dotted identity
are unchanged.
- **Go-like import:** invalid bytes create no import edge or graph node. Valid
import spelling, aliases, local/vendor/internal resolution, visibility,
cycle handling, and initialization order remain unchanged.
Thus raw NUL is never an input to manifest-free package identity, graph/action
keys, symbols, `.wwi`, archive naming, publication names, or persistence keys.
It neither changes local dotted-import boundaries nor introduces a manifest.
#### Failure, publication, persistence, and parity
Cold invalid requests create no unit, assembly, object, archive, executable,
capture, `.new`, `.install`, `.wwtxn.*`, or public output. A warm edit that
introduces NUL stops before a producer or install action, preserving the prior
committed unit/interface/assembly/object/archive generation, tool vouchers,
stamp, and public output byte for byte. Removing the NUL restores the ordinary
selected-source fingerprint; exact restoration may reuse the earlier generation.
Existing producer/runtime failure and transaction rollback remain their own
owners because this branch creates no new rollback mechanism.
Lexer and coordinator state are request/source-local. Concurrent valid and
invalid requests keep independent workdirs, captures, diagnostics, and
products; an invalid request cannot contaminate a valid sibling. The rule adds
no process, wait, or cancellation boundary, so signal, timeout, interruption,
process-group cleanup, and ordinary scratch cleanup retain their established
owners. Validation itself leaves no durable residue. Cstage and WWstage are
semantic twins: diagnostics match exactly, and valid unit/compiler/product
bytes retain their existing byte-identity contract.
`raw_nul_is_rejected_in_every_selected_source` in
`test/package/package_test.ww`, with focused C and WW lexer coverage, proves
literal and comment contexts plus adjacency recovery across escapes,
identifiers, numbers and typed suffixes, operators, comment delimiters, and
EOF; it also proves selected/imported builds, wrong-target exclusion, every
directory-test source role, precedence, cold cleanup, warm rollback and reuse,
concurrent isolation, valid escaped-NUL behavior, runtime/publication behavior,
and no-residue/parity observations.
No format bump. This changes invalid-source acceptance and diagnostics only;
valid source composition and valid `.wwi`, assembly, object, archive,
executable, and retained-test-product bytes are unchanged. Build workdir format
remains `18`, test workdir format remains `19`, and semantic storage format
remains `3`.
## 12. Candidate architectures and hard-gate decision
Five candidates were developed as coherent systems, not as feature bins.

View File

@@ -60,6 +60,16 @@ every other source position, including inside string and rune literals and
comments. Apart from that marker rule, the lexer operates on bytes and
non-ASCII bytes are legal only inside string and rune literals and comments.
Each raw byte `00` (U+0000) is invalid at every physical source position,
including in comments and string or rune literal text. It produces one
positioned `invalid NUL character` error at that byte's source position and
is omitted from the lexer's logical character stream before token recovery.
It therefore cannot split an identifier, number, operator, escape, or comment
boundary into different tokens. This is a source-representation rule, before
package/import interpretation; it does not make U+0000 a package, import, or
artifact identity component. An escape spelling such as `"\\x00"` is not a
raw source byte and remains a legal literal value.
### 2.2 Comments
Line comments only, introduced by `//` and running to end of line. There

View File

@@ -429,6 +429,26 @@ before either operation. The marker is source representation only and never
package/import/action/artifact/publication/persistence identity; no test result
is cached.
After filename eligibility and before package-clause classification, each raw
source byte `00` in a selected production, same-package test,
external-package test, or test-only source is one positioned
`invalid NUL character` error. Comments and literal text do not hide it. The
shared coordinator owns that early package/test rejection and reports every
raw NUL in the invalid physical source. On complete direct
frontend inputs, the C and WW source decoders diagnose each raw NUL and omit it
from their logical character streams before token recovery. Thus a NUL cannot
split an escape, identifier, number or suffix, operator, comment delimiter, or
EOF boundary into a stage-dependent second error. Test loading maintains a
stage-equal source-diagnostic stream before import discovery, graph actions,
producers, test execution, accounting, result output, or retained publication.
An invalid selected test request emits only the existing attributable
`FAIL\n`; it does not construct a variant, generated main, test child, or new
persistent generation. Wrong-target exclusion remains first, so an excluded
file with a raw NUL has no diagnostic or persistence effect. A `\\x00` escape
remains a valid literal value. This rule is limited to raw U+0000; malformed
UTF-8 and the independent per-source BOM rule retain their existing, separate
contracts.
After that eligibility boundary and the coordinator's required package-clause
classification and production `@test` validation parses, the delegated loader
performs selected-basename Go 1.26.5 simple-fold preflight before its graph