diff --git a/docs/spec.md b/docs/spec.md new file mode 100644 index 00000000..6147d9af --- /dev/null +++ b/docs/spec.md @@ -0,0 +1,638 @@ +# The ww Language Specification + +Version 0.1 (2026-06-21). **Status: normative-intent, in progress.** + +ww is a systems language: *Hare semantics + CSP concurrency, no garbage +collector*, with a Rob-Pike / Plan-9 sensibility and Go-style explicit +`package`/`import`. This document is the language contract. Where the two +compiler stages disagree with this text, the text is wrong and is a bug +to be filed (CLAUDE.md rule 10 requires the stages to agree with each +other; this spec requires them to agree with *it*). + +Two implementations track this spec and must stay byte-identical +(rule 10): the C bootstrap frontend `cmd/wcc` (the reference) and the ww +reimplementation `lib/ww/syntax`. The token table in §2 is kept +numerically identical between them on purpose. + +This is not a tutorial. It is terse by design, modelled on the Go +language specification (golang.org/ref/spec): EBNF where a grammar is +worth pinning, prose for semantics, and a normative appendix (§12) that +consolidates every sanctioned divergence from Hare. The appendix is the +highest-value part of this document — it is the single normative home +for rules currently scattered across CLAUDE.md, `.ai/`, and project +memory. + +Sections marked **(reserved)** describe surface that is tokenised but not +yet given meaning; they are written when the feature lands. + +--- + +## 1. Notation + +The grammar uses Extended Backus-Naur Form, following the Go spec's +conventions: + +``` +Production = production_name "=" [ Expression ] "." . +Expression = Term { "|" Term } . +Term = Factor { Factor } . +Factor = production_name | token | Group | Option | Repetition . +Group = "(" Expression ")" . +Option = "[" Expression "]" . +Repetition = "{" Expression "}" . +``` + +`|` is alternation, `{}` is zero-or-more, `[]` is zero-or-one, `()` is +grouping. Terminal tokens are in `"quotes"` or are the named token +classes from §2 (`ident`, `int_lit`, …). + +--- + +## 2. Lexical elements + +### 2.1 Source representation + +Source is UTF-8. The lexer operates on bytes; non-ASCII bytes are legal +only inside string and rune literals and comments. + +### 2.2 Comments + +Line comments only, introduced by `//` and running to end of line. There +are no block comments (Hare lineage; `/* */` is not recognised). + +Two comment forms are *driver directives*, not ordinary comments — they +carry module-bundle structure and lex to dedicated tokens: + +- `//ww:module ` — set the mangling module path for the + decls that follow (token `MODPATH`). +- `//ww:module-reset` — reset the current module to the empty string + before a package-less bundled file (token `MODRESET`). + +These exist for the build driver's source bundling and are not written +by hand in ordinary source. + +### 2.3 Tokens + +Tokens are identifiers, keywords, operators/punctuation, and literals. +Whitespace (space, tab, CR, LF) separates tokens and is otherwise +ignored. A semicolon `;` terminates declarations and statements and is +written explicitly (no automatic insertion). + +### 2.4 Keywords + +The following 30 identifiers are reserved and may not be used as ordinary +identifiers: + +``` +as break case chan const continue +def defer else enum export false +fn for if is import let +match nil package proc return static +struct switch true type void yield +``` + +`chan` and `proc` are reserved for the concurrency surface (§11, +reserved). `package` is the package-clause keyword; `import` introduces +an import. + +### 2.5 Operators and punctuation + +``` +Assignment: = += -= *= /= %= &= |= ^= <<= >>= +Arithmetic: + - * / % +Bitwise: & | ^ ~ << >> +Comparison: == != < <= > >= +Logical: && || ! +Grouping: ( ) { } [ ] +Punctuation: , ; : . .. ... @ ? +Arrows: <- -> => +``` + +`<-` (channel send/receive) and `->` are tokenised but reserved (§11). +`=>` separates a `match` arm pattern from its body. `?` and `!` are the +error operators (§9). `:` introduces a type ascription / cast (§8) and a +field/binding type. `..` is the range operator (`for` range form, §7); +`...` is the variadic / spread marker (§6.3). `@` introduces an +attribute (§5.7). + +### 2.6 Identifiers + +``` +ident = letter { letter | digit } . +letter = "A"…"Z" | "a"…"z" | "_" . +digit = "0"…"9" . +``` + +A lone `_` is the *discard* identifier: legal as a binding name to mean +"ignore this", never as a reference. + +### 2.7 Literals + +``` +int_lit = decimal | hex | octal | binary [ suffix ] . +float_lit = digits "." digits [ exponent ] [ suffix ] . +rune_lit = "'" ( byte | escape ) "'" . +str_lit = '"' { byte | escape } '"' . +bool_lit = "true" | "false" . +nil_lit = "nil" . +``` + +- `int_lit` is held as a `u64`. `nil` is the null pointer value (§3.6). +- A numeric literal may carry a *type suffix* naming its type directly: + `42u8`, `0u64`, `1i64`, `3.0f32`. Suffixes are the fixed-width type + names (`i8 i16 i32 i64 u8 u16 u32 u64 f32 f64`). Without a suffix a + literal is an *untyped constant* (§3.5) whose type is inferred from + context. +- Escapes in rune and string literals: `\\ \" \' \n \t \r \0` and + `\xNN` (two hex digits). Rune literals additionally accept the + Unicode escapes `\uHHHH` and `\UHHHHHHHH`, yielding a codepoint. + +--- + +## 3. Types + +A type describes the layout and operations of a value. Every type has a +`size` and an alignment, both queryable with `size(T)` (§8). No size is +ever written as a bare literal in size-computing code — it is routed +through the type table (CLAUDE.md rule 13). + +### 3.1 Boolean and void + +`bool` (1 byte; `true`/`false`). `void` (size 0) is the empty type, used +as a fn result and as the second arm of a nullable pointer (§3.6). + +### 3.2 Numeric types + +Fixed-width integers: `i8 i16 i32 i64` (signed), `u8 u16 u32 u64` +(unsigned). Floats: `f32 f64`. + +Machine-word types (8 bytes on amd64): `int`, `uint`, `uintptr`, `size`. +`int` and `uint` are a full machine word — Go-style, **not** Hare's 32-bit +`int`. The limit constants (`INT_MIN`, `INT_MAX`, `UINT_MAX`) are +**derived** from `size(int)`, never hardcoded to a width (§12). + +`rune` is a 32-bit Unicode codepoint (4 bytes). + +### 3.3 String and slice + +`str` and `[]T` (slice of `T`) are both **24-byte three-word headers**: +`{ ptr, len, cap }`. Their `.ptr`, `.len`, and `.cap` pseudo-fields are +readable (§8.4). A `str` is UTF-8 bytes; reinterpreting `[]u8` as `str` +is `strings.frombytes` (a pure cast — there is no validation; §12). + +Array: `[N]T` is `N` elements inline (size `N * size(T)`). `[_]T` +infers `N` from the initialiser's element count. + +### 3.4 Aggregate and named types + +``` +type S = struct { f0: T0, f1: T1, … }; // struct +type Name = T; // named type / alias +``` + +- `struct` — named fields laid out in declaration order with natural + alignment padding. Duplicate field names are rejected. +- `tuple` — `(T0, T1, …)`. Each element occupies an **8-byte slot** + (`size((u32, u32)) == 16`), diverging from Hare's packed tuples (§12). +- `enum` — `enum i32 { A = 0, B, … }` (a named integer type with named + constants) or bare `enum { … }`. +- A named type / alias introduces a new spelling for an underlying type. + +TODO(spec): structs carry an internal `packed` property (no padding); +surface its syntax (an `@packed` attribute? a `struct` modifier?) once +confirmed against `cmd/wcc`. Likewise pin the `static let` qualifier +spelling and whether `static` applies outside function bodies. + +### 3.5 Untyped constants + +A literal without a suffix has an *untyped* type (`untyped int`, +`untyped float`, `untyped str`, `untyped rune`, `untyped bool`, +`untyped nil`). An untyped constant takes a concrete type from its use +context (assignment target, operand of a typed operation, fn argument). +This is the inference that lets `let x: u8 = 5;` work without a suffix. + +### 3.6 Pointers and nullability + +`*T` is a pointer to `T`. **A plain `*T` is non-null**; dereferencing it +is unconditional and free. A *nullable* pointer is written as the tagged +union `(*T | void)`; `nil` inhabits it. This is a one-flag fold, zero +extra cost for the non-null case, and it is the only optionality +mechanism — there is no separate `nullable` qualifier keyword (§12). +ww has no runtime null-check net, so a non-null `*T` carries the +no-null obligation the way C does. + +### 3.7 Tagged unions and the error tag + +``` +TaggedType = "(" Type { "|" Type } ")" . +``` + +A tagged union `(A | B | …)` holds one of its variants plus a runtime +tag. Variants are discriminated with `match` (§7) or tested with `is` +and narrowed with `as` (§8). A variant may carry the error mark `!T`, +which participates in the error model (§9): `(i64 | invalid | overflow)` +is a typical fallible result. + +### 3.8 Function and channel types + +`fn(P0, P1, …) R` is a function type. `chan T` is a channel of `T` +(**reserved**, §11). `never` is the bottom type of an expression that +does not return (e.g. a call to `abort`). + +--- + +## 4. Packages and imports + +``` +SourceFile = PackageClause { ImportDecl } { TopDecl } . +PackageClause = "package" ident ";" . +ImportDecl = "import" ident ";" . +``` + +- Every source file begins with a package clause. A directory of `.ww` + files sharing one package name compiles as a single module. +- `import foo;` makes module `foo`'s exported names available as + `foo.Name`. Self-import is rejected. +- An executable's entry module is `package main;` with a `fn main`. +- Only names marked `export` (§5) are visible across module boundaries. + +--- + +## 5. Declarations + +``` +TopDecl = [ "export" ] ( FnDecl | TypeDecl | LetDecl | ConstDecl + | DefDecl ) . +``` + +### 5.1 Functions + +``` +FnDecl = "fn" ident "(" [ Params ] ")" Result "=" Block . + | "fn" ident "(" [ Params ] ")" Result ";" . // bodiless: extern/FFI +Params = Param { "," Param } . +Param = ident ":" Type | ident ":" Type "..." | "..." . +Result = Type | "void" . +``` + +The `=` between signature and body is required. A bodiless fn (`;` +terminator) declares an external symbol (C FFI); the bare `...` C-style +variadic is legal only on a bodiless declaration (§6.3). + +### 5.2 `let` + +``` +LetDecl = "let" ident ":" Type [ "=" Expr ] ";" . +``` + +A mutable binding. Without an initialiser the storage is uninitialised. +A `static` qualifier (`static let …`) gives function-local storage +static lifetime. + +### 5.3 `const` + +`const` introduces an immutable binding of the same shape as `let`. + +TODO(spec): pin the exact `const` vs `def` boundary against `cmd/wcc` +(both keywords exist; this text takes `def` = compile-time constant +[Hare's `def`] and `const` = immutable runtime binding — confirm). + +### 5.4 `def` + +``` +DefDecl = "def" ident ":" Type "=" ConstExpr ";" . +``` + +A compile-time constant definition, typically at module level +(`def AF_INET: i32 = 2;`). Hare's `def`. + +### 5.5 `type` + +``` +TypeDecl = "type" ident "=" Type ";" . +``` + +### 5.6 Visibility + +`export` on a top-level declaration makes the name visible to importing +modules. Unmarked names are module-private. + +### 5.7 Attributes + +`@name` prefixes a declaration with an attribute. The defined attribute +is `@test`: it marks a function as a test (§10). + +--- + +## 6. Functions and calls + +### 6.1 Blocks and results + +A function body is a `Block` (§7). The body's value is its result; an +explicit `return Expr;` returns early. + +### 6.2 Calls + +``` +Call = Expr "(" [ Args ] ")" . +Args = Arg { "," Arg } [ "..." ] . +``` + +Arguments are passed by value (a `str`/`[]T`/`struct` copy is a shallow +header/field copy). Aggregates larger than a register use the +aggregate/`sret` ABI. + +### 6.3 Variadic + +Two distinct variadic forms: + +- **Hare-style** `args: T...` — the callee receives a `[]T`. A call + either gathers loose arguments (`fmt.println(1, "x")`) or forwards an + existing slice with the spread `xs...`. +- **C-style** bare `...` — only on a bodiless (extern) declaration, for + calling C variadic functions across the FFI. `f32` arguments in the + variadic tail are promoted to `f64` per the C ABI. + +--- + +## 7. Statements + +``` +Stmt = Block | LetDecl | ConstDecl | Assign | If | For | Switch + | Match | Return | Break | Continue | Defer | Yield | ExprStmt . +Block = "{" { Stmt } "}" . +``` + +### 7.1 Assignment + +``` +Assign = Expr AssignOp Expr ";" . +AssignOp = "=" | "+=" | "-=" | "*=" | "/=" | "%=" + | "&=" | "|=" | "^=" | "<<=" | ">>=" . +``` + +The compound forms apply to scalar, indexed, and pointer-field targets. + +### 7.2 `if` + +``` +If = "if" "(" Expr ")" Block [ "else" ( If | Block ) ] . +``` + +### 7.3 `for` + +``` +For = "for" "(" Expr ")" Block // condition + | "for" "(" LetDecl Expr ";" Expr ")" Block // 3-clause + | "for" "(" "let" ident ".." Expr ")" Block . // range +``` + +The 3-clause form is `for (let i: size = 0; i < n; i += 1) { … }`. The +range form iterates the elements of a slice or array +(`for (let x .. xs) { … }`). `break` and `continue` apply to the nearest +enclosing loop; `continue` runs the post-clause / advances the range. + +TODO(spec): does the range form bind the *element* or an index, and is +the bound name a copy or a view? Confirm against the checker's +`N_FORRANGE` lowering; the continue-post behaviour was a fixed miscompile +(project memory #138), so the desugaring is worth stating precisely. + +### 7.4 `switch` + +``` +Switch = "switch" "(" Expr ")" "{" { SwCase } "}" . +SwCase = "case" ExprList ":" { Stmt } | "case" ":" { Stmt } . +``` + +Value-matching on a scalar. A `case` may list comma-separated values; +`case:` is the default arm. Arms do not fall through. + +### 7.5 `match` + +``` +Match = "match" "(" Expr ")" "{" { MatchArm } "}" . +MatchArm = "case" [ "let" ident ":" ] Type "=>" ArmBody + | "case" "=>" ArmBody . +ArmBody = Stmt | Block . +``` + +Discriminates a tagged union (§3.7). `case let v: T =>` binds the +narrowed value to `v`; `case T =>` matches without binding; `case =>` is +the default. A `match` used as an expression produces a value via `yield` +(§7.7) from each arm. + +### 7.6 `return`, `break`, `continue`, `defer` + +`return [Expr];` `break;` `continue;`. `defer Stmt;` schedules a +statement to run when the enclosing block exits. + +### 7.7 `yield` + +`yield Expr;` produces the value of an enclosing block- or match- +expression. + +--- + +## 8. Expressions + +### 8.1 Operands + +Identifiers, qualified names (`mod.name`), literals (§2.7), struct/array/ +tuple literals (§8.5), and parenthesised expressions. + +### 8.2 Operators and precedence + +From loosest to tightest binding: + +``` +|| +&& +== != < <= > >= +| ^ +& +<< >> ++ - +* / % +unary: - ! ~ &(addr-of) *(deref) +postfix: call() index[] slice[:] field. ? ! +``` + +`&e` takes the address of an addressable operand; `*p` dereferences. + +TODO(spec): the precedence ladder is transcribed from the parser's +`bprec` (`parse.ww`); verify each tier against `cmd/wcc/parse.c` and +state associativity per tier. ww has no generics — no type parameters +anywhere in the grammar; call this out explicitly once confirmed. + +### 8.3 Casts, `as`, `is` + +``` +Cast = Expr ":" Type . // type ascription / conversion +As = Expr "as" Type . // tagged-union narrowing +Is = Expr "is" Type . // tagged-union variant test (bool) +``` + +`expr: T` converts/ascribes (`len(xs): size`, `t.line: i64`). An integer +cast does not silently truncate beyond the target width. `e as T` +narrows a tagged union to variant `T`; `e is T` tests membership. + +### 8.4 Index, slice, field, pseudo-fields + +`a[i]` indexes an array/slice/str (index operand must be an integer). +`a[lo:hi]` produces a sub-slice. `e.f` selects a struct field or a +qualified name. `.ptr`, `.len`, `.cap` read the header words of a `str` +or slice. + +### 8.5 Composite literals + +``` +StructLit = TypeName "{" [ FieldInit { "," FieldInit } ] "}" . +FieldInit = ident ":" Expr . +ArrayLit = "[" [ Expr { "," Expr } ] "]" . +TupleLit = "(" Expr "," Expr { "," Expr } ")" . +``` + +`S{}` is the zero value of `S`. + +### 8.6 Builtins + +`size(T)` (type size), `len(x)` (str/slice/array length), `alloc(v)!` +(heap-allocate a value, yield `*T`; §9), `append(s, x)` (grow a slice), +`delete(…)`, `abort(msg)` (terminate; type `never`). + +`alloc` takes a *value*: `alloc(T{…})!`, `alloc(T{})!`, or +`alloc(expr)!`. Bare `alloc(T)!` is invalid (it parses `T` as a value +reference) (§12). There is no `free`: ww has no GC and no manual +reclamation in compiler-side code — allocation lives until process exit +(§12). `lib/*` modules that genuinely own a heap buffer free it +explicitly via `os.free`. + +--- + +## 9. Errors + +ww has no exceptions. A fallible operation returns a tagged union whose +error variants carry the `!T` mark (§3.7). Two postfix operators consume +them: + +- `expr?` — if `expr` is an error variant, return it from the current + function (propagate); otherwise yield the success value. +- `expr!` — if `expr` is an error variant, abort; otherwise yield the + success value. + +Error variants are ordinary sentinel types in the union +(`strconv.invalid`, `strconv.overflow`, …), discriminated with `match`. +This mirrors Hare's error idiom, spelled with ww's `!` tag. + +--- + +## 10. Tests + +A function marked `@test` is a test, run by the test harness, not part of +the program. Two placements, following the Go `foo` / `foo_test` model: + +- **White-box**, in-package: a colocated file `package ;` with + `@test` functions, for testing unexported internals. +- **Black-box**, external: `package _test;` with `import ;`, + exercising only the public surface. + +```ww +@test fn adds() void = { + if (1 + 1 != 2) { abort("math broke"); }; +}; +``` + +--- + +## 11. Concurrency (reserved) + +The headline `+CSP` surface — `chan T`, `proc`/`spawn`, channel send/ +receive `<-`, and the `alt` select construct — is tokenised +(`chan`, `proc`, `<-`, `->`) but **not yet given semantics**. This +section is written when the feature lands; until then, use of these +tokens is rejected by the checker. + +**Meta-ruling (normative).** CSP is *the* sanctioned departure from the +Hare-fidelity rules (CLAUDE.md rules 5 and 9) — Hare has no channels, and +the `+CSP` mandate is the project's reason to add them. The concurrency +surface therefore cites the Newsqueak → Limbo → Go lineage, **not** Hare, +and that citation is correct by construction, not a rule-9 violation. The +design of record is `.ai/csp-design-input.md` (library-first: `spawn`, +unbuffered bidirectional `chan T` via `mkchan(T)`, `<-` send/recv, +`recv → (T | closed)` error idiom, `alt` as v2). See §12 entry 9. + +TODO(spec): fill in syntax + semantics (channel construction, send/recv +typing, `spawn` arg-copy ABI, `alt` arm grammar, the `closed` sentinel) +once the MVP lands and the USER meta-ruling is explicitly on record. + +--- + +## 12. Divergences from Hare (normative) + +ww follows Hare's semantics and `lib/` API shapes (CLAUDE.md rules 5, 9) +except for the deliberate, USER-blessed departures below. Each is +normative; the cited reference is the rationale of record. + +1. **`int`/`uint` are a machine word (8 bytes), not Hare's 32-bit + `int`.** All limit constants derive from `size(int)`; `size`/`uintptr` + are 8 bytes. *Ref: project memory "int is a machine word"; USER + 2026-05-26.* + +2. **Tuples use 8-byte slots, not Hare's packed layout.** + `size((u32, u32)) == 16`. Routed through the type table only. + *Ref: USER 2026-06-04 tuple slot ruling; parity task #60.* + +3. **Nullability is the union `(*T | void)`, not a `nullable` + qualifier.** Plain `*T` is non-null; the union fold is one flag, + zero-cost for the non-null path. *Ref: USER 2026-05-26 nullable-kept + ruling.* + +4. **No `_unsafe` suffix convention.** ww is unmanaged (no GC, no safe + baseline), so Hare's `_unsafe` axis does not apply. `bytes`→`str` is + the pure reinterpret `strings.frombytes` (Hare's `fromutf8_unsafe`); + UTF-8 validation is opt-in at the IO source via `utf8.validate`, not + wrapped per construction. *Ref: CLAUDE.md rule 9; ref/hare/strings/ + utf8.ha.* + +5. **The compiler frontend is one package `lib/ww/syntax`** (tok + lex + + ast + sym + typ + parse), a Go-over-Hare consolidation of Hare's + `{ast, lex, parse}` split. Its single consumer is the `wcc` backend. + The internal data shapes still mirror `ref/hare/hare`. *Ref: CLAUDE.md + rule 6; USER #74.* + +6. **Tests use the Go `foo` / `foo_test` model** (§10): external + black-box `package _test;` plus in-package white-box `@test` + files. ww hard-rejects self-import. *Ref: CLAUDE.md rule 9; task #16.* + +7. **No arena / no `free` in compiler-side code.** Per-node + `alloc(value)!`; allocation is reclaimed at process exit. `alloc` + takes a *value* (`alloc(T{…})!`), never a bare type. *Ref: project + memory "drop amalloc", "Phase 0 closed", "alloc syntax form".* + +8. **The checked AST carries a per-node `type_` stamp.** Hare's parse + AST is untyped and harec keeps a separate checked AST in C; ww stamps + `node.type_` in place as a checker invariant. *Ref: project memory + "Hare AST has no per-expr result".* + +9. **CSP** (§11) is the sanctioned departure from rule-5/9 Hare fidelity + — Hare has no channels. The surface cites Newsqueak/Limbo/Go, not + Hare. *Ref: `.ai/csp-design-input.md`; pending explicit USER + meta-ruling.* + +10. **The two compiler stages emit byte-identical asm** for the same + input (CLAUDE.md rule 10); when inference power differs, the richer + stage is aligned *down* to the leaner one. This is a project + invariant, listed here because it is the contract this spec's "the + text is the bug if stages disagree with it" rule rests on. + +--- + +## Appendix A. Anti-rot + +This spec is hand-maintained. To keep it from drifting against the +parser, every fenced ` ```ww ` example is intended to be extracted and +compiled through `w6c`, asserting it parses (or rejects, for +negative examples), wired as a `make spec-check` dependency of +`make test`. The EBNF +productions are not auto-verified; the compiling examples are the drift +alarm. *(Check not yet implemented; filed as a follow-up.)*