Implement the lexer, AST and parser for the SQLite dialect

.github/workflows/difftest.yml

@@ -0,0 +1,44 @@
+# Differential testing against a real SQLite build.
+#
+# This is separate from CI because it downloads the pinned SQLite release
+# and compiles it, which the ordinary test run must not depend on. It is the
+# only thing that exercises error fidelity at scale: the vendored corpus is
+# almost all valid SQL, so mutating it -- and mutating the ~36k inputs of
+# SQLite's own fuzzdata databases, which are the opposite -- is how
+# divergent messages and offsets get found.
+name: difftest
+
+on:
+  schedule:
+    - cron: "17 4 * * 1" # Mondays, early
+  workflow_dispatch:
+    inputs:
+      seed:
+        description: "PRNG seed (default: the run id, so each run is new ground)"
+        default: ""
+      per:
+        description: "mutations per input"
+        default: "60"
+
+jobs:
+  difftest:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version-file: go.mod
+      - name: Cache the pinned SQLite artifacts
+        uses: actions/cache@v4
+        with:
+          path: .sqlite
+          key: sqlite-${{ hashFiles('internal/sqlitesrc/sqlitesrc.go', 'internal/sqlitesrc/oracle/oracle.c') }}
+      # The seed defaults to the run id rather than to a constant: a weekly
+      # job with a fixed seed re-checks the same mutations forever. Roughly
+      # 17M checks, a few minutes on a four-core runner -- the oracle runs
+      # as a pool of batch-mode processes, so the cost is nearly all meyer.
+      - run: >
+          go run ./cmd/difftest
+          -per ${{ github.event.inputs.per || '60' }}
+          -depth 3
+          -seed ${{ github.event.inputs.seed || github.run_id }}

.gitignore

@@ -1,3 +1,4 @@
 # Local cache used by cmd/regenerate-parse: downloaded SQLite release
 # artifacts and the compiled oracle binary.
 .sqlite/
+/oq

CLAUDE.md

@@ -9,36 +9,89 @@ https://sqlite.org/lang.html.
 ## Rules
 
 - **Zero dependencies.** `go.mod` must never gain a `require` line.
+- The keyword table and the `%fallback` set are transcriptions, and tests in
+  `token` rebuild both from the vendored upstream sources to keep them
+  honest. Advancing the pin surfaces a keyword change as a failure.
 - **No parser generators.** Everything is hand-written recursive descent.
 - **Never edit `parser/testdata/*.test` by hand.** Corpus files are produced
   by `cmd/regenerate-parse` from SQLite's own test suite plus a real SQLite
   build (the oracle). `*.metadata.json` sidecars are updated by tooling
-  (`go test ./parser -check-parse`), not by hand.
+  (`go test ./parser -check-parse`), not by hand. The hand-written snapshot
+  inputs under `parser/testdata/ast/` are the exception — see below.
 - Every nontrivial parse function carries a comment naming the `parse.y`
-  rule(s) it implements.
+  rule(s) it implements. `TestEveryRuleIsNamed` enforces this from the other
+  side: no nonterminal of the vendored grammar may go unmentioned.
 - Error messages must match SQLite's parser byte-for-byte
   (`near "X": syntax error`, `unrecognized token: "X"`, `incomplete input`).
 
 ## The corpus
 
 Each `parser/testdata/<name>.test` holds cases extracted from SQLite's
-`test/<name>.test` TCL scripts. For every case, the raw oracle results (one
-line per statement: prepared OK, or the exact error message and offset) are
-stored; the harness derives the expectation from them:
+`test/<name>.test` TCL script; there is one for every script in the pinned
+source tree that yields at least one literal-SQL block. For every case, the
+raw oracle results (one line per statement: prepared OK, or the exact error
+message, offset and parse tail) are stored; the harness derives the
+expectation from them:
 
 - If any statement failed with a **syntax-family** message (`near "…":
   syntax error`, `unrecognized token: "…"`, `incomplete input`), meyer must
   reject the case with that first message.
 - Otherwise meyer must accept the whole case. This includes statements that
   failed **semantically** in SQLite (`no such table: …`) — those parsed
   successfully; meyer does no semantic analysis.
+- …except in text SQLite's parser never reached. A grammar action can fail
+  in the middle of a statement — `sqlite3BeginTrigger` raising `no such
+  table` at the `trigger_decl` reduce, before the trigger body is looked at
+  — and `sqlite3RunParser` then abandons the rest of the statement. The
+  oracle records `pzTail` on every failing statement, so the harness knows
+  which byte ranges are unverified and lets meyer fail inside them.
 
 Known looseness: messages produced by grammar *actions* (e.g. `ORDER BY
 clause should come after UNION not before`) are currently classified as
 semantic, so meyer is permitted to accept such statements. The pattern list
 lives in `internal/testfile` (`syntaxFamily`) and can be extended without
 regenerating the corpus, because the corpus stores raw oracle output.
 
+## Tree shape
+
+Accept/reject conformance cannot see a dropped clause or a mis-associated
+operator, so two further checks stand in for the parse-tree goldens SQLite
+cannot produce:
+
+- **Round trip** (`TestRoundTrip`): every corpus case that parses is
+  rendered back to SQL with `ast.Statements`, re-parsed, and the two trees
+  compared structurally with `internal/dump`. Spans and `Raw` fields are
+  excluded — the renderer promises re-parseability and nothing else.
+- **Snapshots** (`TestASTSnapshots`): `parser/testdata/ast/*.sql` are
+  hand-written and meant to be edited; their `.tree` goldens are rewritten
+  with `go test ./parser -update` and reviewed in the diff.
+
+## Differential testing
+
+The corpus is whatever SQLite's test suite happens to contain, which is
+overwhelmingly valid SQL: fewer than three hundred of its 20,971 cases are
+rejections, so error fidelity is barely exercised by it. `cmd/difftest`
+covers that gap by mutating corpus SQL — truncate, delete, duplicate,
+replace or insert one token — and checking that meyer and a live SQLite
+build still agree on the verdict, the message and the offset.
+
+```sh
+go run ./cmd/difftest                        # sweep the whole corpus
+go run ./cmd/difftest -files select1,expr    # a few files
+go run ./cmd/difftest -per 60 -seed 7        # dig harder, reproducibly
+```
+
+It needs the oracle, so it is not part of `go test ./...`; a separate
+workflow (`.github/workflows/difftest.yml`) runs it weekly and on demand.
+What it finds belongs in `parser/errors_test.go`, whose expectations are
+taken from the oracle rather than written by hand.
+
+Two situations it deliberately skips, because the harness cannot compare
+them rather than because meyer might be wrong: a `;` inside parentheses,
+where `sqlite3_complete` splits statements differently from a real parse,
+and a semantic failure on a statement's last token, which leaves `pzTail` at
+the end of the input with nothing to mark the parse as abandoned.
+
 ## The loop
 
 ```sh
@@ -58,7 +111,7 @@ the fix is in `cmd/regenerate-parse` (or the classification in
 ## Regenerating the corpus
 
 ```sh
-go run ./cmd/regenerate-parse                  # the full starter set
+go run ./cmd/regenerate-parse                  # every script in the pinned tree
 go run ./cmd/regenerate-parse -files select1   # one file
 ```
 

PLAN.md

@@ -173,12 +173,18 @@ domain):
 - **Negative subset**: the ~160 `catchsql` assertions whose expected message
   contains `syntax error` / `unrecognized token` — precise must-reject cases
   with expected messages.
-- **Robustness corpus**: `test/fuzzdata{1,2,4,5,6,8}.db` contain ~36k SQL
-  text fuzz inputs (`xsql` tables) — no oracle, invariant is "terminate
-  without panic". Used as fuzz seeds, not vendored wholesale.
+- **Robustness corpus**: `test/fuzzdata{1..8}.db` contain ~36k SQL text fuzz
+  inputs (`xsql` tables). Used as seeds, not vendored wholesale. *(Read out
+  on demand by `sqlitesrc.FuzzSeeds` and fed to `cmd/difftest`, which does
+  have an oracle for them: the invariant is the full differential one, not
+  merely "terminate without panic".)*
 - **sqllogictest** (~millions of statements, license: "no attribution
   required") — bulk smoke corpus; low grammar diversity. Optional, behind an
-  env-var-gated test, not vendored.
+  env-var-gated test, not vendored. *(Not built. Everything else the tests
+  download is pinned by SHA-256; sqllogictest is distributed as a Fossil
+  checkout with no stable release artifact to pin, and what it would add is
+  volume of ordinary SELECTs — the axis `cmd/difftest` is already furthest
+  along. Worth revisiting only if a pinnable mirror appears.)*
 
 Tooling (`cmd/regenerate`):
 
@@ -342,20 +348,32 @@ meyer/
 │   ├── parse_trigger.go      # trigger decl + restricted body
 │   ├── parse_misc.go         # txn, pragma, attach, vacuum, analyze, …
 │   ├── parser_test.go        # corpus harness (+ -check-parse)
-│   ├── fuzz_test.go          # never-panic, fuzzdata seeds
+│   ├── fuzz_test.go          # never-panic, round-trip under arbitrary bytes
+│   ├── roundtrip_test.go     # render → re-parse over the whole corpus
+│   ├── span_test.go          # byte-offset invariants sqlc slices with
+│   ├── snapshot_test.go      # AST goldens for testdata/ast/*.sql
+│   ├── errors_test.go        # message/offset fidelity
+│   ├── grammar_test.go       # every parse.y nonterminal is named somewhere
 │   ├── bench_test.go
 │   └── testdata/
 │       ├── README.md         # provenance + pinned SQLite version/hash
+│       ├── ast/              # hand-written tour of the node set + goldens
 │       ├── *.test            # consolidated corpus (== / -- format)
 │       └── *.metadata.json   # todo/skip sidecars
 ├── internal/
 │   ├── dump/                 # AST snapshot renderer
+│   ├── roundtrip/            # the round-trip property, stated once
+│   ├── sqlitesrc/            # pinned release: download, oracle, runner
+│   │   └── oracle/oracle.c   # the C oracle (prepare-only classifier)
+│   ├── tclextract/           # TCL brace scanner for the test scripts
 │   ├── testfile/             # corpus file format reader/writer
-│   └── reference/parse.y     # vendored grammar (documentation only)
+│   └── reference/            # vendored parse.y, tokenize.c,
+│                             #   mkkeywordhash.c (checked by tests)
 └── cmd/
     ├── next-test/            # pick next todo case
     ├── debug-parse/          # parse argv SQL, dump tree or error
-    └── regenerate/           # extract TCL corpus + run SQLite oracle
+    ├── difftest/             # mutation differential testing vs the oracle
+    └── regenerate-parse/     # extract TCL corpus + run SQLite oracle
 ```
 
 ## Milestones
@@ -364,10 +382,11 @@ meyer/
    fallback tables), lexer with its own unit tests transcribed from
    `tokenize.test`, AST skeleton, `parser.Parse` returning
    not-implemented errors, corpus format + harness with everything `todo`.
-2. **Corpus** — `cmd/regenerate` extraction + oracle; vendor the starter set
-   (`parser1`, `tokenize`, `keyword1`, `select*`, `expr*`, `e_*`, `with*`,
-   `window*`, `trigger1`, `upsert*`, `returning1`, `altertab*`, `alter*`,
-   plus the syntax-error negative subset).
+2. **Corpus** — `cmd/regenerate` extraction + oracle. *(Done, but wider than
+   planned: rather than vendoring a hand-picked starter set, the tool takes
+   every `test/*.test` script in the pinned tree. That costs 4.4 MB and 943
+   files, and yields 20,971 cases instead of 4,685, with no curation to redo
+   when the pin advances.)*
 3. **Expressions + SELECT** — largest single chunk: precedence ladder,
    subqueries, CTEs, compound selects, VALUES, window functions, FROM/joins.
 4. **DML** — INSERT (+upsert/returning/default values), UPDATE (+from),
@@ -380,7 +399,13 @@ meyer/
    EXPLAIN [QUERY PLAN].
 7. **Hardening** — error-message/offset fidelity pass over the negative
    corpus, fuzzing with fuzzdata seeds, benchmarks, race-clean parallel
-   corpus run, optional sqllogictest smoke gate.
+   corpus run, optional sqllogictest smoke gate. *(Done except the
+   sqllogictest gate. The fuzzdata seeds went to `cmd/difftest` rather than
+   to `FuzzParse`: an oracle that says whether SQLite accepts an input is a
+   far stronger check on them than "does not panic", and mutating them
+   turns 36k inputs into over a million. The corpus harness carries the
+   round trip, the spans and the offsets, so those are checked on every
+   run rather than in a pass.)*
 8. **sqlc integration** (in the sqlc repo, separate effort) — new
    `internal/engine/sqlite/parse.go` calling meyer (mirroring the
    zetajones/googlesql pattern), rewritten `convert.go` (meyer AST →

README.md

@@ -10,23 +10,70 @@ meyer is being built to replace the ANTLR-generated SQLite parser in
 [sqlc](https://github.com/sqlc-dev/sqlc). See [PLAN.md](PLAN.md) for the
 architecture and [CLAUDE.md](CLAUDE.md) for the development workflow.
 
+## Usage
+
+```go
+stmts, err := parser.Parse(ctx, strings.NewReader("SELECT id FROM users WHERE id = ?"))
+```
+
+`ParseString` and `ParseStatement` take a string; `ParseExpr` parses a single
+expression. A rejected input returns a `*parser.Error` carrying SQLite's
+exact message and the byte offset of the fault.
+
+Every node embeds `ast.Span`, so `Pos()` and `End()` give byte offsets into
+the original input — sqlc slices the source with them to find `-- name:`
+comments and to report errors, so they are load-bearing rather than
+diagnostic.
+
+To see what the parser did with something:
+
+```sh
+go run ./cmd/debug-parse 'SELECT sum(x) OVER w FROM t'   # the tree
+go run ./cmd/debug-parse -tokens 'SELECT 1'              # the token stream
+go run ./cmd/debug-parse -render -f query.sql            # re-rendered SQL
+```
+
 ## Status
 
-Under construction: the test corpus and tooling exist; the parser itself is
-being implemented against them.
+The parser covers the whole grammar of the pinned SQLite release and passes
+the full corpus: **20,971 of 20,971 cases**, extracted from every test script
+in SQLite's own test suite.
+
+Still to come, from [PLAN.md](PLAN.md): the sqlc integration itself, and the
+optional sqllogictest smoke gate.
 
 ## Conformance
 
-meyer is tested against SQL extracted from SQLite's own test suite, with
-expectations produced by a pinned build of SQLite itself: meyer must accept
-exactly the statements SQLite's parser accepts, and reproduce SQLite's exact
-syntax-error messages. See [parser/testdata/README.md](parser/testdata/README.md)
-for provenance and the pinned version.
+There is no upstream parse-tree oracle — SQLite cannot dump its parse tree —
+so conformance is defined three ways, in decreasing order of authority:
+
+1. **Accept/reject against SQLite itself.** SQL is extracted from SQLite's
+   test suite and run through a pinned SQLite build. meyer must accept
+   exactly what SQLite's parser accepts, and on rejection produce the
+   identical message and byte offset. See
+   [parser/testdata/README.md](parser/testdata/README.md) for provenance and
+   the pinned version.
+2. **A round-trip property.** Every accepting case is rendered back to SQL,
+   re-parsed, and the two trees compared. This catches dropped clauses and
+   mis-associated operators, which accept/reject cannot see.
+3. **AST snapshots.** A hand-written tour of the node set under
+   `parser/testdata/ast`, reviewed in diffs. These are meyer's own goldens
+   and never outrank the oracle.
+
+Plus two searches for cases the corpus does not contain: a fuzz target
+asserting that arbitrary bytes terminate without panicking, that a rejection
+is always a `*parser.Error`, and that anything accepted survives the round
+trip; and `cmd/difftest`, which checks meyer against a live SQLite build,
+message and byte offset included, over corpus SQL and the ~36k inputs of
+SQLite's own `fuzzdata` databases — each one directly and then mutated a
+token at a time.
 
 ## Acknowledgments
 
-SQLite and its test suite are public domain — https://sqlite.org/copyright.html.
-This project's grammar and test corpus derive from them.
+SQLite and its test suite are public domain —
+https://sqlite.org/copyright.html. meyer's grammar and test corpus derive
+from them; `internal/reference/` keeps the upstream `parse.y` and
+`tokenize.c` alongside the port, as documentation.
 
 ## License
 

ast/ast.go

@@ -1,10 +1,17 @@
 // Package ast declares the syntax tree for the SQLite SQL dialect.
 //
-// This is the milestone-1 skeleton: the interfaces exist so the parser API
-// and test harness are stable, and node types are added as the parser is
-// implemented (see PLAN.md for the full intended node set).
+// Node names follow SQLite rather than PostgreSQL or sqlc: the tree is a
+// direct rendering of src/parse.y, and every node carries a comment naming
+// the grammar rule it comes from. Mapping onto another representation is a
+// consumer's job.
+//
+// Every node embeds Span and so reports byte offsets into the original
+// input; sqlc slices the source with those offsets, so they are load-bearing
+// rather than diagnostic (see PLAN.md).
 package ast
 
+import "reflect"
+
 // Node is implemented by every syntax tree node. Positions are byte offsets
 // into the original input.
 type Node interface {
@@ -13,9 +20,13 @@ type Node interface {
 	Children() []Node
 }
 
-// Stmt is implemented by all statement nodes.
+// Stmt is implemented by all statement nodes. SetSpan is part of the
+// interface because the parser widens a statement's span once it has seen
+// the terminating semicolon, and a node that could not be widened would get
+// a quietly wrong span rather than a compile error.
 type Stmt interface {
 	Node
+	SetSpan(Span)
 	stmtNode()
 }
 
@@ -33,3 +44,33 @@ type Span struct {
 
 func (s Span) Pos() int { return s.Start }
 func (s Span) End() int { return s.Stop }
+
+// SetSpan replaces the node's extent. The parser uses it to widen a
+// statement's span once its terminating semicolon has been consumed.
+func (s *Span) SetSpan(sp Span) { *s = sp }
+
+// Walk calls fn for n and, unless fn returns false, for its descendants in
+// source order.
+func Walk(n Node, fn func(Node) bool) {
+	if isNil(n) || !fn(n) {
+		return
+	}
+	for _, c := range n.Children() {
+		Walk(c, fn)
+	}
+}
+
+// isNil reports whether n is either a nil interface or a typed nil pointer.
+// Children() slices are assembled from optional fields, so typed nils are
+// common and must not leak into a traversal.
+func isNil(n Node) bool {
+	if n == nil {
+		return true
+	}
+	v := reflect.ValueOf(n)
+	switch v.Kind() {
+	case reflect.Pointer, reflect.Interface, reflect.Slice, reflect.Map:
+		return v.IsNil()
+	}
+	return false
+}