Implement the lexer, AST and parser for the SQLite dialect

[kyleconroy]

Runs "the loop" from CLAUDE.md to completion, then keeps going. The parser covers the whole grammar of the pinned SQLite release, and the corpus it is checked against grew from the planned starter set to SQLite's entire test suite.

Corpus: 21,326 of 21,326 cases passing, extracted from all 945 test scripts that contain literal SQL.

The parser

• token/ — the keyword table and %fallback ID set transcribed from tool/mkkeywordhash.c and parse.y, resolved for the pinned build's feature set (no SQLITE_ENABLE_ORDERED_SET_AGGREGATES, so WITHIN is not a keyword), plus the id/ids/idj/nm token classes.
• lexer/ — a port of sqlite3GetToken(): four quoting styles, blob literals with even-length/hex validation, the four bind-parameter spellings including TCL-style $foo::bar(baz), _ digit separators, and the rule that 1abc is one illegal token rather than two good ones. Plus the WINDOW/OVER/FILTER resolution sqlite3RunParser does around the tokenizer. Illegal tokens stay in the stream and are reported when the parser reaches one, so tokenizer and syntax errors interleave in SQLite's order.
• ast/ — nodes for every statement and expression form, each naming the parse.y rule it comes from, all embedding Span.
• parser/ — recursive descent over the whole grammar. Expression precedence is a climbing loop over the eleven levels of the %left/%right block. One habit recurs and is stated once in the package doc: SQLite's LALR parser shifts a token as soon as some rule can begin with it and only then finds the rule does not continue, so where a keyword can introduce only one thing, meyer consumes it before checking what follows. Peeking would put the error on the keyword; SQLite puts it on what came next.

Conformance

Accept/reject against SQLite is the authority, with the exact message and byte offset. Three things sit under it, because accept/reject cannot see a dropped clause or a mis-associated operator:

• A round-trip property (internal/roundtrip) — every accepting case is rendered back to SQL, re-parsed, and the trees compared structurally; rendering must also be idempotent. Used by the corpus test, the fuzz target and difftest.
• AST snapshots — a hand-written tour of the node set under parser/testdata/ast, reviewed in diffs.
• Span invariants — a span is a valid range, a child lies within its parent, statements are disjoint and in order. sqlc slices the source with these, and nothing else checks them.

And two searches for what the corpus does not contain:

• FuzzParse — arbitrary bytes terminate without panicking, a rejection is a *parser.Error with an in-range offset, and anything accepted survives the round trip. 16M+ executions clean.
• cmd/difftest — checks meyer against a live SQLite build on the verdict, the message and the offset. Its inputs are the corpus plus the ~36k strings in SQLite's fuzzdata*.db databases, which are inputs that have broken a SQLite parser at some point; each is checked directly and then mutated a token at a time (truncate, delete, duplicate, replace, insert; chained for higher orders). Over 100M mutations run clean. A weekly workflow runs ~17M of them against a fresh seed, so it actually protects the repo.

Finally, three tests read the vendored upstream sources and rebuild what meyer transcribed from them — all 147 keywords with their TK_ codes, all 73 %fallback entries, and the requirement that every one of parse.y's 133 nonterminals is named in a parser comment. Advancing the pin surfaces a grammar change as a failure.

Bugs these found

Deep nesting killed the process. meyer recurses on the goroutine stack, where exhaustion is a fatal runtime error no caller can recover from. Nesting is now counted at the three points the grammar can cycle through and rejected past 10,000 with SQLite's own "Recursion limit" wording — SQLite gives up around 2,500 and the deepest corpus case reaches 32.

ParseString panicked out of the library on an input beginning with an illegal byte: its recover was installed after newParser, which reads the first token. The fuzzer found it on its fifth seed.

NOTNULL and NOT NULL are different trees — the first is a comparison-level operator, the second takes its precedence from NOT — and the AST recorded only a bool, so the renderer wrote NOT NULL for both. 26M mutations found it; nothing else could have.

A NUL byte ends a token, not just the token stream. SQLite's tokenizer walks a NUL-terminated buffer, so an unterminated ' or [ stops at an embedded NUL and is illegal there, and a -- comment stops at one rather than at the next newline. meyer read on to the end of the Go string. Routing every scan through the cursor helper that already stood in for the C terminator fixes all of them at once. The fuzzdata corpus found this on its first run — 1,813 disagreements from the one cause.

difftest found nine more, almost all the shift-then-decide shape above: FROM t NOT NOT INDEXED reported at the first NOT instead of the second, AS NOT ( at the NOT. Plus: a BETWEEN lower bound stops at a top-level AND but absorbs a weaker operator and everything binding tighter; CREATE TABLE t(a), STRICT parses because table_option_set recurses on an empty prefix; DEFAULT 'xyzzy'. c is an error at the dot; and a register reference errors from a grammar action, which does not stop SQLite at once. The span test found a named constraint whose span excluded its own CONSTRAINT clause.

Corpus and tooling

• The oracle records pzTail. SQLite can stop short of the end of a statement two ways, and either leaves text it never parsed. A grammar action can fail part-way through — sqlite3BeginTrigger raising no such table before the trigger body is looked at — and sqlite3RunParser abandons the rest; the corpus was demanding meyer accept three trigger bodies containing a deliberate syntax error SQLite never saw. And sqlite3_complete, which splits the script, is cruder than the tokenizer, so it can hand prepare two statements at once and only the first gets parsed. pzTail marks exactly which bytes went unverified in both cases.
• The oracle no longer decides for itself whether a chunk holds a statement. It had a small whitespace-and-comments scanner for that, which is a second, cruder tokenizer, and it got the quirks wrong — a lone vertical tab is CC_ILLEGAL rather than space, and a /* with nothing after the star is TK_SLASH TK_STAR rather than an unterminated comment. Both showed up as phantom disagreements. prepare answers the question exactly, by returning a NULL statement.
• One fuzz input contained PRAGMA hard_heap_limit=6874. Pragmas run at prepare time and that one is process-global, so it starved every later connection in the oracle process and the batch run died with cannot open :memory: on a flat heap. The heap limits are reset between records.
• Extraction covers every script in the pinned tree, and do_eqp_test as well as do_execsql_test/do_catchsql_test: 4.4 MB, no curation to redo when the pin advances.
• internal/sqlitesrc owns the pin and a batched oracle runner. Full regeneration takes ~2.5s rather than forking a process per script. Batch mode opens a fresh connection per script — reusing one leaks PRAGMA writable_schema across scripts, which silently changed five corpus files before it was caught.

Not done

The optional sqllogictest smoke gate, and milestone 8 (the sqlc integration, which lives in the sqlc repo). PLAN.md records why the sqllogictest gate is not being built: everything else the tests download is pinned by SHA-256, and sqllogictest has no stable release artifact to pin.