Profile the parser, and make the token stream cheap to build

[kyleconroy]

The benchmarks landed with the parser but nobody had read them. This is the first profiling pass.

	before	after
Lex/Join	4336 ns	2413 ns	−44%
Lex/Window	3840 ns	2972 ns	−23%
Lex/Simple	547 ns	403 ns	−26%
Render/Simple	269 ns	165 ns	−39%, 4 allocs → 1
Render/Join	1223 ns	789 ns	−36%, 7 allocs → 1
Render/Window	1340 ns	970 ns	−28%, 7 allocs → 1
Parse/Join	10609 ns	8373 ns	−21%
Parse/Window	9839 ns	8680 ns	−12%
Parse/Simple	1726 ns	1527 ns	−12%
Corpus (21,326 cases)	362 µs	313 µs	−14%, 377 KB → 279 KB

The two that mattered

token.Token carried its own spelling. Text was exactly src[Pos:End] — redundant with the offsets already in the struct, and the redundancy was not free. A string field puts a pointer in Token, which costs a write barrier per token to build the slice and leaves the whole slice as something the garbage collector has to walk afterwards. A token slice is the largest single allocation a parse makes.

Text is now a method taking the source. That takes Token from 40 bytes to 24 and out of the collector's sight entirely. A microbenchmark of the two shapes, building a 90-token slice:

with a string field (40 B/token)   1862 ns    4096 B
pointer-free       (24 B/token)     678 ns    2304 B     2.7x

Every caller has the source in hand — it just lexed it — so nothing had to be threaded anywhere new.

Keyword lookup was a Go map. Hashing the spelling of every identifier was a fifth of the lexer: mapaccess2_faststr plus aeshashbody. It is now indexed on length and first letter, which are free to compute and between them very nearly a perfect hash — 147 keywords into 93 groups, the largest of four, 1.58 candidates on average. Each candidate is compared byte by byte, upper-casing as it goes, so nothing is allocated or copied and there is no buffer.

SQLite's own keywordCode() hashes the first byte, the last byte and the length, but mixes them with a remainder mod a prime. Under power-of-two masking that hash degrades badly (73 probes worst case at 256 slots), and a parser reading a keyword every few bytes should not be doing a division.

Three smaller ones

• The renderer sized its strings.Builder from nothing and grew it four times for a short statement. A node's span says how long the output will be within a few bytes, since rendering rewrites the whitespace and nothing else.
• The token slice's size estimate was len(src)/4 by guess. Measured over the corpus, SQL runs to 3.7 bytes per token, and a divisor of three fits 93% of cases in one allocation where four fits 80%. Checked against the Corpus benchmark rather than the four handpicked queries, which flip depending on which side of the threshold they happen to land.
• Resolving WINDOW, OVER and FILTER needs a second pass over every token, and almost no SQL contains any of the three. Noting whether one turned up costs a comparison per token instead.

What did not work

Slab-allocating the AST nodes. parseQualifiedRef and identOf are 59% of the objects a SELECT allocates, so batching them looked obvious — it took Parse/Join from 109 allocations to 60 and did not move the clock at all, while raising memory 20%. Go's small-object allocator is already cheap; what costs is pointer-bearing bytes, and a slab adds them. That result is what pointed at the Token change, so it earned its keep, but it is not in this branch.

Interning the AST's strings with unique is worse still, for the same underlying reason: those strings are views into the input and never allocate, so there is nothing to deduplicate, and unique.Make costs ~54 ns a call — more than building the node it would go in.

substrings into src        1971 ns    2528 B
unique.Handle[string]      4415 ns    1968 B     2.3x slower

Behaviour

Unchanged, and checked rather than assumed:

• all 21,326 corpus cases, plus the round trip, span and snapshot properties
• cmd/difftest: 3,085,034 mutations over the corpus and SQLite's fuzzdata seeds, 0 disagreements
• FuzzParse: 7,022,062 executions, no failures

TestLookup is new. It walks every keyword in three casings and the shapes the index has to reject before it can subscript anything — too short, too long, first byte not a letter. The corpus reaches Lookup constantly but only through whatever spellings SQLite's test suite happens to use, so a group pointing at the wrong run of the table could have gone unnoticed for any keyword that suite never writes.

One API break

token.Token.Text is a method now, not a field. That is what bought the 2.7x. Cheap while nothing outside this repo consumes the token stream; worth doing before something does.

---

Generated by Claude Code