A pure Go parser for the SQLite SQL dialect. Zero dependencies, hand-written recursive descent — no parser generators.
(a Meyer lemon, for SQLite's Lemon parser generator)
meyer is being built to replace the ANTLR-generated SQLite parser in sqlc. See PLAN.md for the architecture and CLAUDE.md for the development workflow.
go get github.com/sqlc-dev/meyerpackage main
import (
"errors"
"fmt"
"github.com/sqlc-dev/meyer/ast"
"github.com/sqlc-dev/meyer/parser"
)
func main() {
const src = `SELECT u.name, count(o.id) FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.created_at > :since`
stmts, err := parser.ParseString(src)
if err != nil {
var perr *parser.Error
if errors.As(err, &perr) {
fmt.Printf("%s at byte %d\n", perr.Message, perr.Offset)
}
return
}
for _, stmt := range stmts {
ast.Walk(stmt, func(n ast.Node) bool {
switch n := n.(type) {
case *ast.TableRef:
if n.Name != nil {
fmt.Println("table:", n.Name.Name.Name)
}
case *ast.BindParam:
fmt.Printf("param: %s at %d:%d\n", n.Raw, n.Pos(), n.End())
}
return true
})
}
}table: users
table: orders
param: :since at 100:106
ParseString and ParseStatement take a string, Parse takes an
io.Reader, and ParseExpr parses a single expression. A rejected input
returns a *parser.Error carrying SQLite's exact message and the byte offset
of the fault; its Error() renders as line:column: message.
_, err := parser.ParseString("SELECT FROM t")
fmt.Println(err) // 1:8: near "FROM": syntax errorEvery node embeds ast.Span, so Pos() and End() give byte offsets into
the original input — sqlc slices the source with them to find -- name:
comments and to report errors, so they are load-bearing rather than
diagnostic. ast.String and ast.Statements render a tree back to SQL,
which is a re-parseable rendering rather than a formatter.
The full API is on
pkg.go.dev: parser for the
entry points, ast for
the node set, and lexer
and token if you want
the token stream on its own.
To see what the parser did with something:
go run ./cmd/debug-parse 'SELECT sum(x) OVER w FROM t' # the tree
go run ./cmd/debug-parse -tokens 'SELECT 1' # the token stream
go run ./cmd/debug-parse -render -f query.sql # re-rendered SQLThe parser covers the whole grammar of the pinned SQLite release and passes the full corpus: 21,326 of 21,326 cases, extracted from every test script in SQLite's own test suite.
Still to come, from PLAN.md: the sqlc integration itself, and the optional sqllogictest smoke gate.
There is no upstream parse-tree oracle — SQLite cannot dump its parse tree — so conformance is defined three ways, in decreasing order of authority:
- Accept/reject against SQLite itself. SQL is extracted from SQLite's test suite and run through a pinned SQLite build. meyer must accept exactly what SQLite's parser accepts, and on rejection produce the identical message and byte offset. See parser/testdata/README.md for provenance and the pinned version.
- A round-trip property. Every accepting case is rendered back to SQL, re-parsed, and the two trees compared. This catches dropped clauses and mis-associated operators, which accept/reject cannot see.
- AST snapshots. A hand-written tour of the node set under
parser/testdata/ast, reviewed in diffs. These are meyer's own goldens and never outrank the oracle.
Plus two searches for cases the corpus does not contain: a fuzz target
asserting that arbitrary bytes terminate without panicking, that a rejection
is always a *parser.Error, and that anything accepted survives the round
trip; and cmd/difftest, which checks meyer against a live SQLite build,
message and byte offset included, over corpus SQL and the ~36k inputs of
SQLite's own fuzzdata databases — each one directly and then mutated a
token at a time.
SQLite and its test suite are public domain —
https://sqlite.org/copyright.html. meyer's grammar and test corpus derive
from them; internal/reference/ keeps the upstream parse.y and
tokenize.c alongside the port, as documentation.
MIT