Looking for a SQLite statement parser

[phi...@percolate.com]

Hello,

I am the creator of rqlite (https://github.com/rqlite/rqlite), a distributed relational database built on SQLite.

I'm looking for a lexer and parser for SQL -- specifically SQLite, to improve the ease-of-use of rqlite. I've thought about looking into using the C code exposed by the SQLite source itself (which is what https://github.com/mattn/go-sqlite3 does more generally). But someone might know of a lexer and parser written in Go. I know how to build it myself, but would rather avoid the effort if possible.

What I am really trying to do is to determine, for the end-user, whether the Exec() (https://golang.org/pkg/database/sql/#DB.Exec)  or Query() (https://golang.org/pkg/database/sql/#DB.Query) method should be called on the standard Go SQL interface, given a user-supplied SQL statement. Right now the user must know what type of statement she is executing, and send the statements to the right endpoint. I'd really like rqlite to just do the right thing, and determine this automatically. Hence my idea to parse the SQLite command, create an AST, and work it out.

Or perhaps there is another way of thinking about this?

Thanks,

Philip

[Vasiliy Tolstov]

cznic/ql have own lexer for SQL and it syntax very similar.

31 Янв 2017 г. 16:33 пользователь <phi...@percolate.com> написал:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philip O'Toole]

Great -- thanks for the reference, I'll take a look.

Philip

--

Philip O'Toole
Director, Engineering
Get inspired. See what's new on the Percolate blog: blog.percolate.com

[Jacob Shandling]

> parse the SQLite command, create an AST

Hello Mr. O'Toole,

This is precisely the task I am looking to carry out, followed by walking the AST. Did you ever end up finding a good Go parser for SQLite, or building one yourself? I see your rqlite/sql package, but am unclear as to whether it has this functionality.

Thank you,

Jacob Shandling 

On Tuesday, January 31, 2017 at 10:33:38 PM UTC-8 Philip O'Toole wrote:

Great -- thanks for the reference, I'll take a look.

Philip

On Tue, Jan 31, 2017 at 10:02 AM, Vasiliy Tolstov <v.to...@selfip.ru> wrote:

cznic/ql have own lexer for SQL and it syntax very similar.

31 Янв 2017 г. 16:33 пользователь <phi...@percolate.com> написал:

Hello,

I am the creator of rqlite (https://github.com/rqlite/rqlite), a distributed relational database built on SQLite.

I'm looking for a lexer and parser for SQL -- specifically SQLite, to improve the ease-of-use of rqlite. I've thought about looking into using the C code exposed by the SQLite source itself (which is what https://github.com/mattn/go-sqlite3 does more generally). But someone might know of a lexer and parser written in Go. I know how to build it myself, but would rather avoid the effort if possible.

What I am really trying to do is to determine, for the end-user, whether the Exec() (https://golang.org/pkg/database/sql/#DB.Exec)  or Query() (https://golang.org/pkg/database/sql/#DB.Query) method should be called on the standard Go SQL interface, given a user-supplied SQL statement. Right now the user must know what type of statement she is executing, and send the statements to the right endpoint. I'd really like rqlite to just do the right thing, and determine this automatically. Hence my idea to parse the SQLite command, create an AST, and work it out.

Or perhaps there is another way of thinking about this?

Thanks,

Philip

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Julian Peterson]

Hi,

For lexer/parser generation, I've used antlr with some success. https://www.antlr.org/

The go code generation is a little rough atm, but not a show stopper.

Also, they have grammars for many languages, so they might be useful even on their own: https://github.com/antlr/grammars-v4 (sqlite is in there)

Cheers,

Julian

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/4b82f10e-1790-47f8-9506-9bd996f497bfn%40googlegroups.com.

[Jim Idle]

I revamped the Go code generation not so long ago - performance is sound with reasonably well written grammars, and very good with good grammars. Unfortunately the sample grammars for SQL are poor and will give massive performance problems with all targets (Java, C++, etc). I decided not to spend too much time on optimizing bad grammar specifications. The Go target is used extensively within Google.

Writing grammars looks easy because ANTLR will make a parser out of almost anything - but when it is written with huge ambiguities, it will explore every possible way that an input could be parsed and the performance will suffer. Unfortunately, SQL is the type of input where it is very easy to write a very ambiguous grammar, and so people have done so. I do not recommend using them. The MySQL grammar by Mike Lischke is performant - the MySQL one in grammars-v4 should not be used.

I have this available though:

It should be a reasonable task to cut down to SQLlite from the full TSQL spec.

That said, I have not tried the sample grammar for sqlite so I do not know what state it is in. You might try it and see what the performance is like before changing an existing grammar... [EDIT] - I see that Mike has worked on the sqlite grammar, so you may well have a good starting point there,

Jim

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CALGF0hWTGU8Pa%2BXOPH4XzmGw-c4SBPc2wX6JXVXTUO%3Df5Gukiw%40mail.gmail.com.

[Jacob Shandling]

Thanks for the references Julian and Jim!