[ANN] Quill - Compile-time Language Integrated Query for Scala

[Flávio Brasil]

http://getquill.io

Quill provides a Quoted Domain Specific Language (QDSL) to express queries in Scala and execute them in a target language. The library's core is designed to support multiple target languages and the current version features support for Structured Language Queries (SQL).

1. Boilerplate-free mapping: The database schema is mapped using simple case classes.
2. Quoted DSL: Queries are defined inside a quote block. Quill parses each quoted block of code (quotation) at compile time and translates them to an internal Abstract Syntax Tree (AST)
3. Compile-time SQL generation: The db.run call reads the quotation's AST and translates it to the target language at compile time, emitting the SQL string as a compilation message. As the query string is known at compile time, the runtime overhead is very low and similar to using the database driver directly.
4. Compile-time query validation: If configured, the query is verified against the database at compile time and the compilation fails if it is not valid.

[Simon Ochsenreither]

Very interesting! Is this an academic/commercial/open-source project?

[Flávio Brasil]

It's an open source project under Apache 2, developed in my free time.

I'll be creating some github issues with a roadmap in the next days. Contributors are welcome. :)

2015-11-27 11:16 GMT-08:00 Simon Ochsenreither <simon.och...@gmail.com>:

Very interesting! Is this an academic/commercial/open-source project?

--
You received this message because you are subscribed to the Google Groups "scala-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Ochsenreither]

Alright, thanks for the information. How does it compare to Slick?

[Flávio Brasil]

Hi Simon,

I've just published a comparison document comparing Quill and Slick:

https://github.com/getquill/quill/blob/master/SLICK.md

Best,

Flavio

2015-11-27 12:01 GMT-08:00 Simon Ochsenreither <simon.och...@gmail.com>:

Alright, thanks for the information. How does it compare to Slick?

--

[Simon Ochsenreither]

Thanks, very interesting!

What was the reasoning behind the explicit quoting requirement (instead of making e. g. query the method that does that)?

Just wondering because there are a few interesting bits and pieces which would be interesting to use for other backends (like collections).

The code which assembles the query seems to be nicely separated from the code that executes it, which looks pretty good!

[Simon Ochsenreither]

Just a few additional questions:

- How hard would it be to write a new backend (like collections)?
- How hard is it to extend the "query" language with additional operations?

[Flávio Brasil]

> What was the reasoning behind the explicit quoting requirement (instead of making e. g. query the method that does that)?

I'm not sure I understand the question, could you give an example?

> - How hard would it be to write a new backend (like collections)?

> - How hard is it to extend the "query" language with additional operations?

The integration with a new backend is relatively simple, the backend itself is the complex part. I'd say that for scala code generation, it'd be better to avoid quill's internal AST and use only its quotation mechanism. That way, there's no need to reflect all possible Scala constructs in the internal AST.

--

[Simon Ochsenreither]

> What was the reasoning behind the explicit quoting requirement (instead of making e. g. query the method that does that)?

I'm not sure I understand the question, could you give an example?

I was wondering why

quote{
  query[Person].filter(...).map(...)
}

instead of

query[Person].filter(...).map(...)

Is it because it's currently hard to pass the information on between the different operations?

Thanks!

[Flávio Brasil]

I see. Quill's initial implementation was using the second approach, but I decided to introduce `quote` for three reasons:

1. It makes easier to identify the boundaries of a quotation

2. Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

3. The slowest part of the compilation is lifting/unlifting the quotation's AST. The second approach requires lifting/unlifting for each method call, while using `quote` lifting/unlifting is required only when a quotation is used as part of another quotation.

Best,

Flavio

--

[Simon Ochsenreither]

2. Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

Mhh, could you expand on that?
 

3. The slowest part of the compilation is lifting/unlifting the quotation's AST. The second approach requires lifting/unlifting for each method call, while using `quote` lifting/unlifting is required only when a quotation is used as part of another quotation.

Interesting, I didn't expect it to have such an impact. Do you have a rough estimation of the difference?

My motivation is largely that I'm looking into having a sane, common API for these "bulk-operations" (for the lack of a better description), so that we can write the same code once, apply it to different sources of data (databases, collections, Spark, ...) and execute it efficiently.

That sane API is actually quite close to what you are doing, by letting all operations compose against a generic type first (without executing it), instead of carrying large type-signatures to rebuild the data-strcuture after every step eagerly.

[virtualeyes]

> Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

think he means you can do:

quote{
  (id: UserId) => query[User].filter(_.id == userId)
}

instead of Slick's (legacy) approach to generating a prepared statement:

Parameters[UserId].flatMap{
  id=> User.filter(_.id === id)
}

Quill's approach is appealing, though it would be great to somehow get rid of all the query[T]s and get the bare T syntax that Slick gets through implicit conversion. Would need to tag the case classes with a marker trait for that to happen though since Quill by default doesn't use table mappings. On that note, would be nice to include foreign key relationships thereby gaining that concise Slick implicit join syntax:

for{
  ur <- UserRole; u <- ur.userFk; r <- ur.roleFk
} yield(u,ur,r)

ideally extended to joins*:

quote{
  (t,p) <- Team join Player on $(_.teamFk)
  (_,s) <- p join Scoring on $(_.playerFk)
} yield(t,p,s)

but the latter is just me wanting to concise all the things ;-)

Quill author, inner join seems to be missing in the library, reason for this??

[Flávio Brasil]

2015-12-03 7:54 GMT-08:00 virtualeyes <sit...@gmail.com>:

> Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

think he means you can do:

quote{
  (id: UserId) => query[User].filter(_.id == userId)
}

instead of Slick's (legacy) approach to generating a prepared statement:

Parameters[UserId].flatMap{
  id=> User.filter(_.id === id)
}

Exactly. It makes the information about the quotation's inputs accessible for beta reduction, since Quill uses closed quotation.

 

Quill's approach is appealing, though it would be great to somehow get rid of all the query[T]s and get the bare T syntax that Slick gets through implicit conversion. Would need to tag the case classes with a marker trait for that to happen though since Quill by default doesn't use table mappings.

On first thought, I'm not in favor of this approach. Please open a github issue with a proposal in case you'd like to discuss it further.

 

On that note, would be nice to include foreign key relationships thereby gaining that concise Slick implicit join syntax:

for{
  ur <- UserRole; u <- ur.userFk; r <- ur.roleFk
} yield(u,ur,r)

ideally extended to joins*:

quote{
  (t,p) <- Team join Player on $(_.teamFk)
  (_,s) <- p join Scoring on $(_.playerFk)
} yield(t,p,s)

but the latter is just me wanting to concise all the things ;-)

Sounds like good features to be added. Feel free to open issues for them.

 

Quill author, inner join seems to be missing in the library, reason for this??

Monadic joins are translated to implicit inner joins. Are there cases where explicit inner joins are required?

 

On Thursday, December 3, 2015 at 3:06:20 AM UTC-5, Simon Ochsenreither wrote:

2. Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

Mhh, could you expand on that?
 

3. The slowest part of the compilation is lifting/unlifting the quotation's AST. The second approach requires lifting/unlifting for each method call, while using `quote` lifting/unlifting is required only when a quotation is used as part of another quotation.

Interesting, I didn't expect it to have such an impact. Do you have a rough estimation of the difference?

My motivation is largely that I'm looking into having a sane, common API for these "bulk-operations" (for the lack of a better description), so that we can write the same code once, apply it to different sources of data (databases, collections, Spark, ...) and execute it efficiently.

That sane API is actually quite close to what you are doing, by letting all operations compose against a generic type first (without executing it), instead of carrying large type-signatures to rebuild the data-strcuture after every step eagerly.

--

[Flávio Brasil]

2015-12-03 0:06 GMT-08:00 Simon Ochsenreither <simon.och...@gmail.com>:

2. Functions can be declared using the normal syntax within quotations. The second approach would require something like `param[Int].map(i => ...)`.

Mhh, could you expand on that?
 

3. The slowest part of the compilation is lifting/unlifting the quotation's AST. The second approach requires lifting/unlifting for each method call, while using `quote` lifting/unlifting is required only when a quotation is used as part of another quotation.

Interesting, I didn't expect it to have such an impact. Do you have a rough estimation of the difference?

It's considerably slower, but a compilation cache would make it as fast as the first approach.

 

My motivation is largely that I'm looking into having a sane, common API for these "bulk-operations" (for the lack of a better description), so that we can write the same code once, apply it to different sources of data (databases, collections, Spark, ...) and execute it efficiently.

That sane API is actually quite close to what you are doing, by letting all operations compose against a generic type first (without executing it), instead of carrying large type-signatures to rebuild the data-strcuture after every step eagerly.

--

[Simon Ochsenreither]

Wouldn't the issue of retaining information go away completely with TASTY? I think one wouldn't even need a custom AST to represent the query (at least not as input, but as a transformation step it might be useful).

[Flávio Brasil]

Yes, with TASTY the implementation would probably be simpler and more performant. 

2015-12-04 7:46 GMT-08:00 Simon Ochsenreither <simon.och...@gmail.com>:

Wouldn't the issue of retaining information go away completely with TASTY? I think one wouldn't even need a custom AST to represent the query (at least not as input, but as a transformation step it might be useful).

--