Slick 3.0.0-RC1 released

[Stefan Zeiger]

We have just released Slick 3.0.0-RC1, the first release candidate for “Reactive Slick”. You can find the source code here: https://github.com/slick/slick/tree/3.0.0-RC1. Builds for Scala 2.10 (2.10.4 recommended) and 2.11 (2.11.4 recommended) are available from Maven Central, as usual.

These are the major new features in Slick 3.0:

• A new API for composing and executing database actions.
• Support for the Reactive Streams API for streaming results from the database.
• Improved configuration of database connections via Typesafe Config, including built-in support for HikariCP.
• Support for nested Option types and non-primtive Option types in the Lifted Embedding.
• Properly typed outer join operators based on the improved Option support.
• A new macro-based implementation of the Plain SQL API with optional compile-time checking and type inference for embedded SQL statements.
• Many small improvements and bug fixes in the schema code generator.
• Optional multi-line formatting and indenting of generated SQL code for more readable debug output.

Most of the manual has been updated. The “Slick Testkit” and “Plain SQL Queries” chapters will follow soon. None of the standard Activator templates are available at the moment. They will be ported to 3.0 before the final release. There is a new section in the Migration Guide for upgrading from Slick 2.1. The commercial Slick Extensions package with additional database drivers is also available for this RC. See the manual for details on adding it to your build.

Please provide feedback on the stability of this RC if you encounter any problems. We are planning to publish the final 3.0.0 release if no blockers are discovered in the next two weeks.

Here is a complete list of changes since 2.1.0:

• bd0c9c4 Try to prevent some spurious failures on Travis CI
• 58a1db9 Proper treatment of CHAR(1) columns in JdbcModelBuilder:
• 7f0f62f Make code generator tests usable from other slick-extensions:
• 25370cb Fix bugs detected by Slick Extensions upgrade:
• 12faf5d Upgrade to Reactive Streams 1.0.0.RC3
• 648184c Package structure and project layout refactoring:
• 9c730c0 Upgrade to Reactive Streams 1.0.0.RC2:
• 24770de Add DatabaseConfig as a higher-level configuration abstraction:
• 759cbc2 Restore #$ support in the macro-based Plain SQL interpolation:
• 62ddadc Macro-based PlainSQL API Implemented TypedStaticQuery as tsql
• d63a018 UUID regression testkit:
• 0a44b74 Update LiftedEmbedding.scala
• 5af0d9b Distinguish between BLOB and BYTEA in PostgreSQL model builder
• ec7f2aa Use TestDB for code generator tests:
• cc8d3f0 Proper support for string lengths:
• 41c717a Support fetch size in statement parameters.
• 965604c Use force-insert semantics for inserting from queries and expressions.
• 0e58e7d Deprecate the Direct Embedding
• 0ba0dd5 Remove the blocking API that was introduced in 3.0.0-M1
• 2f64c79 Rename Action to DBIO:
• 86974bd Simplify SQLBuilder and formatting/indenting:
• 20378ec Quote SqlType argument in code generator.
• b1dca21 Make ColumnOptions more modular:
• 7272180 Use unconstrained text type as default on PostgreSQL
• 6cec3f7 Prefer prepareStatement(sql) instead of prepareStatement(sql, Int, Int)
• a3eec4f Support transaction isolation levels
• 6ea7fdb Generic fusion implementation for FlatMapAction and CleanupAction:
• 706f2b1 Allow the SQL statements to be overridden in SqlActions.
• a325f5e Run a user-defined init function on all created JDBC Statements
• 3315a3b Refactoring of ActionContext:
• dc687de Avoid long nested sequences of SchemaDescription.++ in generated code:
• 62d5e92 Quote illegal identifiers in generated Scala sources.
• 58c4ed5 Make ‘desiredColumnOrder’ in code generator non-final.
• a645205 Support statement parameters in the new Action-based API.
• b1c5b66 Switch JDBC metadata API and code generator to the new API.
• c92a615 Fix typo in orm-to-slick.rst
• d4f019f Move profile-level Action definitions outside of profiles.
• 5cd423f Deprecate Invoker convenience methods and iteratees
• 7d576c8 Properly deprecate .Implicit and .simple in all profile levels
• 30c135e Upgrade to sbt 0.13.7
• e9274ec Improve build system for documentation:
• a6e3b9b Further modularization of Slick:
• 4ab4a6b Switch Direct Embedding over to the new Action-based API.
• ac37f1c Update documentation for 3.0:
• e6562c7 Simplify Action Effect handling:
• 06d1ca0 Increase timeout for Reactive Streams TCK tests.
• efbe6f6 Bump version numbers:
• 4d42528 Add concurrency stress test for streaming API:
• 80929a6 Documentation plus minor improvements and fixes:
• 273b957 Update to Reactive Streams 1.0.0.M3 and fix remaining PR review issues:
• 688d7f2 Convert more test cases and add missing features:
• d862e0f Optimize the streaming engine and fix some bugs:
• 9ad62a9 Implement streaming via Reactive Streams API:
• 6a85767 Convert several test cases to the new API.
• 8ef6cde Add basic transaction support for database actions.
• 09590fd Implement session pinning for database Actions:
• 66a9b41 Basic implementation of action monad and asynchronous execution engine:
• bf17772 Use HikariCP by default (instead of no connection pooling):
• 4569c52 Simplify connection pool configuration
• cc92c26 Simplify thread pool configuration:
• 423e78c Remove the option to disable auto-commit for new connections.
• e4dddcd Remove BoneCP support and drop configuration compatibility with Play.
• 812100e Add HikariCP support in addition to BoneCP.
• bb58103 Add primitives and configuration for async execution of blocking I/O
• 0ea777a Fix broken build due to merging nested options support
• 8d4a02f Ensure generated file ends with a newline character.
• 9cdf7ec Make comment format match others in file.
• ef5f9a8 Avoid indenting blank lines, to avoid trailing whitespace.
• 8225a8f Add Option-extended joins:
• 75583dc Lifted Embedding encoding of nested and multi-column Option types:
• de389a4 Remove the Column type.
• 5da96b4 Update orm-to-slick.rst
• 7a1d119 modified to use GlobalConfig.
• 777dbeb Add SimpleLiteral factory (and fix one unrelated unchecked warning)
• 82b258f Use Unicode symbols in tree and table dumps for better readability.
• f54d272 Add BoneCP support to Database.forConfig.
• 369ad73 Use application.conf for Slick settings.
• 61d032b Corrected docs to use “inSet” instead of “inSeq”
• b0b0252 Fix forceInsertAll for single argument
• 770d0a1 Update build for 2.2.0-SNAPSHOT
• cb8183d Added support for string function repeat.
• 57aa9e7 Workaround to pass Travis CI.
• 7a58f7f Try for SQL String Indentation.

--
Stefan Zeiger
Slick Tech Lead
Typesafe - Build Reactive Apps!
Twitter: @StefanZeiger

[Matan Safriel]

Thanks Stefan!

In http://slick.typesafe.com/doc/3.0.0-RC1/gettingstarted.html#adding-slick-to-your-project, is there anything shorter to use in place of the following?

Await.result(db.run(setup), Duration.Inf)

I can wrap it myself, but wondering if there's anything inherent in the Slick api, that comes to mind for substituting this composition of 6 components. Likewise for "result(db.run(coffees.result)" 

would it not make sense to provide api that returns the result of a run...?

By the way, cool API with being able to use DBIO.seq there.. 

Matan

[Stefan Zeiger]

On 2015-02-24 1:57, Matan Safriel wrote:

In http://slick.typesafe.com/doc/3.0.0-RC1/gettingstarted.html#adding-slick-to-your-project, is there anything shorter to use in place of the following?

Await.result(db.run(setup), Duration.Inf)

No. The intention is that you use Slick in a reactive application which is written in a non-blocking asynchronous style, so you should never use Await. It is only needed for code snippets which have to run standalone. This was a strategic decision for the whole platform. It is technically feasible to add a blocking API that makes this case simpler and more efficient (see https://github.com/slick/slick/pull/1061).

[Stefan Zeiger]

On 2015-02-20 19:39, Stefan Zeiger wrote:

None of the standard Activator templates are available at the moment. They will be ported to 3.0 before the final release.

The main tutorial "Hello Slick" is now available for 3.0.0-RC1: http://typesafe.com/activator/template/hello-slick-3.0

[Sam Halliday]

Nice!

Looking forward to trying out the new streaming API :-)

Does each Source hold/require a persistent connection to the database, or will it perform incremental SELECTs so that it doesn't hog the connection resource? e.g. imagine you have maximum 10 connections but want 100 different Sources as part of stream graphs with slow consumers.

It would be good to be able to swap between repeating SELECTs that do fixed batch loads (e.g. get rows 1-100, then 101-200, etc with a new connection/transaction each time), and an analogy to Spring's `preFetch` (which is very efficient but hogs a connection per query). In the former, will there be any pre-fetching? e.g. in the 1-100, 101-200 example there would be a delay when issuing each batch query so the time to get each row will have spikes, it would be good to perform some read-ahead to eliminate such spikes.

I'm primarily interested in creating Sources from very large tables that cannot be held in memory.

Best regards,

Sam

[Stefan Zeiger]

On 2015-02-25 23:05, Sam Halliday wrote:

Does each Source hold/require a persistent connection to the database, or will it perform incremental SELECTs so that it doesn't hog the connection resource? e.g. imagine you have maximum 10 connections but want 100 different Sources as part of stream graphs with slow consumers.

Streaming is done from a single ResultSet. It will keep a Connection open while it is stalled due to back-pressure but it does not block a Thread. Ideally, this is just a database cursor on the server side. It can be a separate TCP connection but that's still relatively cheap (a file handle).

Note that you can stream any composite action as long as the "last" action (the one that eventually returns some data) supports streaming. The action starts running when you attach a Subscriber to the Publisher and the stream ends when the whole action is done (i.e. if you have any cleanup actions after the streaming action, they run before the stream terminates; otherwise there would be no way to signal errors that happen during cleanup).

It would be good to be able to swap between repeating SELECTs that do fixed batch loads (e.g. get rows 1-100, then 101-200, etc with a new connection/transaction each time), and an analogy to Spring's `preFetch` (which is very efficient but hogs a connection per query). In the former, will there be any pre-fetching? e.g. in the 1-100, 101-200 example there would be a delay when issuing each batch query so the time to get each row will have spikes, it would be good to perform some read-ahead to eliminate such spikes.

That's in interesting idea. It couldn't be done transparently because you don't get any consistency between individual pages, and you'd need to start at the CompiledQuery level because you have to compile the pagination support into the query. DatabaseComponent.runInContext currently pins the session when you run a fused SynchronousDatabaseAction but as long as you only need to stream a single action it should be possible to implement the desired behavior in a SynchronousDatabaseAction (which gives you most of the low-level streaming machinery for free)

I'm primarily interested in creating Sources from very large tables that cannot be held in memory.

You also get that from a single ResultSet (as long as you don't rev up consistency or result set levels which can make the server or client cache all data behind the scenes). When a stream is stalled on the subscriber, Slick will buffer only a single element ahead, thus causing the JDBC driver to prefetch the next result set page when it happens at a page boundary. It does not read the whole result into memory first.

[Sam Halliday]

Thanks Stefan,

On Thursday, 26 February 2015 13:45:49 UTC, Stefan Zeiger wrote:

On 2015-02-25 23:05, Sam Halliday wrote:

Does each Source hold/require a persistent connection to the database, or will it perform incremental SELECTs so that it doesn't hog the connection resource? e.g. imagine you have maximum 10 connections but want 100 different Sources as part of stream graphs with slow consumers.

Streaming is done from a single ResultSet. It will keep a Connection open while it is stalled due to back-pressure but it does not block a Thread. Ideally, this is just a database cursor on the server side. It can be a separate TCP connection but that's still relatively cheap (a file handle).

OK, I suspected as much. I have written this sort of code in the past and it works well so long as there is no contention on connections (e.g. via JdbcTemplate and preFetch which backpressures by only giving what you take).

However...

It would be good to be able to swap between repeating SELECTs that do fixed batch loads (e.g. get rows 1-100, then 101-200, etc with a new connection/transaction each time), and an analogy to Spring's `preFetch` (which is very efficient but hogs a connection per query). In the former, will there be any pre-fetching? e.g. in the 1-100, 101-200 example there would be a delay when issuing each batch query so the time to get each row will have spikes, it would be good to perform some read-ahead to eliminate such spikes.

That's in interesting idea. It couldn't be done transparently because you don't get any consistency between individual pages, and you'd need to start at the CompiledQuery level because you have to compile the pagination support into the query. DatabaseComponent.runInContext currently pins the session when you run a fused SynchronousDatabaseAction but as long as you only need to stream a single action it should be possible to implement the desired behavior in a SynchronousDatabaseAction (which gives you most of the low-level streaming machinery for free)

Yes, this is the problem. I have encountered (actually a few times in the last month) problems when it is not possible to hold a database connection for too long or the system will be starved. In such situations, it is necessary to do batch loading of SELECT (or batch REMOVEs to avoid OOMs on the DB).

In my project's codebase I'm currently trying to cache/readahead such calls so that there is no waiting at the boundaries. Here we potentially have hundreds of subscribers all wanting a stream of events from a large `SELECT *` query but realistically we can only do it over 10 or 20 live connections. This particular code is not using Slick or Reactive Streams so I doubt anything I produce would be of use to you as it is a simple read-ahead exercise.

Best regards,

Sam

[Stefan Zeiger]

On 2015-02-24 17:27, Stefan Zeiger wrote:

On 2015-02-20 19:39, Stefan Zeiger wrote:

None of the standard Activator templates are available at the moment. They will be ported to 3.0 before the final release.

The main tutorial "Hello Slick" is now available for 3.0.0-RC1: http://typesafe.com/activator/template/hello-slick-3.0

"Slick Plain SQL" has also been updated: http://typesafe.com/activator/template/slick-plainsql-3.0

[teste...@gmail.com]

It seems that the github code hasn't been updated even though the link says that it's the new version.

[teste...@gmail.com]

By the way, in the Hello world example, the reason why each example is connected between a flatMap inside a long Future is to force it to execute it sequentiality? it's the first time I see something like that :)

[Stefan Zeiger]

On 2015-03-09 11:30, teste...@gmail.com wrote:

By the way, in the Hello world example, the reason why each example is connected between a flatMap inside a long Future is to force it to execute it sequentiality? it's the first time I see something like that :)

It's a straight port of the old version. Since the database is modified by the tutorial, there are some parts that need to run in the correct order. Others could be run in parallel but they print to stdout so you wouldn't be able to make much sense of the interlaved output.

On Monday, March 9, 2015 at 10:37:27 AM UTC+1, teste...@gmail.com wrote:

It seems that the github code hasn't been updated even though the link says that it's the new version.

It appears that the links with branch names on the Activator site are broken. You need to switch to the branch `slick-3.0`.

[Stefan Zeiger]

On 2015-03-21 21:55, adna...@gmail.com wrote:

 I was trying to create a scala program with slick 3.0 and I am getting the following error:


Error:scalac: bad symbolic reference. A signature in BasicDriver.class refers to term typesafe
in package com which is not available.
It may be completely missing from the current classpath, or the version on
the classpath might be incompatible with the version used when compiling BasicDriver.class.

I am using scala 2.10.4 and slick 2.10-3.0.0-RC1 and my IDE is IDEA 14.0.3 community edition

Is this a new project or did you upgrade from Slick 2.1? In the latter case, please try to reimport it in IntelliJ. This could be a classpath issue. BasicDriver has references to com.typesafe.config.Config. OTOH, Slick already used Typesafe Config in 2.1.

[adna...@gmail.com]

Thank you for you answer Stefan.

This is a new project created from scratch and I am using the library published on Maven Central repository.