API challenge: Executor, Configuration, ExecuteListener, ConnectionProvider

[Christopher Deckers]

Hi Lukas,

OK, here is the API challenge of the day.
This is to sum up my views on Executor and Configuration and hopefully also make users of the library react! :)

I believe there are several cases for Executor creation:

- Create an Executor with ad-hoc parameters.

- Create an Executor with a Configuration.

- And eventually: create an Executor with a Configuration, overridden with ad-hoc parameters.

I am mainly focusing on the 2nd/3rd cases: creating an Executor with a Configuration.

When someone uses a Configuration, I believe that the purpose is to not repeat parameters, to have the same parameters for all Executors. In some cases, which I think are limited, someone could have more than one Configuration, but would still use those as shared parameters for sets of Executors. The purpose of this e-mail is to suggest strengthening this notion of Configuration as a global shareable set of parameters.

In my view, because a Configuration shares parameters, it should by nature not share object instances that have a state that depends on execution. Otherwise, 2 calls from different threads could fail if they happen at the same time; 2 sequencial queries where the first one lazily fetches the results could also fail.
The best a Configuration should do in this area is to provide a factory of such execution-dependant objects. Actual instances should be set on the Executor directly.

The main objects that may have a state which depends on execution currently are:

- ExecuteListener: an actual instance is plugged to the Configuration object. Storing state directly in that class is very dangerous and the workaround is to use the data map of the ExecuteContext to store and retrieve execution-related state. Failing to do so may work until 2 threads call it at the same time, etc.

- ConnectionProvider: the specification is very loose on how to implement it, which means that the same connection could be acquired by 2 threads using the same configuration.

Because of the current structure of the API, it is easy to misuse and to share a configuration object which eventually shares stateful execute listeners and connections (through connection providers). It would eventually blow up randomly depending on threading or query execution flows (preferably once deployed in production) and would of course be very hard to debug.

I think ExecuteListener and ConnectionProvider should be replaced so that the user can:

- Set an actual instance of X (see below) on an Executor instance.

- Set a factory of X in the Configuration, which can be used by many Executor instances.

About ExecuteListener, the user could either:

1. Define an ad-hoc ExecuteListener instance:

Executor executor = new Executor(configuration);
executor.getExecuteListeners().add(new MyExecuteListener());

2. Define a global ExecuteListener factory:

configuration = new Configuration();

configuration.getExecuteListenerFactories().add(new ExecuteListenerFactory() {

  public ExecuteListener createExecuteListener() {

    return new MyExecuteListener();

  }

});

// Everywhere the configuration is used, then:

Executor executor = new Executor(configuration);

A few notes:

- If per-execution state is needed, the factory would just create a new instance, which is the general case. Of course, the factory can supply the same instance of the ExecuteListener if it wishes so, but then it is not a factory anymore and as such the implementor will have to create its ExecuteListener subclass with thread safety in mind. Such cases are for example for aggregation of statistics over multiple queries and threads, and is by nature required to be coded in a thread safe way.

- If using a factory seems like clutter, keep in mind that the configuration is to define shared parameters and is not done at many places in the code. Moreover, with Lambda in Java 8 this turns into a one liner.

About the ConnectionProvider, the user could either:

1. Define an ad-hoc connection instance:

Executor executor = new Executor(conn, sqlDialect);

2. Define a global factory:
configuration = new Configuration();

configuration.setConnectionProvider(new MyPoolBasedConnectionProvider());

// Everywhere the configuration is used, then:

Executor executor = new Executor(configuration);

A few notes:

- The ConnectionProvider would not be loose in its meaning as it is currently and it would specify that every connection that is obtained through acquire() cannot be returned twice until it is returned with release(conn). This way, the connection provider can be used by concurrent queries.

- A side effect, but not the main motivation, is that the ConnectionProvider would be a safe place to acquire/release dedicated connections. 3rd party ExecuteListener or whatever tools could make use of such capability.

- Maybe we should also have signatures like: "new Executor(Configuration, conn)". The general idea would be that the configuration is used, but the executor can override some aspects of it like what I did with the ExecuteListener above. In fact, maybe we should have an API so that user can call "new Executor(Configuration)" and then call "executor.setConnection(conn)" if so they wish.

- Because there is this idea that the Configuration is shared, when an Executor is instanciated it would create an internal copy to merge overridden aspects (connection, execute listeners, even rendering capabilities that could be accessed through executor.getConfiguration()).

I strongly believe that it would make the API easier to understand (global vs ad-hoc) and would reduce the chances of misusing it. Please let me know if my thoughts are completely off-track or if such API would be beneficial to jOOQ.

-Christopher

[Lukas Eder]

Hi Christopher,

I did not forget this thread :-)
However, I want to clear the other topics first, before going into
this one. This one still needs some thinking on my side.

Cheers
Lukas

2013/3/30 Christopher Deckers <chr...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "jOOQ User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jooq-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Lukas Eder]

Christopher,

Finally I get to answering this E-Mail. It's been a busy week!

A couple of definitions to keep up to date with the latest changes

- DSL: The new name of what was called "Factory" before
- DSLContext: The new name of what was called "Executor" before
- DSL.using(...), returning DSLContext: The new way of doing what
could be done through "new Executor(...)" before (with the same
overloading)

This is the latest state as available on Github master and on the
Sonatype SNAPSHOT repository for 3.0-RC3. Note, that using() was
chosen rather than with(), to avoid issues with Scala/Groovy reserved
words. See comments by Ben Manes here:
https://github.com/jOOQ/jOOQ/issues/2379

Now, comments inline:

2013/3/30 Christopher Deckers <chr...@gmail.com>:

> Hi Lukas,
>
> OK, here is the API challenge of the day.
> This is to sum up my views on Executor and Configuration and hopefully also
> make users of the library react! :)

Hehe, no one seemed to have reacted to this. :-)
It's a tough and complex topic! Unlike with the other topics in that
area, this one is harder to get "right" - whatever "right" is.

> I believe there are several cases for Executor creation:
> - Create an Executor with ad-hoc parameters.
> - Create an Executor with a Configuration.
> - And eventually: create an Executor with a Configuration, overridden with
> ad-hoc parameters.
>
> I am mainly focusing on the 2nd/3rd cases: creating an Executor with a
> Configuration.

OK, I agree with that.

> When someone uses a Configuration, I believe that the purpose is to not
> repeat parameters, to have the same parameters for all Executors. In some
> cases, which I think are limited, someone could have more than one
> Configuration, but would still use those as shared parameters for sets of
> Executors. The purpose of this e-mail is to suggest strengthening this
> notion of Configuration as a global shareable set of parameters.

Yes, I agree with that, too. While it is a good use-case to create
ad-hoc configurations through what you called "ad-hoc parameters", in
the long run, users will reuse mostly a single, global Configuration.

> In my view, because a Configuration shares parameters, it should by nature
> not share object instances that have a state that depends on execution.
> Otherwise, 2 calls from different threads could fail if they happen at the
> same time; 2 sequencial queries where the first one lazily fetches the
> results could also fail.
> The best a Configuration should do in this area is to provide a factory of
> such execution-dependant objects. Actual instances should be set on the
> Executor directly.

I wish to exclude the DSLContext / Executor from these thoughts. It is
complicating things unnecessarily. It has no state / lifecycle of its
own. Introducing it in the way I originally did was confusing. The
thought of having "new Executor" along with the name "Executor" seemed
to indicate that it has a life of its own. It doesn't. It had a 1-to-1
relationship with Configuration, adding DSL capabilities. As I will
explain further down, everything you want to do can be done without
allowing "Executor" to have its own lifecycle.

There are only these types of lifecycle in jOOQ:

- The Configuration's. It is allowed to be global, but if desired, it
can also be "local", i.e. execution-based. Let's assume it is global.
- The ExecuteContext's. It is defined to be "local", i.e.
execution-based. An ExcuteContext will go through at least the
ExecuteListener.start() and ExecuteListener.end() events. For that, it
may (but is not required to) go through ConnectionProvider.acquire()
and ConnectionProvider.release() events. An ExecuteContext is always
understood in the context of the Configuration that created it.
- The RenderContext's and BindContext's. These are irrelevant for this
discussion.

Let's reiterate the above, looking at the most up-to-date API:

// Wrap a Configuration in a DSLContext:
DSLContext ctx = using(configuration);
// ctx2 is no different from ctx1. It is just a DSL-object.
DSLContext ctx2 = using(configuration);
// Create a local ExecuteContext from the above Configuration
ctx.fetch("SELECT 1 FROM DUAL");
// Create another local ExecuteContext from the above Configuration
ctx.fetch("SELECT 2 FROM DUAL");

> The main objects that may have a state which depends on execution currently
> are:
> - ExecuteListener: an actual instance is plugged to the Configuration
> object. Storing state directly in that class is very dangerous and the
> workaround is to use the data map of the ExecuteContext to store and
> retrieve execution-related state. Failing to do so may work until 2 threads
> call it at the same time, etc.
> - ConnectionProvider: the specification is very loose on how to implement
> it, which means that the same connection could be acquired by 2 threads
> using the same configuration.

I agree. Although, I think that the ConnectionProvider *should* be
loose to cover all use cases.

> Because of the current structure of the API, it is easy to misuse and to
> share a configuration object which eventually shares stateful execute
> listeners and connections (through connection providers). It would
> eventually blow up randomly depending on threading or query execution flows
> (preferably once deployed in production) and would of course be very hard to
> debug.

Yes, it is hard to get it right. Do note that this was much harder to
get right before jOOQ 3.0, before the introduction of a
ConnectionProvider. In order to get it right, a properly pooled (and
thus shareable) DataSource was needed.

Even now, looking at rather elaborate integrations shows how hard
things can get:
https://gist.github.com/ben-manes/5316351

While ConnectionProvider was easy to implement in this case

Yes, here the configuration should hold a sort of
ExecuteListenerFactory (actual name may differ), rather than actual
ExecuteListener state. Where I don't agree is the distinction between
1. and 2. for two reasons:

A) Users can already create new Configurations and set them on
per-execution configured DSLContexts
B) jOOQ can provide default implementations for the
ExecuteListenerFactory, e.g. a DefaultExecuteListenerFactory, which
would contain a constant list of ExecuteListeners. This default
behaviour would model your case number 1., or today's behaviour in
jOOQ.

Again, I don't think that we need the additional complexity by adding
the possibility of creating "per-Executor" setups. Users that need
such a setup can already

A) Create new Configurations and set them on per-execution configured
DSLContexts
B) Use jOOQ's existing default implementations, e.g.
DefaultConnectionProvider, by calling the following:

// Create an ad-hoc Configuration
using(connection, dialect);

> I strongly believe that it would make the API easier to understand (global
> vs ad-hoc) and would reduce the chances of misusing it. Please let me know
> if my thoughts are completely off-track or if such API would be beneficial
> to jOOQ.

No, you're completely on-track. The actual solution might in fact be a
bit simpler, as you described above. The main thing here is that there
shouldn't be a List<ExecuteListener> in Configuration. Instead, there
should be an ExecuteListenerProvider, as an analogue to
ConnectionProvider.

I think its contract should be loose as well, though. I feel that
setting strong contracts on these Provider types might:

1. Make things even harder to implement correctly
2. Not solve the actual problem, which is ExecuteListener state in the
Configuration

I have registered #2388 for the above. This will go into 3.0-RC3:
https://github.com/jOOQ/jOOQ/issues/2388

If you feel that there is still a missing piece as I might not have
gotten the full picture yet (e.g. contracts too loose, additional
lifecycle for DSLContext), feel free to continue this discussion.

Thanks for your feedback!

Cheers
Lukas

[Witold Szczerba]

On 6 April 2013 23:30, Lukas Eder <lukas...@gmail.com> wrote:

2013/3/30 Christopher Deckers <chr...@gmail.com>:

> When someone uses a Configuration, I believe that the purpose is to not
> repeat parameters, to have the same parameters for all Executors. In some
> cases, which I think are limited, someone could have more than one
> Configuration, but would still use those as shared parameters for sets of
> Executors. The purpose of this e-mail is to suggest strengthening this
> notion of Configuration as a global shareable set of parameters.

Yes, I agree with that, too. While it is a good use-case to create
ad-hoc configurations through what you called "ad-hoc parameters", in
the long run, users will reuse mostly a single, global Configuration.

Hi again,
I know it may sound boring, you may have enough of me on that subject already, so I promise this is my last attempt to try to convince you that:

****
Sharing a configuration object across entire application just to be able to get an Executor in return, is an anti-pattern, it is evil and every time when someone suggest to do so, an innocent kitten dies.
****

Thanks, and I am sorry for inconvenience. I just had to be sure I did everything I could, to let you know about this, and to stop you from polluting the beautiful JOOQ DSL with that "using(configuration)" thing. I do respect your right to do as you will and I am grateful I can still just separate the creation of Executor instances (I hope you will continue to provide an interface for this as it was in 2.x). The only inconvenience for me is that I will have to explain to everyone in my projects that using DSL#using is forbidden, please declare a dependency on Executor if you want to use it, but it is extremely low price for this great product.

Regards,
Witold Szczerba

[Lukas Eder]

Hi Witold ;-)

2013/4/7 Witold Szczerba <pljos...@gmail.com>:

>
> On 6 April 2013 23:30, Lukas Eder <lukas...@gmail.com> wrote:
>>
>>
>> 2013/3/30 Christopher Deckers <chr...@gmail.com>:
>>
>>
>> > When someone uses a Configuration, I believe that the purpose is to not
>> > repeat parameters, to have the same parameters for all Executors. In
>> > some
>> > cases, which I think are limited, someone could have more than one
>> > Configuration, but would still use those as shared parameters for sets
>> > of
>> > Executors. The purpose of this e-mail is to suggest strengthening this
>> > notion of Configuration as a global shareable set of parameters.
>>
>> Yes, I agree with that, too. While it is a good use-case to create
>> ad-hoc configurations through what you called "ad-hoc parameters", in
>> the long run, users will reuse mostly a single, global Configuration.
>>
>
> Hi again,
> I know it may sound boring, you may have enough of me on that subject
> already, so I promise this is my last attempt to try to convince you that:
>
> ****
> Sharing a configuration object across entire application just to be able to
> get an Executor in return, is an anti-pattern, it is evil and every time
> when someone suggest to do so, an innocent kitten dies.
> ****

I'm not convinced yet. But not because I "strongly believe" that the
current solution is perfect, I might just (still!) not understand your
point of view. Let's put it this way:

// jOOQ 2.x:
new Factory(...).select();

// jOOQ 3.0-RC1 and RC2:
new Executor(...).select();

// jOOQ 3.0-RC3
DSL.using(...).select();

They're exactly the same, so I think I haven't introduced something
new, right? Or are you talking about the fact that now, using(...) is
overloaded such that a Configuration object can be passed to it -
which wasn't possible before? The relation between XX and
Configuration was this:

jOOQ 2.x: Factory implements Configuration
jOOQ 3.0-RC1 and RC2: Executor implements Configuration
jOOQ 3.0-RC3: DSLContext "has-a" Configuration

So, in my opinion, things haven't changed fundamentally between 2.x
and 3.x. But things have gotten better / easier for those that argue
that a "configuration" is a global, thread-safe thing, given that I
follow Christopher's advice on this thread. But you don't have to make
"configuration" a global thing. You can choose the configuration
instance's scope yourself.

So, where exactly am I still wrong? Can you explain without resorting
to kittens?

> Thanks, and I am sorry for inconvenience. I just had to be sure I did
> everything I could, to let you know about this, and to stop you from
> polluting the beautiful JOOQ DSL with that "using(configuration)" thing. I
> do respect your right to do as you will and I am grateful I can still just
> separate the creation of Executor instances (I hope you will continue to
> provide an interface for this as it was in 2.x).

Yes, DSLContext is now an interface. It was probably a bad idea to
make RC1 / RC2's Executor a class.

> The only inconvenience for
> me is that I will have to explain to everyone in my projects that using
> DSL#using is forbidden, please declare a dependency on Executor if you want
> to use it, but it is extremely low price for this great product.

Thanks. Maybe you will end up by liking this part, too :-)

Cheers
Lukas

[Witold Szczerba]

7 kwi 2013 10:48, "Lukas Eder" <lukas...@gmail.com> wrote:
>
>
> I'm not convinced yet. But not because I "strongly believe" that the
> current solution is perfect, I might just (still!) not understand your
> point of view. Let's put it this way:
>
> // jOOQ 2.x:
> new Factory(...).select();
>
> // jOOQ 3.0-RC1 and RC2:
> new Executor(...).select();
>
> // jOOQ 3.0-RC3
> DSL.using(...).select();

Hi,
to a answer your question, what I specifically don't like is that now, starting from 3.0, JOOQ will propagate an anti pattern of getting the configuration object to be able to "enter" the DSL.

Look at the very first example of 2.6 branch from jooq.org landing page:

----------------------
create.selectFrom(BOOK)
      .where(PUBLISHED_IN.equal(2011))
      .orderBy(TITLE)
----------------------

That is it. Entire example. What does it tell you? Get the "create" (I call this instance a jooq in my app) and then write the code as in example. Where to get the "create" from? It doesn't matter. This is not the business of JOOQ. Create it by yourself, use Spring, Guice, CDI, inject manually, use the lookup pattern, there are many options, none of them are the JOOQ's interest.

Now, I can see all the examples are going to be rewritten like:

----------------------
using(configuration)
      .selectFrom(BOOK)
      .where(PUBLISHED_IN.equal(2011))
      .orderBy(TITLE)
----------------------

,which means: get the configuration from somewhere, which is breaking, at least, the principle of least knowledge. IMHO this is one of the most important and also most ignored principle in programing in general.

Also, I do believe this is polluting the DSL, because there is no equivalent of the first statement in SQL. And last, but not least, when writing:
import static org.jooq.DSL.*
we do import all the goodness of JOOQ plus the "using" evil stuff. One cannot import everything BUT the "using" stuff. This is of course the minor problem. We can just ignore that method, but the "damage" caused by encouraging developers to use it, as the "official" documentation says, as the DSL API says, is the thing I am trying to avoid.

Regards,
Witold Szczerba

[Lukas Eder]

Hi Witold,

And thanks for clarifying.

2013/4/7 Witold Szczerba <pljos...@gmail.com>:

True. Let's have a look at an example:

DSLContext create = DSL.using(configuration);

In every day work, no difference there :-)

Rest assured that because of a significant amount of inertia and
lazyness, I'm not going to rewrite the whole manual and replace all
the examples. I don't even think that you should be writing "using()"
in front of every database call, myself. There will be a couple of
examples actually promoting the use of "using()". But I agree with you
that most of the time, people will store this DSLContext instance
somewhere, or create a factory method for it, or whatever. It doesn't
matter in everyday SQL work. So, things will stay the same. In other
words, I'd expect people to do the right thing out of lazyness.

I want to repeat my example of the XML Transformer. The complete call
chain to transform a document is this:

TransformerFactory.newInstance().newTransformer().transform(source, target);

But users will soon get wary of this and store the TransformerFactory
and / or the Transformer instance in some local context, e.g. by
pooling transformers. The above will become

transformer.transform(source, target);

The same applies to JDBC. No one will explicitly fetch a Connection
from a DataSource for every statement:

dataSource.getConnection().createStatement();

They have some sort of local context, where the Connection is stored:

connection.createStatement();

But it would be terribly wrong for jOOQ not to provide *some* sort of
API to construct that local DSLContext for you. And from what I've
learned in recent discussions, people had preferred writing
"DSL.using()", rather than "new Executor()".

Now as far as the "anti-pattern" is concerned, I understand what you
mean. You mean that org.jooq.DSL shouldn't do two things at once. But
I disagree with you, here. Unlike Ceylon, for instance, Java doesn't
have package-level (i.e. "top-level") methods, which are global in
scope. Unlike Scala, Java doesn't have implicit methods either. In
Java, every method has to be put in a class and dereferenced
explicitly. My personal preference would be not to need any
org.jooq.DSL type at all, but to put all DSL methods (including
"using()") into the top-level, i.e. into the "org.jooq" package. In
JavaScript, I'd still create a DSL global namespace-object for that.
There has to be some entry-point to the DSL. And that's this one
class.

While, in the context of a "regular API", I'd probably agree with you,
I don't entirely agree with you in the context of an internal DSL. As
a matter of fact, I don't even think that there are best practices or
"patterns" vs. "anti-patterns" in modern internal DSLs. I'm not aware
of many Java tools out there, that actually go as far as jOOQ.

So the driving force here is the learning curve, which is less steep,
if "using()" and "other types" of top-level DSL methods are all put
into the "org.jooq.DSL" type.

> Now, I can see all the examples are going to be rewritten like:
>
> ----------------------
> using(configuration)
> .selectFrom(BOOK)
> .where(PUBLISHED_IN.equal(2011))
> .orderBy(TITLE)
> ----------------------
>
> ,which means: get the configuration from somewhere, which is breaking, at
> least, the principle of least knowledge. IMHO this is one of the most
> important and also most ignored principle in programing in general.

Then, as I said, the principle was broken before. Before, we could've written:
----------------------
new Factory(configuration)
.selectFrom(BOOK)
.where(PUBLISHED_IN.equal(2011))
.orderBy(TITLE)
----------------------

This is exactly the same. It's even worse as the DSL is broken through
the non-DSL "new" keyword.

> Also, I do believe this is polluting the DSL, because there is no equivalent
> of the first statement in SQL.

In your own examples, you mentioned the use of CONNECT. I don't think
that "using()" is too far away from that.

> And last, but not least, when writing:
> import static org.jooq.DSL.*
> we do import all the goodness of JOOQ plus the "using" evil stuff. One
> cannot import everything BUT the "using" stuff. This is of course the minor
> problem. We can just ignore that method, but the "damage" caused by
> encouraging developers to use it, as the "official" documentation says, as
> the DSL API says, is the thing I am trying to avoid.

Then, those rules will have to make it into your own "best practices".
Many users use jOOQ as a scripting / batch DSL for SQL. A nice example
can be seen here:
https://github.com/jOOQ/jOOQ/issues/2372

"using()" may come in handy in such situations. Or not, as users might
still assign the DSLContext instance to a "create" reference. It's up
to you.

So thanks again, for clarifying your thoughts. The conclusion here is
that the "anti-pattern" of putting "too much" into org.jooq.DSL stands
against the "anti-pattern" of distributing DSL entry points across
various API elements.

Or to put things positively, I chose a gentle learning curve over a
clean separation of concerns in the case of DSL entry points. This
conclusion will make it into the tutorial and manual.

Cheers
Lukas!

[Witold Szczerba]

Hello again,
I can see your point of view. I don't agree, but I do understand driving force behind your decision.

Thanks,
Witold Szczerba

[Lukas Eder]

2013/4/7 Witold Szczerba <pljos...@gmail.com>:

> Hello again,
> I can see your point of view. I don't agree, but I do understand driving
> force behind your decision.

At least we agree to disagree :-)

Cheers
Lukas

[Christopher Deckers]

Witold, Lukas,

> Where
> to get the "create" from? It doesn't matter.

Oh yes it does. If the same object instance is supplied to several calls, this could result in awful issues in case of multi-threading or usage of fetchLazy.

The only safe way to execute an SQL call in an application without worrying of concurrent uses is to call "using()". Only the Configuration is guaranteed (if not by implementation, by contract) to be a safe object to use across an application.

 

You mean that org.jooq.DSL shouldn't do two things at once.

I agree with Witold here, as you know. For me, JOOQ maps SQL to Java and the DSL aspect of JOOQ is what makes it possible. But the using() keyword is breaking this mapping. This is why I said there should be a DSL that maps the SQL call, and a DSL to map the rendering call. This addition of "using" to the SQL mapping DSL is something for the lazy, which rots the API, confuses users with regards to intent, etc. I will stop here before I am going to repeat myself too much, and I also understand the difference in point of view :)

Lukas, would it be possible to add a method for those who want to make that distinction? I am looking for a DSL like:

    load(configuration).bind(Query).execute()

where:
- "load()" could be in "DSL" (since "using" is there...)
- would return an interface that only exposes bind() which implementation can be shared (only configuration is stored, which is global/sharable).

- bind() returns an interface with only rendering capabilties (which DSLContext can implement, and which DSLContextImpl can actually be the underlying object that is returned) and which is a contextual object (not to be shared).

Keep up the good work! :)

Cheers,

-Christopher

[Christopher Deckers]

Hi Lukas,

I wish to exclude the DSLContext / Executor from these thoughts. It is
complicating things unnecessarily. It has no state / lifecycle of its
own. Introducing it in the way I originally did was confusing. The
thought of having "new Executor" along with the name "Executor" seemed
to indicate that it has a life of its own. It doesn't. It had a 1-to-1
relationship with Configuration, adding DSL capabilities.

Configuration does not expose methods with Connection. DSLContext creation does, which means it is not safe to use because it may have a state (especially if the part of the application a developer is working on is independant from the part of the API that creates the DSLContext).

JOOQ could garantee this by having only one "using" method:
    using(Configuration).

For quick runs of queries that do not need listeners (and other Configuration stuff), you could have something like:

    new UnsafeConfiguration(Connection, Dialect)

which lazy users could use:
    using(new UnsafeConfiguration(Connection, Dialect))

This would clearly show the intent, warn the users, etc. They would know that sharing the resulting Configuration may be dangerous (and its Javadoc would document why it is unsafe, because of the way it provides connections).

 

As I will
explain further down, everything you want to do can be done without
allowing "Executor" to have its own lifecycle.

It can, but it is not guaranteed. When 20-50 developers work on the same business application, with many modules and so on, you can only rely on the hard rules set by libraries. If a rule is not clear or has several different meaning, it is dangerous, especially if that library is used throughout the whole application.

If I were the only developer on the applications I work on, I wouldn't care :)

I agree. Although, I think that the ConnectionProvider *should* be
loose to cover all use cases.

By being loose, you prevent use cases (like 3rd party tools taking a Configuration as a parameter to use connections for their own needs, a 3rd party execute listener that needs a dedicated connection, etc.).
Please, show me a use case where being loose helps compared to having the hard rule of "acquire" always returning a dedicated connection?

I know you want to be loose on every aspects, but that results in a fuzzy API where multi-developers project cannot make assumptions (or wrong ones). These unspecified aspects then have to be handled through company policies, provided the issues are identified, and may be occasionaly overlooked (blowing up in production of course).

Yes, it is hard to get it right. Do note that this was much harder to
get right before jOOQ 3.0, before the introduction of a
ConnectionProvider. In order to get it right, a properly pooled (and
thus shareable) DataSource was needed.

I am not sure how connections taken from a DataSource are supposed to be returned to their pool. Any experience on real applications using DataSource to explain the lifecycle?

 

B) jOOQ can provide default implementations for the
ExecuteListenerFactory, e.g. a DefaultExecuteListenerFactory, which
would contain a constant list of ExecuteListeners.

Let's take the example of the DebuggerListener. How should I code it? The whole point of the factory is so that DebuggerListener is a new instance per run and thus can retain state. How would JOOQ instanciate it if it were to automatically use such kind of stateful listener but wanted a static list of listeners?

 

I don't think that we need the additional complexity by adding
the possibility of creating "per-Executor" setups.

By-the-way, I agree to that, I just mentioned it to cover all cases from global to specific.

 
 

If you feel that there is still a missing piece as I might not have
gotten the full picture yet (e.g. contracts too loose, additional
lifecycle for DSLContext), feel free to continue this discussion.

I wonder whether we understand each other, because you say:

- Loose contracts allow more flexibility.

- Hard contracts make them hard to implement.

which is in opposition with what I say:

- Hard contracts do not restrict flexibility.

- Hard contracts remove ambiguity: any tool/developer knows how to use/implement without mistakes.

- Loose contract lead to API misuse, bugs and potential bureaucracy.

Hope this helps,

-Christopher

[Witold Szczerba]

On 7 April 2013 17:18, Christopher Deckers <chr...@gmail.com> wrote:
>
> Witold, Lukas,
>
>> > Where
>> > to get the "create" from? It doesn't matter.
>
>
> Oh yes it does. If the same object instance is supplied to several calls, this could result in awful issues in case of multi-threading or usage of fetchLazy.
> The only safe way to execute an SQL call in an application without worrying of concurrent uses is to call "using()". Only the Configuration is guaranteed (if not by implementation, by contract) to be a safe object to use across an application.

I promised not to mention about this anymore, but I have to deny what
was said above.
Christopher, this is not true what you say about "the only safe
way...". The solution you mention, this is the "using(configuration)"
is flawed. I am not going to continue about this anymore, as
everything was said.

There are at least few (maybe a dozen) ways to guarantee, across
entire application, that Executor will be used safely without
spreading the configuration across entire application, forcing users
to call some factory to exchange config for Executor. In my most
recent application, using EJB/CDI container, everything works
correctly "out of the box". Previous one used plain Tomcat without any
extra containers, and yet it was also enough to keep the details of
creating Executor (called FactoryOperations in 2.6) away from it's
customers.

Regards,
Witold Szczerba

[Durchholz, Joachim]

> Sharing a configuration object across entire application just
> to be able to get an Executor in return, is an anti-pattern,
> it is evil and every time when someone suggest to do so, an
> innocent kitten dies.

Well, you do have context objects like that all over the place:
- JDBC connections
- Hibernate (or other ORM) sessions
- File handles
- Network connections

Yes, they are evil. One of the main reasons for the existence of frameworks like Spring is the injection of such "resource" objects.

However, they are unavoidable for the following purposes:
1) To allow replacing the context underneath a computation. E.g. run the same JDBC on top of different databases.
2) To avoid recreating the context for every operation.

For Jooq, the litmus test is:
Would it be useful to have a variant of Executor that takes its parameters not from a configuration object but from a set of parameters?
How "wide" would that interface be? ("wide" = "many" parameters, or parameters with "complex interrelationships".)
Would you want to use overloading on Executor constructors? Would overloading run into uniqueness issues?

In the latter case, you need a Configuration construction kit anyway; the best remedy would be to make it do very little processing, postponing all the heavy lifting to the point in time when the query is actually executed.
However, this approach also delays any error checking until execution time, which is unhelpful for finding bug causes.

Designing such interfaces is a trade-off between a LOT of conflicting goals, and sometimes it's the choice between killing kittens and seal babies; the art of design is to find a compromise where you kill neither (but maybe somewhere else a hyena baby starves - no matter what you do, some trade-offs simply do not have an answer that can't be questioned).

Regards,
Jo

[Lukas Eder]

I feel there is still some confusion left about the purpose of the
DSLContext (which was called the Executor in jOOQ 3.0-RC2).

DSLContext is that part of the jOOQ DSL that needs / wraps a
Configuration to operate. The important type we're discussing about
here is Configuration, not DSLContext. DSLContext just "adapts"
Configuration and offers more DSL methods.

Some have argued that DSLContext should really be extracted in a
separate, independent DSL or API, which I have declared as out of
scope for jOOQ 3.0 (without denying that it might be a good idea in a
longer run).

But apart from wrapping the Configuration, there is no purpose /
lifecycle whatsoever related to a DSLContext. In that way, nothing has
changed between jOOQ 2.x and 3.x, except for a single DSL method
"using()" and the naming: from 2.x's FactoryOperations (which was
implemented by Factory) to 3.0's DSLContext (which is implemented
privately).

Cheers
Lukas

PS: Christopher, I hope I'll find time soon to reply to your e-mail!

2013/4/8 Durchholz, Joachim <Joachim....@hennig-fahrzeugteile.de>:

[Lukas Eder]

Hi there,

2013/4/7 Christopher Deckers <chr...@gmail.com>:

> Hi Lukas,
>
>> I wish to exclude the DSLContext / Executor from these thoughts. It is
>> complicating things unnecessarily. It has no state / lifecycle of its
>> own. Introducing it in the way I originally did was confusing. The
>> thought of having "new Executor" along with the name "Executor" seemed
>> to indicate that it has a life of its own. It doesn't. It had a 1-to-1
>> relationship with Configuration, adding DSL capabilities.
>
>
> Configuration does not expose methods with Connection. DSLContext creation
> does, which means it is not safe to use because it may have a state
> (especially if the part of the application a developer is working on is
> independant from the part of the API that creates the DSLContext).
>
> JOOQ could garantee this by having only one "using" method:
> using(Configuration).
>
> For quick runs of queries that do not need listeners (and other
> Configuration stuff), you could have something like:
> new UnsafeConfiguration(Connection, Dialect)
> which lazy users could use:
> using(new UnsafeConfiguration(Connection, Dialect))
>
> This would clearly show the intent, warn the users, etc. They would know
> that sharing the resulting Configuration may be dangerous (and its Javadoc
> would document why it is unsafe, because of the way it provides
> connections).

The "UnsafeConfiguration" is really called DefaultConfiguration, and
it exists :-)
And I'm making it public for reuse.

But the contract on "using(Connection, SQLDialect)" is clear. It is
convenience for "using(new DefaultConfiguration(Connection,
SQLDialect))". I bet you will try to argue against this convenience.
jOOQ always overloads methods for tedious repetitive tasks. It has
been like this before, and I'd like to keep it this way.

>> As I will
>> explain further down, everything you want to do can be done without
>> allowing "Executor" to have its own lifecycle.
>
>
> It can, but it is not guaranteed. When 20-50 developers work on the same
> business application, with many modules and so on, you can only rely on the
> hard rules set by libraries. If a rule is not clear or has several different
> meaning, it is dangerous, especially if that library is used throughout the
> whole application.
>
> If I were the only developer on the applications I work on, I wouldn't care
> :)

OK, then let's learn from the "best" and I'll reiterate one of my
examples. Let's have a look at javax.sql.DataSource. Its contract
clearly specifies:

1. Basic implementation -- produces a standard Connection object
2. Connection pooling implementation -- produces a Connection object
that will automatically participate in connection pooling. This
implementation works with a middle-tier connection pooling manager.
3. Distributed transaction implementation -- produces a Connection
object that may be used for distributed transactions and almost always
participates in connection pooling. This implementation works with a
middle-tier transaction manager and almost always with a connection
pooling manager.

Now, this is a definite invitation to shoot yourself in the foot, from
an API perspective. You can simply not know what happens with the
returned Connection, except if you read docs. Still, I've worked in
teams of 20-50 developers, and some architect has solved this problem
about 10 years ago, and ever since, no one cared where the connection
really came from. It came from some static method calling Weblogic's
JNDI mechanisms.

The same applies to a jOOQ Connection. Only 1-2 guys from the 20-50
developers will actually try to understand how to use an
org.jooq.Configuration. All the others will operate on a
pre-configured DSLContext object, that is obtained through a static
method, through dependency injection, you name it.

>> I agree. Although, I think that the ConnectionProvider *should* be
>> loose to cover all use cases.
>
>
> By being loose, you prevent use cases (like 3rd party tools taking a
> Configuration as a parameter to use connections for their own needs, a 3rd
> party execute listener that needs a dedicated connection, etc.).
> Please, show me a use case where being loose helps compared to having the
> hard rule of "acquire" always returning a dedicated connection?

- A batch job. You always want precisely the same connection for every query.
- Within a distributed javax.transaction.UserTransaction. Your
ConnectionProvider doesn't even know whether it should return the same
connection, or a new one. The container manages that.

Hardening the contract will make it more difficult to use jOOQ at all.
It makes it simpler for 3rd party tools that want to acquire
connections through jOOQ's API (let's name them the jOOQ Console). But
should they really do that in the first place? If the console needs a
dedicated connection, then it should provide a Popup for user input.
You can't just expect to get a dedicated connection from all
installations of jOOQ, no matter what the connection / transaction
model is, and think they will be able to give you one. Such a use-case
shouldn't drive the design decisions of a jOOQ core
ConnectionProvider.

Or to name your example of the Runnable and the Swing context:

I won't add Configuration.getDedicatedConnectionForJOOQConsole() :-)

Please tell me, if I'm missing the point here, and if you know a
concrete use-case of a 3rd party tool that really relies on jOOQ
providing it with a "dedicated" connection through the
ConnectionProvider API.

> I know you want to be loose on every aspects, but that results in a fuzzy
> API where multi-developers project cannot make assumptions (or wrong ones).
> These unspecified aspects then have to be handled through company policies,
> provided the issues are identified, and may be occasionaly overlooked
> (blowing up in production of course).

Yes. And No! Loose doesn't mean fuzzy. Loose just means that jOOQ
defines a minimal contracts on its SPI for jOOQ to work.

So the architect / lead-developer implements that once, and then
everyone gets back to business. Any tight contract I will define now
will be thrown back at me because I hadn't thought of this and that
and yet another way to handle connections, transactions, etc.

>> Yes, it is hard to get it right. Do note that this was much harder to
>> get right before jOOQ 3.0, before the introduction of a
>> ConnectionProvider. In order to get it right, a properly pooled (and
>> thus shareable) DataSource was needed.
>
> I am not sure how connections taken from a DataSource are supposed to be
> returned to their pool. Any experience on real applications using DataSource
> to explain the lifecycle?

Yes. Weblogic, JTA environments (I quote that all the time). The
lifecycle is this:

Within the DAO:
============

1. I get a DataSource via JNDI. That's a lot of magic to me already.
2. I get a Connection from that DataSource. It "magically" knows in
which JTA transaction I am in, right now.
3. Connection.close(). This puts it in a "special" state, whose
documentation I hadn't found on Weblogic
4. I can get another Connection from the DataSource. "Magically", it
will be the same as before, with that "special" state reset.
5. I close it again.

Within the Service:
==============

1. I get a UserTransaction via JNDI. More magic.
2. I call some DAOs. See above
3. I call tx.commit() or tx.rollback(). From now on, the DAOs don't
work anymore, as they cannot get a DataSource via JNDI. But if I
didn't close all my connections before, I get exceptions here.

The above is really easy to use from a user perspective, although I
don't want to look at Weblogic's internals :-)

Anyway, this leads to a real dumb ConnectionProvider implementation
that just delegates everything to Weblogic. Again, here in this gist
by Ben Manes, you have the same "dumb" ConnectionProvider
implementation that "just doesn't care". Spring already got everything
right:
https://gist.github.com/ben-manes/5316351

So, "loose" is the way to go for ConnectionProvider, in my opinion.
Loose doesn't mean fuzzy. The contract is crystal clear. jOOQ will
release() every connection that was acquired(). Clients know their
transaction model better than jOOQ. They will know how to properly
implement "acquire()".

>> B) jOOQ can provide default implementations for the
>> ExecuteListenerFactory, e.g. a DefaultExecuteListenerFactory, which
>> would contain a constant list of ExecuteListeners.
>
>
> Let's take the example of the DebuggerListener. How should I code it? The
> whole point of the factory is so that DebuggerListener is a new instance per
> run and thus can retain state. How would JOOQ instanciate it if it were to
> automatically use such kind of stateful listener but wanted a static list of
> listeners?

Yes, I got that wrong in my first understanding of
ExecuteListenerProvider. You're right about this. So let's re-iterate:

- Configuration.executeListenerProviders() returns a list of
ExecuteListenerProvider
- ExecuteListenerProvider implementations are free to choose whether
they return new instances at every call
- There are convenience methods to register "global" ExecuteListeners,
which are wrapped in a DefaultExecuteListenerProvider

Does that wrap it up? I'll update the code on GitHub ASAP...

>> I don't think that we need the additional complexity by adding
>> the possibility of creating "per-Executor" setups.
>
>
> By-the-way, I agree to that, I just mentioned it to cover all cases from
> global to specific.

Finally one less open topic! :-)

>> If you feel that there is still a missing piece as I might not have
>> gotten the full picture yet (e.g. contracts too loose, additional
>> lifecycle for DSLContext), feel free to continue this discussion.
>
>
> I wonder whether we understand each other, because you say:
> - Loose contracts allow more flexibility.
> - Hard contracts make them hard to implement.
> which is in opposition with what I say:
> - Hard contracts do not restrict flexibility.
> - Hard contracts remove ambiguity: any tool/developer knows how to
> use/implement without mistakes.
> - Loose contract lead to API misuse, bugs and potential bureaucracy.

I don't agree with all of this, as you know :-)
Besides, jOOQ really doesn't need any harder contract on the
ConnectionProvider. The jOOQ Console may need it, but I won't consider
the Console as a driving force for core contracts.

[Christopher Deckers]

Hi Lukas,

concrete use-case of a 3rd party tool that really relies on jOOQ
providing it with a "dedicated" connection through the
ConnectionProvider API.

Let's go back to the origins. In our application (the simple version being a fat client deployed on all of our clients' users' computers with direct database access), the contract of getting a connection through our connection providing API is hard: you must release what you acquire and it is reserved. All our integrated tools like our console can get connections from that central place. At any time, even over a remote session with a client when troubleshooting an issue, we can fire the console to log and/or execute statements.

This saved us on many occasions, both at development time as well as in production. My goal was to try to ease this firing of the console in the context of JOOQ too.

Let's sum up:

1. Is ConnectionProvider a generic way of acquiring connections: No.
2. Does JOOQ provide a centralized/generic way of acquiring dedicated connections: No.

Conclusion: it is unclear whether tools have to create their own mechanics to get dedicated connections. Depending on the type of ConnectionProvider, this may or may not be suitable, but using connections when it is not suitable could result in awful corruptions.

I originally thought that with little API tweaks maybe users could create a Configuration/ConnectionProvider per target and pass them to JOOQ DSL as well as tools. Too bad! :)

Cheers,

-Christopher

[Lukas Eder]

Hi Christopher,

2013/4/12 Christopher Deckers <chr...@gmail.com>:

> Hi Lukas,
>
>> concrete use-case of a 3rd party tool that really relies on jOOQ
>> providing it with a "dedicated" connection through the
>> ConnectionProvider API.
>
>
> Let's go back to the origins. In our application (the simple version being a
> fat client deployed on all of our clients' users' computers with direct
> database access), the contract of getting a connection through our
> connection providing API is hard: you must release what you acquire and it
> is reserved. All our integrated tools like our console can get connections
> from that central place. At any time, even over a remote session with a
> client when troubleshooting an issue, we can fire the console to log and/or
> execute statements.

True, I remember you telling me this. It is a nice architecture for a
client application.

> This saved us on many occasions, both at development time as well as in
> production. My goal was to try to ease this firing of the console in the
> context of JOOQ too.
>
> Let's sum up:
> 1. Is ConnectionProvider a generic way of acquiring connections: No.
> 2. Does JOOQ provide a centralized/generic way of acquiring dedicated
> connections: No.

Correct. And:

3. Is any of this needed? No. ConnectionProvider is an SPI exposed by
jOOQ to inject behaviour into jOOQ. It is not meant to be reused in a
more general context. You may already have a very specific way of
providing connections in your application, true. But then, just
implement ConnectionProvider in addition to that, as an adapter to
your mechanism.

In other words, there are already excellent connection pooling
libraries out there. Here's where jOOQ's mission stops.

> Conclusion: it is unclear whether tools have to create their own mechanics
> to get dedicated connections. Depending on the type of ConnectionProvider,
> this may or may not be suitable, but using connections when it is not
> suitable could result in awful corruptions.

Yes it could. That's why the architect evaluating jOOQ is worth his money. :-)

> I originally thought that with little API tweaks maybe users could create a
> Configuration/ConnectionProvider per target and pass them to JOOQ DSL as
> well as tools. Too bad! :)

They still can. But jOOQ is just one target / actor in this
environment. And it shouldn't be the "leading" actor. It is a consumer
of the main architecture...

Cheers
Lukas

[Lukas Eder]

2013/4/12 Lukas Eder <lukas...@gmail.com>:

This is now implemented on Github master and contained in the latest
3.0.0-SNAPSHOT. I'll release 3.0.0-RC3 today.

Last-minute feedback before RC3 is welcome ;-)
If not, I think the current ExecuteListenerProvider will more or less
match your expectations. It is indeed a good solution to allow for
injecting stateless / stateful ExecuteListeners into jOOQ without jOOQ
noticing.

[Christopher Deckers]

Hi Lukas,

 

> I bet you will try to argue against this convenience.
> jOOQ always overloads methods for tedious repetitive tasks.

For me, it was important to understand what the ConnectionProvider is and what it is not. All the rest, including this explicit distinction between safe and unsafe was or was not relevant depending on that.

 
 

> Anyway, this leads to a real dumb ConnectionProvider implementation
> that just delegates everything to Weblogic.

ConnectionProvider is an adapter to whatever mechanism the user prefers, needed for JOOQ to handle Connections, DataSources or whatever custom mechanisms behind a minimalistic contract. It is not meant to be a reliable factory that tools can use (which is the aspect I wanted to clear).

 

> - There are convenience methods to register "global" ExecuteListeners,
> which are wrapped in a DefaultExecuteListenerProvider

DebugListener will remain stateless by retaining data in the context. The reason is that users can easily configure a DebugListener but not a Provider, if I were to use that mechanism instead. Anyway, the contract on the data map placed in the context is now much better than before (per execution that is).

 

Last-minute feedback before RC3 is welcome ;-)

Welcome RC3!

If not, I think the current ExecuteListenerProvider will more or less
match your expectations. It is indeed a good solution to allow for
injecting stateless / stateful ExecuteListeners into jOOQ without jOOQ
noticing.

Yes, for internal use (by JOOQ users that is), it is a good fit. For a generic listener that is meant to be distributed to users, better rely on the data map to retain states.

> I don't agree with all of this, as you know :-)

It is only a matter of context. What you say is true considering the behavior is managed externally to the library. The question I was raising was whether it should be somewhat managed to allow for additional tools to work (which was what I believed ConnectionProvider was doing the first time I saw it).

Keep up the good work!

Cheers,

-Christopher

[Lukas Eder]

Hi Christopher,

And thanks for the feedback!

2013/4/12 Christopher Deckers <chr...@gmail.com>:

> Hi Lukas,
>
>>
>> > I bet you will try to argue against this convenience.
>> > jOOQ always overloads methods for tedious repetitive tasks.
>
>
> For me, it was important to understand what the ConnectionProvider is and
> what it is not. All the rest, including this explicit distinction between
> safe and unsafe was or was not relevant depending on that.
>
>
>>
>> > Anyway, this leads to a real dumb ConnectionProvider implementation
>> > that just delegates everything to Weblogic.
>
>
> ConnectionProvider is an adapter to whatever mechanism the user prefers,
> needed for JOOQ to handle Connections, DataSources or whatever custom
> mechanisms behind a minimalistic contract. It is not meant to be a reliable
> factory that tools can use (which is the aspect I wanted to clear).

So, do you think there is room for more clarification in the Javadoc / manual?

>> > - There are convenience methods to register "global" ExecuteListeners,
>> > which are wrapped in a DefaultExecuteListenerProvider
>
>
> DebugListener will remain stateless by retaining data in the context. The
> reason is that users can easily configure a DebugListener but not a
> Provider, if I were to use that mechanism instead. Anyway, the contract on
> the data map placed in the context is now much better than before (per
> execution that is).

Yes, decoupling ExecuteContext from Configuration was a good idea!

Cheers,
Lukas

[Christopher Deckers]

Hi Lukas,

So, do you think there is room for more clarification in the Javadoc / manual?

Nothing in particular comes to mind. The changes you made to RC3 Javadoc of Configuration and ConnectionProvider are perfectly in line with this clarified behavior and the overall API is much better and consistent.

Note: did you perform a mass find/replace or did you really want to have "an DSLContext" in the Javadocs (generally in the form "an {@link DSLContext}")?

Cheers,

-Christopher

[Lukas Eder]

Hi Christopher,

2013/4/12 Christopher Deckers <chr...@gmail.com>:

> Hi Lukas,
>
>> So, do you think there is room for more clarification in the Javadoc /
>> manual?
>
>
> Nothing in particular comes to mind. The changes you made to RC3 Javadoc of
> Configuration and ConnectionProvider are perfectly in line with this
> clarified behavior and the overall API is much better and consistent.

OK, thanks for the feedback.

> Note: did you perform a mass find/replace or did you really want to have "an
> DSLContext" in the Javadocs (generally in the form "an {@link DSLContext}")?

Thanks for the hint. This is fixed in Configuration:
https://github.com/jOOQ/jOOQ/commit/64995c1836b40283db060a4d96c724b9b2c3dada

I found two occurrences of this in Configuration...

Cheers
Lukas