suggestion: using Adapters to allow different connections

[Yves Senn]

Hey QC folks,

There are many requests to integrate QC better with Active Record. Just assigning the connection

on startup does not really work as it can be closed. I've worked up a simple proof of concept

to separate the connection handling out into adapters.

Code: https://github.com/ryandotsmith/queue_classic/compare/rails_integration...adapters

Currently the AR adapter is very similar to the PG one but we could make more use of AR and provide

a different implementation for example for `transaction` etc.

On my way I noticed that there are no tests verifying the listening functionality.
I wrote https://github.com/ryandotsmith/queue_classic/compare/rails_integration...adapters#L9R125
but somehow the test sometimes results in a deadlock. 

Maybe I messed up the Mutext, let me know if you see anything.

Feedback is appreciated.

Cheers,

-- Yves

[Ryan Smith]

It is still not clear to me how this will solve the Rails related issues. Do we have a clear understanding of what problems we have with active record integration?

I did a quick github issues search and came across the following active record related issues:

* Fallback to ActiveRecord::Base.connection

* Zeus, QC and NOT using DATABASE_URL causes tests to break

* "PG::Error: connection is closed" when enqueuing a job in a connection shared with ActiveRecord

Among these issues I noticed problems when assigning active record's raw connection to QC:Conn.connection. 

One observation here is that this approach doesn't make sense conceptually. E.g. You can assign an active record connection to a queue_classic connection. Instead, I think it makes sense to have the queue_classic modules that require a database connection (QC::Worker, QC::Setup, and QC::Queries) either use the active record connection or the queue_classic connection. I think this is where the idea of Yves' adapter approach comes in quite handy.

If we have an abstract connection that the aforementioned modules can use, then we must be sure that the methods we send to that abstract connection are available and safe. E.g. We probably shouldn't call #disconnect on a connection supplied by active record. Another example is related to LISTEN/NOTIFY. I am not sure that active record has a good interface for interacting with these types of messages.

Another concern I have is related to connection pooling. I am very eager to get a threaded worker included into queue_classic. To do this effectively, we will need the ability to use several connection between threads. My early benchmarks showed that running 10 threads sharing 3 connections offered a 2x throughput increase vereses the single-connection/sing-threaded worker. How can we design our abstract connection to take advantage of Sequel and ActiveRecords connection pooling features?

In terms of a next step, I propose that we consider the changes that Yves has already suggested and I propose that we define what our abstract connection model should look like? Once we have this abstraction in place, I can take a stab at implementing the QC connection for this adapter. Perhaps someone else can take a look at implementing the active record adapter. (Perhaps Yves has already done this.)