Multiples Queues

[Ryan Smith]

I propose that Queue Classic support them. There are a few things that
need to be flushed out before we start work, I would love feedback
regarding these issues.

* How to store the queues in Postgres

My first thought is to create a 1-1 mapping between PG tables and
queues. I think this will make queries to the queue more readable.
i.e. select count(*) from email_jobs; -vs- select count(*) from jobs
where queue_name = 'email_jobs';

I am sure there are some points of discussion regarding performance on
this topic.

* Syntax

I had a great discussion with James Edward Gray II regarding this
feature and he suggested the following:

Remove the singleton pattern from the QC::Queue class.
Provide class methods on the Queue class that will use a default jobs
table.
Instantiate an object of the Queue class to add another queue to the
system.

The syntax might look like this:

QC::Queue.enqueue "Class.method", arg1, ..., argN #=> Uses default
jobs table

@queue = QC::Queue.new(:email_jobs) #=> Uses user defined jobs table
@queue.enqueue "Class.method", arg1,...,argN

* Next Steps

Refactor the Queue class and remove the singleton pattern.
Remove references to the jobs table in DurableArray and Database
classes.

I am going to sketch out what this might look like and push to a
branch on github. Ill use multiple_queues as the branch name.

[JEG2]

On Apr 23, 6:31 am, Ryan Smith <this.ryansm...@gmail.com> wrote:
> I propose that Queue Classic support them.

I vote: heck yeah! :)

>
> * How to store the queues in Postgres
>
> My first thought is to create a 1-1 mapping between PG tables and
> queues.

Agreed. It keeps like data together.

> * Syntax
>
> I had a great discussion with James Edward Gray II regarding this
> feature and he suggested the following:
>
> Remove the singleton pattern from the QC::Queue class.
> Provide class methods on the Queue class that will use a default jobs
> table.
> Instantiate an object of the Queue class to add another queue to the
> system.
>
> The syntax might look like this:
>
> QC::Queue.enqueue "Class.method", arg1, ..., argN  #=> Uses default
> jobs table
>
> @queue = QC::Queue.new(:email_jobs)  #=> Uses user defined jobs table
> @queue.enqueue "Class.method", arg1,...,argN

I like this and I'm sure it has nothing to do with it being my
idea! :)

> * Next Steps
>
> Refactor the Queue class and remove the singleton pattern.
> Remove references to the jobs table in DurableArray and Database
> classes.
>
> I am going to sketch out what this might look like and push to a
> branch on github. Ill use multiple_queues as the branch name.

Can you also push the signal handling code we worked on when it's
convenient? I just wanted to thumb through the project as I saw it
that night.

James Edward Gray II

[Ryan Smith]

James, I believe master on github has the code we were working with. In fact, here is our commit: https://github.com/ryandotsmith/queue_classic/commit/c71d0201e6d6f0e2b0f54e5082af41426b8aa0f8
I made a few doc-related commits after this one, so master should be what you remember.

Also, I got a fair bit accomplished on the plane home from RDRC2 regarding multiple queues. I had to do a fair bit of refactoring.... Some notable changes:

1) No more QC.enqueue QC.dequeue. We have to use QC::Queue.enqueue and QC::Queue.dequeue (etc...)
2) If you add a new queue, you must rake create_queue[queue_name] first. This creates the table and loads the locking function.
3) There will be a locking function for each queue.

Regarding #3

I fear that it might have violated the DRY principal. But, the locking function has references to the table everywhere. Perhaps I need to look at the plpgsql reference to see how I can pass in a parameter telling the function which table to work on. The more I think about it, there should definitely be a way to do something like this :

SELECT * FROM lock_head_on("jobs");
or
SELECT * FROM lock_head_on("priority_invoice_jobs"); etc...