priority queue with mongodb

[bizz]

Hi,

I want to use mongodb to create jobs priority queue.
I created a collection containing all the jobs, each job has priority
and minimum time for the next execution.
Now, I want to query for bulk of jobs to execute. (I use the java
client to do so)
I use the find method to query, sort and limit the jobs I want, and in
return I get a cursor.

My problem:
After I get the results I would like to mark the jobs as being
processed. I could come up with the next options:
1. go over all the results (one by one with the cursor) and collect
all the ids, then use the update method to mark the jobs.
2. move all the jobs to a new collection dedicated to the processing
client
3. Not use the find method, and instead use FindAndModify for the
number of jobs I want, one by one

The options listed above have bad performance, because I have to go
over the jobs one by one to collect their ids so I could mark them as
being executed.

If I had one of the following features I would have been able to
perform this task:
1. FindAndModify on bulk of entries
2. Update with limit (so I could update using the original query)
3. An option to get all the results from the find and not a cursor

I couldn't think of any other way that I could do this task with
better performance,
Anyone have an Idea?

Thanks

[Michael Dirolf]

I think the best bet will probably be to use findAndModify. The
update/limit that you suggest would be subject to race conditions,
etc. Some sort of bulk findAndModify would be interesting but for now
I would just do multiple calls.

> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com.
> To unsubscribe from this group, send email to mongodb-user...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>

[bizz]

I tried that but the performance is very bad, comparing to other
options.

On Aug 2, 6:29 pm, Michael Dirolf <m...@10gen.com> wrote:
> I think the best bet will probably be to use findAndModify. The
> update/limit that you suggest would be subject to race conditions,
> etc. Some sort of bulk findAndModify would be interesting but for now
> I would just do multiple calls.
>

[Dustin Norlander]

findAndModify is what we use for a priority queue. We are able to
process > 1000 tasks per second on a single mongo instance using this
approach. Just make sure your indexing and querying strategy is
optimal.

[Jonathan Gaillard]

Our two libs provide priority support:

https://github.com/dominionenterprises/mongo-queue-php

https://github.com/dominionenterprises/mongo-queue-csharp