Cursor Timeout

[Michael Xavier]

I have a long running process that is wrapped in a Product.all.each do
{|product| ...} but the problem is that after about an hour, my cursor
times out. I cannot find documentation for overriding this timeout.
I'd prefer to just not have a timeout globally but if I need to limit
it to a particular query, I'll need a bit of guidance on how to do it
the right way.

So far I've tried Product.all(:timeout => false).each but that gives
me the error:

ArgumentError: Timeout can be set to false only when #find is invoked
with a block.

So then I tried Product.all(:timeout => false).each {|cursor|
cursor.each {|product| puts product.id}}.execute, that raises the same
find error. I just really need an example of the least roundabout way
to actually execute a query without timeout.

[venetanji]

Have you considered using something like delayed_job or resque to deal
with the long running processes?

removing the timeout might solve the problem for now, but it's not a
good idea to have the actual task invoked inside a loop.

G

[Michael Xavier]

It's a scheduled job and in my opinion, reasonable. The problem is
that its a task that has to get run daily for this app to even
function and its been down for the past week because I cannot trick
mongoid into recognizing that I've supplied a block even though I
have. I can't really see a reason why the cursor *should* timeout by
some arbitrary constant in this case and the feature is supposed to be
supported, I have just been unable to actually take advantage of that
feature.

[Kyle Banker]

Check out the code example here:

http://www.mongodb.org/display/DOCS/Frequently+Asked+Questions+-+Ruby#FrequentlyAskedQuestions-Ruby-IkeepgettingCURSORNOTFOUNDexceptions.What%27shappening%3F

[Michael Xavier]

Something in mongoids' implementation of Collection prevents that
example from working and detecting blocks properly.

Product.collection.find({}, :timeout => false) {|cursor| cursor.each {|
product| puts product.id}}

ArgumentError: Timeout can be set to false only when #find is invoked
with a block.

from /home/michael/.rvm/gems/ruby-1.8.7-p249@metrics/gems/
mongo-1.0.7/lib/../lib/mongo/collection.rb:149:in `find'
from /home/michael/.rvm/gems/ruby-1.8.7-p249@metrics/bundler/
gems/mongoid-dab07d8/lib/mongoid/collections/master.rb:15:in `send'
from /home/michael/.rvm/gems/ruby-1.8.7-p249@metrics/bundler/
gems/mongoid-dab07d8/lib/mongoid/collections/master.rb:15:in `find'
from /home/michael/.rvm/gems/ruby-1.8.7-p249@metrics/bundler/
gems/mongoid-dab07d8/lib/mongoid/collection.rb:50:in `find'
from (irb):8

It does not make sense as there is very clearly a block associated
with the find.

On Aug 27, 11:25 am, Kyle Banker <k...@10gen.com> wrote:
> Check out the code example here:http://www.mongodb.org/display/DOCS/Frequently+Asked+Questions+-+Ruby...

[venetanji]

I'm using resque with mongoid and the way i do this is by having the
perform method behave in 2 ways:

http://gist.github.com/554911

Without arguments, i loop through all the records and schedule one job
per record
With id, i do the actual task, in my case an upsert (no need for
another find if you pass the whole record when you queue the job
above, I just pass fan_count cos that's what i need :-)

This way you can process your records concurrently and jobs get to
fail without breaking the loop. Maybe for just an upsert this isn't
worth it. I mainly use this method to sync data with facebook, a
process that can have a lot of exceptions and can take a long time.

Hope this helps, even though it doesn't solve the original problem...

[Michael Xavier]

Hi, thanks for your response. That's an interesting strategy but I
think using a separate job queuing system is overkill for the task I'm
trying to perform. I don't really even care how long the task I'm
performing takes, I just need it done. The point, I think, is that
mongoid should support operations that are going to take a long time.
So far since any timeout option I've been able to find raises the same
block exception, I've had to drop down to the driver level.
Unfortunately, the cursor *still* times out. My code looks along the
lines of:

def recalculate_stats!
Mongoid.database[collection.name].find({}, :timeout => false) do
|cursor|
cursor.each do |row|
Product.new(row).do_stuff
end
end
end

Obviously this route is pretty ugly and unmaintainable, but it also
does not seem to do any good for the timeout issue. I'd honestly be ok
with just disabling all timeouts in the system. I've dug through the
documentation and hacked away as best I could with no success. I can't
imagine I'm the first mongoid user who has wanted to do something
without cursor timeouts. Any help in that direction that you guys
could provide would be greatly appreciated.