How to clear all tasks in a queue if you don't know args?

[Joseph Purcell]

I see there is a method for deleting a task queue:

api.tasks.del(queue, taskName, args, count, next)

However, this is unusable if you don't know the arguments you passed. I took a look through resque and I see that it is storing queues in the Redis "list" data type with each value as a JSON encoded string:

LRANGE resque:queue:default 0 999
*3
$123
{"class":"crawl","queue":"default","args":{"url":"http://example.com/","crawler_id":"5325fb2db61bde94f77b142c"}}
$127
{"class":"crawl","queue":"default","args":{"url":"http://example.com/post","crawler_id":"5325fb2db61bde94f77b142c"}}
$130
{"class":"crawl","queue":"default","args":{"url":"http://example.com/article","crawler_id":"5325fb2db61bde94f77b142c"}}

I have discovered I can delete the entire "default" queue by talking to resque directly, but obviously that will delete all task queues. So, my question is, is there a way to clear a specific task queue? (Perhaps I'm using resque in the wrong way altogether, and I would be open to suggests there too.)

[Joseph Purcell]

On second thought, it looks like I will have to re-think this a bit because I don't just want to empty the queue for "crawl" tasks in the "default" queue. I want to empty the queue for "crawl" tasks in the "default" queue that match a specific "crawler_id".

That aside, my question remains though now out of curiosity than necessity.

[Evan Tahler]

Deleting a queue in resque is as simple as deleting the list, IE `del resque:queue:default` in your example.  Keep in mind that this might cause problems for any worked currently working on a job from that list

--
 Evan Tahler
 412.897.6361
 evant...@gmail.com

--
You received this message because you are subscribed to the Google Groups "actionHero.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionhero-j...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joseph Purcell]

 Yes, that makes sense.

However, is there a way to delete the queue for a specific task? For example, if I enqueue three tasks:

          api.tasks.enqueue('crawl', {url:url, crawler_id:crawler_id}, 'default');
          api.tasks.enqueue('crawl', {url:url, crawler_id:crawler_id}, 'default');
          api.tasks.enqueue('crawl', {url:url, crawler_id:crawler_id}, 'default');

And then hit a route /api/stop_crawler?crawler_id=1, I want to be able to delete the above three enqueued tasks. For example, if the following code could be run, that would be ideal for what I'm trying to do:

          api.tasks.dequeue('crawl', {crawler_id:crawler_id}, 'default'); // delete all enqueued tasks that match crawler_id:crawler_id

But, I really don't think there is a way to do this. For one, Resque uses a LIST data type in Redis, which has only one option for deleting, LREM, which requires that you know the entire value that you inserted into the list. And two, if I just run `del resque:queue:default` directly to Redis it will delete any other tasks that have been sent to the "default" queue, regardless of what crawler_id it is. In other words, this would halt ALL crawlers.

So, I think instead of using the Action Hero's task API for enqueuing tasks to keep track of my queue for URLs to crawl, I will create a new queue for each crawler using the Resque API directly. Then, I should be able to run: `del resque:que:<crawler_id>` where the crawler_id is the crawler I want to stop.

[Evan Tahler]

Reseque may not be the tool for you if you plan on removing jobs (which isn’t a normal pattern in resque).

A few patterns come to mind:

- queue-per-crawler (then you can delete the queue, as you mention)

- single queue, but look up if the URL is “active” in a database (which means stale jobs would just be ignored and eventually drained) 

- use another tool which gives you access to modify the object while in the enqueued state (literally a row in a database might be better).

Up to you! 

--
 Evan Tahler
 412.897.6361
 evant...@gmail.com

From: Joseph Purcell josephd...@gmail.com
Reply: Joseph Purcell josephd...@gmail.com

[Joseph Purcell]

Thank you, those are excellent options. I will refactor using one of those suggestions.