Processing queues in isolation

[Adrian Florian]

We currently have a testing server for running scrapers. Let's say that they run in 'scraper_jobs'. 

Multiple users using this system to testrun a scraper, so let's say that Joe wants to test run the scraper for 'Site 1', then Sidekiq would start and enqueue some jobs in 'scraper_jobs' for that site. At the same time Molly is also test running 'Site 2', still Sidekiq running in 'scraper_jobs'. 

There are multiple problems with this:

- Molly would have to wait until Joe's queue is processed

- if Molly wants to pause her queue, because they start to fail, she will pause Joe's queue as well.

Is there a way that the queues would be processed independently depending on who started them?

[Mike Perham]

For each person, use:

1. a different Redis instance

2. a different database on the same Redis instance

3. a different namespace on the same Redis database

Personally I'd do (2), you can have up to 16 databases by default with Redis so up to 16 testers.

Mike

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To post to this group, send email to sid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/7c48ff80-8a6b-4aaf-b4c3-2551423691eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.