messages assigned to busy consumer when other consumers are idle

[Rohit Hira]

User submits N jobs to my C# application, each job has different priority and the job priority can be changed in real time by the user through web UI. All these jobs are long-running jobs and must be executed in the priority order (highest to lowest).

I have 10 consumers on 4 different machines (separate console instances of my application running from different folders), each consumer has separate connection and channel and prefetch = 1, using push based BasicConsume approach.

Here is my solution for this problem -

Step 1. App checks if any consumer is idle or not (I used database to store this data)

Step 2. If at least one consumer is idle, then select the first job in priority order and publish message for this job id in the queue.

Step 3. any idle consumer should receive this message and process the job (mark itself as busy in database).

Step 4. if all consumers are busy, then wait for 15 seconds and then go to step 1.

Problems with Rabbitmq -

1. Consumers are not able to receive the message immediately after publishing it. 

I experimented with 30 seconds delay for step 4 as well, but same problem exists.

2. Messages are assigned to busy consumers while idle consumers are available

Since consumers are not picking up messages immediately, I end up publishing more messages than no. of idle consumers available and ultimately it leads to a scenario where jobs are executed out of order and only a few consumers are busy and others are idle, but many jobs are still waiting to be executed.

Should I switch to pull based basicGet method approach? Is it better suited to my use case?

Is there any better way to track the current status of consumers?

Any other approach I should try ?

Many Thanks 

Rohit