Ruote workers priority / reserving

[Marco Sehrer [ninjaconcept.com]]

Hi,

is there a way to 'reserve' workers for specific participants. 

We have the requirement that some workitems need to be processed quite instantly (1-2 seconds). 

But under medium workload the engine reacts really slow and needs minutes to process new workitems, even with a lot of worker-processes (currently 5). 

If it would be possible to reserve workers for specific jobs, we could ensure the workitems would be processed within time. 

Or is it possible to give an workitem some kind of priority?

How many workers do you usually run in production? 

Thanks,

Marco

[John Mettraux]

On Tue, Jul 31, 2012 at 12:39:42PM +0200, Marco Sehrer [ninjaconcept.com] wrote:
>
> is there a way to 'reserve' workers for specific participants.

Hello,

basically, there are two ways.

a) use Participant#on_accept?
http://ruote.rubyforge.org/implementing_participants.html#accept

b) subclass the Worker to make it discard certain msgs based on the
participant name

> We have the requirement that some workitems need to be processed quite instantly (1-2 seconds).
> But under medium workload the engine reacts really slow and needs minutes to process new workitems, even with a lot of worker-processes (currently 5).

What is medium workload for you guys?
What storage implementation do you use?
What version of ruote?
What version of Ruby?
Is the datastore on the same host as the workers?
Are the workers disseminated on multiple hosts?
What kind of deployment? EC2? Own servers?

I'm now in the middle of an effort to optimize ruote-sequel (
https://groups.google.com/d/topic/openwferu-users/ZFfqxAIRgsw/discussion )
maybe you're using ruote-sequel as well.

> If it would be possible to reserve workers for specific jobs, we could ensure the workitems would be processed within time.
> Or is it possible to give an workitem some kind of priority?

Sorry, there is no priority for workitems in ruote. The two techniques above
could be used for prioritizing, but the regular work (outside of participant
"execution") has to be done as well.

At first, I can help you make it faster (if I know the details).

> How many workers do you usually run in production?

In production for now I personnaly only have limited systems with two workers
and it's more like for backup than for load. One or two new flows per day, a
bit of activity every 1 or 2 hours. Small office automation.

I don't know for others, maybe they'll notice the thread and pass some info.


Best regards,

--
John Mettraux - http://lambda.io/jmettraux

[Marco Sehrer [ninjaconcept.com]]

HI John,

is there a way to 'reserve' workers for specific participants.

Hello,

basically, there are two ways.

a) use Participant#on_accept?
http://ruote.rubyforge.org/implementing_participants.html#accept

hm, I guess this would not work, because we need to reserve workers for participant, not participants for workitems (?)

b) subclass the Worker to make it discard certain msgs based on the
participant name

thanks for this tip,

looks as a working solution, I will give it a try.

maybe this is something for a future ruote update?

We have the requirement that some workitems need to be processed quite instantly (1-2 seconds).

But under medium workload the engine reacts really slow and needs minutes to process new workitems, even with a lot of worker-processes (currently 5).

What is medium workload for you guys?

about 20-50 workitems per minute ..

What storage implementation do you use?

currently we use the Redis storage, btw. ruote-mon works so far for us, but its a bit slower then redis, so currently ruote-redis

What version of ruote?

2.3.0

What version of Ruby?

1.9.3

Is the datastore on the same host as the workers?

yes, all on one machine .. 

Are the workers disseminated on multiple hosts?

no 

What kind of deployment? EC2? Own servers?

own server - latest ubuntu - 4 CPUs XEON 8GB RAM .. 

but performance feels almost the same as on my macbook pro 

I'm now in the middle of an effort to optimize ruote-sequel (
https://groups.google.com/d/topic/openwferu-users/ZFfqxAIRgsw/discussion )
maybe you're using ruote-sequel as well.

do you think ruote-sequel will better perform then redis ?

btw. what is ruote's workitem dispatch frequency? each 0.1 seconds? each second?  

If it would be possible to reserve workers for specific jobs, we could ensure the workitems would be processed within time.

Or is it possible to give an workitem some kind of priority?

Sorry, there is no priority for workitems in ruote. The two techniques above
could be used for prioritizing, but the regular work (outside of participant
"execution") has to be done as well.

sure, I will give the worker-subclass a try 

At first, I can help you make it faster (if I know the details).

How many workers do you usually run in production?

In production for now I personnaly only have limited systems with two workers
and it's more like for backup than for load. One or two new flows per day, a
bit of activity every 1 or 2 hours. Small office automation.

sounds far less as for what we use it .. .:-/

we have ruote-jobs once a day which will span up to thausends of workitems .. but they have no high prio. 

then we have a regular workload of about 1 item per second, sometimes more .. 

for the later I would like to reserve one or more ruote workers ..

thanks a lot for your help so far

I don't know for others, maybe they'll notice the thread and pass some info.

yes that would be interesting 

ciao, 

Marco 

Best regards,

--
John Mettraux - http://lambda.io/jmettraux

--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to openwfe...@googlegroups.com
to unsubscribe : send email to openwferu-use...@googlegroups.com
more options : http://groups.google.com/group/openwferu-users?hl=en

Schöne Grüße Marco

-- 

NinjaConcept GmbH
Marco Sehrer Geschäftsführung
Amalienstrasse. 44
76133 Karlsruhe
fon:	(+49) 0721 1803523-1
fax:	(+49) 721 961402-99
mobile:	(+49) 151 20314416
email:	m...@ninjaconcept.com
www:	http://www.ninjaconcept.com/

[Mario Camou]

Hi John, Marco and all,

One idea. Would it be an option to have two distinct engines? Perhaps one for the batch jobs and one for the regular jobs, and use the OS process prioritization to give more resources to the regular one?

Just an idea...

[Marco S]

Hi John, 

b) subclass the Worker to make it discard certain msgs based on the
participant name

how would this be implemented? 

in process(msg) ? 

how to savely discard the message?

Thank you!

[northox]

You can simply reply without modifying the workitem if it does not meet certain requirements. 

Sent from Mobile

--

[Marco S]

I tried this, but this will just to ignore the messages, no other ruote-worker-process will pick up the work for this messages.

module Ruote

  class MyWorker < Worker

    def process(msg)

      participant_name = msg['workitem']['participant_name'] rescue nil

      unless participant_name =~ /high_priority/

        return false

      else

        puts "Ruote::MyWorker.process(#{participant_name})"

        super msg

      end

    end

  end

end

Am Dienstag, 31. Juli 2012 12:39:42 UTC+2 schrieb Marco S:

[John Mettraux]

On Tue, Jul 31, 2012 at 04:59:50PM +0200, Mario Camou wrote:
>
> One idea. Would it be an option to have two distinct engines? Perhaps one
> for the batch jobs and one for the regular jobs, and use the OS process
> prioritization to give more resources to the regular one?
> Just an idea...

Hello Mario,

I think it's a good idea. I don't know how Marco's user access workitems,
but a consolidated view (high + low priority workitems) could be possible.

Makes sense. Thanks for sharing your idea,

[John Mettraux]

On Tue, Jul 31, 2012 at 11:12:06AM -0700, Marco S wrote:
>
> I tried this, but this will just to ignore the messages, no other
> ruote-worker-process will pick up the work for this messages.
>
> module Ruote
> class MyWorker < Worker
> def process(msg)
> participant_name = msg['workitem']['participant_name'] rescue nil
> unless participant_name =~ /high_priority/
> return false
> else
> puts "Ruote::MyWorker.process(#{participant_name})"
> super msg
> end
> end
> end
> end

Hello Marco,

instead of "return false", you could do

@context.storage.put_msg(msg['action'], msg)
return true

OK, I've been thinking about your situation and tried a small experiment:

https://github.com/jmettraux/ruote-redis/tree/experiment_pipe_pop

It's pipelining 28 rpops. I found (via https://github.com/jmettraux/ruote-redis/blob/experiment_pipe_pop/test/bm/loaded.rb ) that it performs faster than the current (mea culpa) single pop.

I haven't tried with multiple workers.

Would you have time to test that (in your staging env somehow)?

[Marco Sehrer [ninjaconcept.com]]

>
> Hello Marco,
>
> instead of "return false", you could do
>
> @context.storage.put_msg(msg['action'], msg)
> return true
>

Hi John,

now the worker does its job and only for the specified participants, great :-)

> OK, I've been thinking about your situation and tried a small experiment:
>
> https://github.com/jmettraux/ruote-redis/tree/experiment_pipe_pop
>
> It's pipelining 28 rpops. I found (via https://github.com/jmettraux/ruote-redis/blob/experiment_pipe_pop/test/bm/loaded.rb ) that it performs faster than the current (mea culpa) single pop.
>
> I haven't tried with multiple workers.
>
> Would you have time to test that (in your staging env somehow)?
>

great, I gave it a try, our test-suite runs green. and as far as I can see, it works correctly with multiple workers too.

I took a manual benchmark with 30 workitems running thru a workflow:

2 workers - 1:45 ruote-redis
2 workers - 1:37 ruote-redis-experimental

But this is not very representative as we have some time-consuming jobs going on, I think the pure ruote-speedup must be far more.
Would be cool, to have this optimization in master soon.

Thanks a lot!
Marco

> --
> John Mettraux - http://lambda.io/jmettraux
>

[Marco Sehrer [ninjaconcept.com]]

HI Mario, John and all :-)

thanks for your idea! 

It was something I initially thought about, but I fear its hard to separate the work to distinct engines at the moment, because we items would need to cross-communicate between the two engines.

But, right, with a separate engine it would probably  work :-)

@John

What would be awesome, is a preferred dispatching of some messages. 

I'm not sure, but currently if there are some other time-consuming jobs (participant A  and participant  B)  before my 'important' job (participant C), I would have to wait for that workitem to get processed, regardless if I would use a  special ruote-worker only accepting this kind of workitems in participant C. 

Workitems are always dispatched in order they come in unless some special process-scheduling expression is involved, right ?

What about a 'priority' feature? 

process do 

  prepare_someting

  do_something_unimporarant

  cursor prio:1 do

    super_important_job

  end

end

The idea is this:

There may be jobs which are not really important at the beginning, but part of an workflow, it does not matter how fast they get processed. Then later, as soon an workitem reaches a certain 'milestone', it should be processed as fast as possible.

Would that be possible to implement?

Thanks,

Marco

[John Mettraux]

On Wed, Aug 01, 2012 at 08:11:46PM +0200, Marco Sehrer [ninjaconcept.com] wrote:
>
> great, I gave it a try, our test-suite runs green. and as far as I can see, it works correctly with multiple workers too.
>
> I took a manual benchmark with 30 workitems running thru a workflow:
>
> 2 workers - 1:45 ruote-redis
> 2 workers - 1:37 ruote-redis-experimental
>
> But this is not very representative as we have some time-consuming jobs going on, I think the pure ruote-speedup must be far more.
> Would be cool, to have this optimization in master soon.

Hello Marco,

I merged it into master. The number of pop attempts is configurable via a
'pop_count' option (defaults to 28).

Maybe the next way to improve would be to have pool of threads for the
participants so that one participant doesn't monopolize the whole worker. The
number of threads could be limited (so we don't run out of database
connections like you described previously). Just tell me if you'd be
interested by such a refinement. I cannot do it immediately, I have to work
on ruote-sequel for Chad.

Cheers,

[John Mettraux]

On Wed, Aug 01, 2012 at 08:54:38PM +0200, Marco Sehrer [ninjaconcept.com] wrote:
>
> What about a 'priority' feature?
>
> process do
> prepare_someting
> do_something_unimporarant
> cursor prio:1 do
> super_important_job
> end
> end
>
> The idea is this:
> There may be jobs which are not really important at the beginning, but part of an workflow, it does not matter how fast they get processed. Then later, as soon an workitem reaches a certain 'milestone', it should be processed as fast as possible.
>
> Would that be possible to implement?

Hello Marco,

yes, this would be possible somehow. The way I imagine it is:

concurrence do
alice :priority => 1
bob :priority => 2
charly :priority => 3 # default priority maybe
end

"dispatch" messages with priority 1 or 2 would be processed before any other
message ("launch", "apply", ...) Priority 1 would come before priority 2.

This would be a participant only attribute.

A variant on this, one that would be closer to your sketch, would be

concurrence do
sequence :priority => 1 do
# ...
end
sequence :priority => 2 do
# ...
end
end

where "apply" messages are prioritized so sections of workflow would get
priority.

...

Before that I'd prefer to try and implement a thread pool (as described in
https://groups.google.com/d/msg/openwferu-users/xSDFdqp4wws/eXS7OnyEIaUJ ).
I have the impression that the thread pool might speed up things, you seem to
have lots of IO bound participants.

Wdyt?

[Marco Sehrer [ninjaconcept.com]]

Am 02.08.2012 um 05:56 schrieb John Mettraux:

On Wed, Aug 01, 2012 at 08:11:46PM +0200, Marco Sehrer [ninjaconcept.com] wrote:

great, I gave it a try, our test-suite runs green. and as far as I can see, it works correctly with multiple workers too.

I took a manual benchmark with 30 workitems running thru a workflow:

2 workers - 1:45 ruote-redis

2 workers - 1:37 ruote-redis-experimental

But this is not very representative as we have some time-consuming jobs going on, I think the pure ruote-speedup must be far more.

Would be cool, to have this optimization in master soon.

Hello Marco,

I merged it into master. The number of pop attempts is configurable via a
'pop_count' option (defaults to 28).

cool cool :-)

Maybe the next way to improve would be to have pool of threads for the
participants so that one participant doesn't monopolize the whole worker. The
number of threads could be limited (so we don't run out of database
connections like you described previously). Just tell me if you'd be
interested by such a refinement. I cannot do it immediately, I have to work
on ruote-sequel for Chad.

That would be awesome, any improvement concerning faster message dispatching would be great  :-)

btw. this was in my mind when I talked about the :priority feature, to let some independent high-prio workitems  get dispatched faster, but I see this should not be part of the DSL, I guess.

Thank you very much for the great support!

Marco

Cheers,

--
John Mettraux - http://lambda.io/jmettraux

--
you received this message because you are subscribed to the "ruote users" group.
to post : send email to openwfe...@googlegroups.com
to unsubscribe : send email to openwferu-use...@googlegroups.com
more options : http://groups.google.com/group/openwferu-users?hl=en