Stream processing workers

[Alex SV]

Hi everybody,

Disco workers are map-reduce workers, but inside the disco is a distributed task execution system.

Is it possible to make long-living workers, which process some streaming data (for example events from message queue), which are monitored and controlled by the master, and which could run general map-reduce jobs periodically?

Regards

Alex

[Prashanth Mundkur]

Yes, this has been thought about, and the worker protocol was designed
to support this case, but it needs some support in the disco master.

https://github.com/discoproject/disco/issues/67

--prashanth

[Alex SV]

Just curious, will it go live sometime? :)