Using a queue as a tap

[kay]

Hi,
I want to use cascading as "kind of a streaming" flow and not for
batch processing. Is there a way to use a queue s a tap (both for
source and sink) in cascading? I want to pop an element from the tap,
perform some operations and put the result datum back to the tap.
Also, I want to use it as a distributed process whereby each process
on each hadoop node will read form a centralized queue.
Any suggestion?

[Chris K Wensel]

On Hadoop this would not be possible.

but using Cascading 2.0 using our in memory local mode planner (vs the Hadoop planner) you can do this, but it would not be distributed.

if you need streaming and distribution, you can likely use Storm (or something equivalent) by itself, or build a new Cascading planner that plans on to Storm (or something equivalent) so you don't have to use the Storm apis for processing.

ckw

> --
> You received this message because you are subscribed to the Google Groups "cascading-user" group.
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.
>

--
Chris K Wensel
ch...@concurrentinc.com
http://concurrentinc.com

[kay]

Thanks. Looking into Storm. It fits the bill better for sure.

> > For more options, visit this group athttp://groups.google.com/group/cascading-user?hl=en.