Workflow to compute term probability

[Renaud Delbru]

Hi,

What would be the workflow to be able to compute the probability of a
term ?
Given a document collection, I can apply the word count example to get
the list of term with their frequency. But, then I need to divide this
frequency by the number of distinct terms. How could I integrate the
count of distinct terms in the workflow ? And how to use the result to
divide the word count assembly output ?

The basic idea will be first to apply the word count assembly (output:
'term', 'count'), then to apply an aggregator that will count the
number of distinct terms, i.e. number of tuples, and output the
following fields 'term', 'count', 'n', with 'n' the number of distinct
terms in the collection.
The problem is that 'n' is the same for all the tuples. My
understanding is that it is a little bit inefficient and not optimal
(the same value 'n' will be duplicated in every tuple). Is there a way
to store this value in some "pipe context variable", like the context
variable of an aggregator, but with a larger scope i.e. a pipe ?

Thanks,
--
Renaud Delbru

[Chris K Wensel]

If you want to count values reliably across the whole cluster, you
might consider setting up zookeeper and writing a function that
increments a value there.

cheers,
chris

--
Chris K Wensel
ch...@concurrentinc.com
http://www.concurrentinc.com

[Renaud Delbru]

Hi Chris,

I have started to write a tuple counter function based on zookeeper.
In the meantime, I have seen that the Debug operator internally uses a
context variable. Will the context variable be suitable to count the
number of tuples ? Is this not reliable to count value across the
whole cluster ?

Thanks
--
Renaud Delbru

[Chris K Wensel]

The context variable can only hold values for the current process and
the current thread (it is conceivable an operation will be executed
simultaneously by many threads in the future).

if you need shared memory across the cluster, you need something like
hbase or zookeeper, depending on the size of the data shared.

ckw

[Renaud Delbru]

Ok. I see. Zookeeper is needed for this use case. Thanks for the
clarification.

I finished a first version of the CountTuple operation (implemented as
a filter, similarly to the Debug operation). Would you be interested
that I share the code as a contribution ?

--
Renaud Delbru

[Chris K Wensel]

Feel free to put any code you want to share on GitHub, and i will link
to it from the site.

thanks!

chris

[Ilya Haykinson]

You could also conceivably use memcached for shared state -- it has an atomic increment function, and, lacking any disk storage, is very fast. Of course the difference with Zookeeper being that in case of memcached node failure you may get data loss -- but it's likely to be an easier solution to set up.

-ilya

[Renaud Delbru]

Thanks for your suggestion, but I am looking at a solution that use
native Hadoop / HBase cluster deployement.

> course the difference with Zookeeper being that in case of memcached node
> failure you may get data loss -- but it's likely to be an easier solution to
> set up.

Not sure about this, since zookeeper is already setup with newer HBase
versions.

--
Renaud Delbru

[Renaud Delbru]

HI Chris,

As you know, I worked on a zookeeper counter within cascading to count
tuples in a stream.
While the counter works perfectly, i.e. I can increment it using the
TupleCount cascading operator, I encounter difficulties to use the
value of the counter inside a Cascading pipe.

1) I cannot use it inside an ExpressionFunction. The expression is
compiled when creating the cascading flow graph (ElementGraph), and at
this time, the counter is not yet created. I had to create a
DivisionFunction that fetch the counter value in the Function.operate
method (I am not sure if this is very efficient if I have to process
millions of tuples).

2) To be sure that the counter has been fully incremented, I had to
divide my pipe into two flows. The first flow processing my tuples and
counting the number of tuples, and the second flow that use the
zookeeper counter to compute the probability. In this way, I am sure
that when the second flow is executed, the zookeeper counter should be
fully incremented.

This is not very clean, and the cascading flow becomes difficult to
understand. Maybe, you have some ideas on how to improve such data
flow.

Regards
--
Renaud Delbru

[Chris K Wensel]

Two flows seems reasonable. Flows are the unit of work, you are
guaranteed one is complete before the other is begun.

Using a Cascade to join the two Flows into one #complete() call might
be useful (if the second flow uses the output data of the first, not
just the counter).

(am making a note to explore pluggable ways to identify dependencies
between flows, not just by source/sink data).

ckw