Can someone give me an example of ApplyQuantile use?

24 views
Skip to first unread message

Ryan Michael

unread,
Sep 17, 2012, 8:58:07 PM9/17/12
to dat...@googlegroups.com
I've been hitting my head against a wall for a good 4 hours trying to get ApplyQuantile to work.  Can someone give me the most basic working example possible?  All I need is to take an input stream, determine the quantiles for a value in that stream, and then replace that value with the quantile.  Here's what I have so far:

register 'java/include/datafu-0.0.5.jar';
define Quantile datafu.pig.stats.StreamingQuantile('101');
define ApplyQuantile datafu.pig.stats.ApplyQuantile('101');

prefs = LOAD '$input' USING PigStorage() AS (user_id:chararray, item_id:chararray, preference:float);
grouped_prefs = GROUP prefs ALL;
quantiles = FOREACH grouped_prefs GENERATE Quantile(prefs.preference) as value;
processed = FOREACH prefs GENERATE ApplyQuantile(prefs.preference, quantiles.value);

STORE processed INTO '$output';

Generating quantiles works fine, but as soon as I try to use them I start getting errors.  This code throws "ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Expected a tuple", but I've tried about 200 variations and none of them seem to work either.

What am I missing? :(
-Ryan 
Reply all
Reply to author
Forward
0 new messages