Compute percentile

[Rizwana Rizia]

Hi,

I am very new to PigLatin,

Can anyone tell me if there is any feature in DataFu, that can help calculate percentiles?

For example: Given a set of data, I can tell for each data what is it's percentile.

Example input: 1, 2, 3, 4, 5, 6, 7, ,8 ,9, 10
Output gives you 0th percentile, 10th percentile, 20th percentile, ..........., 100th percentile.

Thanks in advance.
Rizwana Rizia

[Matt Hayes]

Yes check out either Quantile or StreamingQuantile.  

You can define it like this:

define Quantile datafu.pig.stats.Quantile('0.0','0.10','0.20','0.30','0.40','0.50','0.60','0.70','0.80','0.90','1.0');

Or, the shorthand:

define Quantile datafu.pig.stats.Quantile('11')

Note that in the shorthand version it is 11, not 10, because there are 11 values being produced.

[Oscar Wilde]

Hello ; 

I would like to use the output from the Quantile to filter my input, for example how do i get all temperatures (as per the example ) that are within 10% to 90% percentile?

Thanks

[Matthew Hayes]

I gave a talk where I walked through this type of scenario.  Check out the slides here:

http://www.slideshare.net/matthewterencehayes/datafu

Slide 12 is where I apply the filter.  

-Matt

--
You received this message because you are subscribed to the Google Groups "DataFu" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datafu+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.