Urgent How to Increase the REDUCERS performance in Pig

[Krishna]

Hi Group,

Can any one tell me how to improve the performance of Reducers in PIG SCRIPT, I am using 

SET default_parallel 40; so No.of Reducers is 40 running parallel but its tack more then 2 hours to complete my 10m data.

Can any one Tell me how can i Improve my Reducers performance, any query's  its great help.

 Regards.

[Senthil Kumar]

Hi Krishna,

Why do you set number of reducers to 40 ?Is there any need for it?

You need to set number of reducers based upon the need. 

Can you tell me how much each reducer is taking?? it will pinpoint where is the bottleneck.

Thanks

Senthil Kumar A

https://www.linkedin.com/in/senthilka

[Aravind]

 Adding few more to Senthil's comment, 

  Also it depends on the cluster size ... What is the size of ur cluster ? How many reducers that it has on a whole ?

  How much time on an average each reducer takes ? You can also monitor the CPU , memory utilization of reducers and the cluster overall so see any resource contention happening. 

Thanks & Regards

Aravind

--
You received this message because you are subscribed to the Google Groups "Hadoop Users Group (HUG) Chennai" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chennaihug+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Aravindakumar.V
mobile : +1 856 952 3632