Scalding: How to tune blockJoing

[Mansoor Ashraf]

I am doing a self join between a fairly large and skewed dataset and running into situations where two reducers are getting pegged with most of the data. On the other hand if i run with skew join, the data is more widely distributed but you have mentioned on a couple of occasion that skewJoin is not reliable. Can you provide some pointer on how to tune the ratios so that block join is more distributed. Currently I have the ratio set to 1 and 1 for both left and right and I have 2 reducers running for 12 hours while all other were finished in less than an hour.

Thanks

 left.blockJoinWithSmaller('subType -> 'subType1, right,1,1)