Significance of rhipe_map_buff_size

[Sphinx]

Hi,
Can you please tell me what is the significance of
rhipe_map_buff_size?

How does it effect the performance of map reduce job?

Regards,
Sphinx

[Saptarshi Guha]

Yes, it isn't a great design decision. the values and keys are
contained in the R vectors
map.values and map.keys. rhipe_map_buff_size is the length of the of
these vectors.
So if the input format is 'text', this means that map.values will
contain map_buff_size number of lines.

In the airline example, this means that the the map.values will
contain rhipe_map_buff_size number of
data frames (each of which were several thousands of rows) - this is
why in the airline (some of them)
i specify rhipe_map_buff_size a small number because when the text
input was saved as data frames,
the entire map.values (default is 5000 lines of text ) is saved as a data frame.

So if one reads this source, it isn't a good idea to allow the length
of map.values to have the default length -that implis
5000 data frames of 5000 rows each. It's better to set
rhipe_map_buff_size to something like 5-10.

That the user has to think about the size of rhipe_map_buff_size is bad design.

Regards
J

And a happy new year (belated, in advance or on time) to all.

[Sphinx]

Hi Saptarishi,

Belated Happy new year :-).

If rhipe_map_buffer_size = 20 that means for every split that RHIPE
receives will have only 20 map.values,and these 20 map.values will go
to the reducer part and there if we want we need to aggregate the
results if we want. Please let me know if my understanding is right?

Regards,
Sphinx

[Saptarshi Guha]

they go to the reducer if you send them unchanged.
otherwise I did not understand your question.