SHUFFLE TIME in REDUCE phase

[Amit Sangroya]

Hello, 

I want to know how Starfish computes the shuffle time in the reduce phase.

I my experiment, shuffle time is very high. But in Starfish it is less.

I am computing the shuffle time (from job history file) as

SHUFFLE_FINISHED - REDUCE_START_TIME

However, same value is not there from Starfish. Why this is so?

Thanks,

[YoungKun Min]

Hello, Amit

I believe starfish uses performance models in their paper. Did you look at the chapter 6 in this technical paper(http://www.cs.duke.edu/starfish/files/hadoop-models.pdf)? There are too many variables and they are not real implementations, but I believe there are some points of starfish's view.