Equijoin output not written to specified output folder

[Anand Jay]

Hi,

I have been trying to write the output of equijoin to a specific hdfs output folder. However the output of the first of the 3 map reduce processes involved is written to this location while the final output is written to a temp folder.

The following simple example will recreate this: 

folder <- "hdfs://localhost:8020/user/jayanak/test"

eq.out <- equijoin(left.input = to.dfs(keyval(1:10, (1:10)^2)), right.input = to.dfs(keyval(1:10, (1:10)^3)),outer="left",output=folder)  

as.data.frame(from.dfs(eq.out)$val) # gives correct (final) output 

as.data.frame(from.dfs(folder)$val)  # gives the output of the first map-reduce process

I would appreciate any suggestions on how I can overcome this problem. I am not sure whether I have missed something simple. 

Thanks in advance.

Anand