typebytes_reader error: unsupported type 149

[David Wang]

Platform: HDP 2.2, 100 nodes

I am running a rmr2 job. The mappers did not do real work, just distributing the data to reducers. Sometimes, I met an error "Unsupported type: 149" in the reducer. This error happened just after loading the library because I added a print in the very beginning of the reduce function and could not print anything.

After checking the source code, this error might be caused by the function "typebytes_reader". Then I check the version of R and rmr2 in the cluster and found two versions. 

R - 3.1.0 and 3.1.2

rmr2 - 3.1.0 and 3.3.1

Is this error caused by the inconsistency of the R and rmr2? 

Any thoughts?

David

[Antonio Piccolboni]

The inconsistency in rmr2 will cause this. 149 is one of the extended type codes introduced in later versions.

Antonio

[David Wang]

I am going to ask the system admin to upgrade all packages. Once I have results, will report here. 

[David Wang]

The system admin has updated rmr2 to 3.3.1 and R to 3.1.2. However, the problem still exists. I attached the following sample codes:

	library("rmr2")
	reduce_func <- function(key, vals) {
	write("in reduce func", stderr())
	count = 0
	for (v in vals) {
	status_str = paste("record ", count, " out of ", length(vals), sep="")
	status(status_str)
	increment.counter('mapper_reducer', 'count', 1)
	write(v, stderr())
	}
	return(NULL)
	}
	map_func <- function(., vals) {
	write("in map func", stderr())
	key_list = list();
	val_list = list();
	for (v in vals) {
	increment.counter('mapper_counter', 'count', 1)
	for (i in 1:N_reducers) {
	key_list = c(key_list, i)
	val_list = c(val_list, paste0(as.character(i), "|", v))
	}
	}
	if(length(key_list) > 0) {
	return(keyval(key_list, val_list))
	} else {
	return(NULL)
	}
	}
	inputFile = "/user/myname/test/f.txt"
	outputFile = "/user/myname/test/test_rmr"
	N_reducers = 500
	opt = rmr.options(backend="hadoop",
	backend.parameters=list(hadoop=list(D=paste0("mapreduce.job.reduces=", N_reducers)))
	)
	mapreduce(input=inputFile, input.format="text",
	output=outputFile, output.format=make.output.format("csv", sep="|"),
	map=map_func,
	reduce=reduce_func,
	vectorized.reduce=FALSE)

f.txt is jsut a text file containing 4 lines:

a

b

c

d

If I set N_reducers = 50, there is no problem. 

If I increase N_reducers = 500, I saw both "Unsupported type: 149" and "Error: !is.null(template) is not TRUE". Also a lot of reducers seems not really receive data.

Any ideas?

Thanks,

[David Wang]

Code is 

library("rmr2")
	reduce_func <- function(key, vals) {
	write("in reduce func", stderr())
	count = 0
	for (v in vals) {
	status_str = paste("record ", count, " out of ", length(vals), sep="")
	status(status_str)
	increment.counter('mapper_reducer', 'count', 1)
	write(v, stderr())
	}

if(length(key) > 0) { return(keyval(key, vals)) } else { return(NULL) }

	}
	map_func <- function(., vals) {
	write("in map func", stderr())
	key_list = list();
	val_list = list();
	for (v in vals) {
	increment.counter('mapper_counter', 'count', 1)
	for (i in 1:N_reducers) {
	key_list = c(key_list, i)
	val_list = c(val_list, paste0(as.character(i), "|", v))
	}
	}
	if(length(key_list) > 0) {
	return(keyval(key_list, val_list))
	} else {
	return(NULL)
	}
	}
	inputFile = "/user/myname/test/f.txt"
	outputFile = "/user/myname/test/test_rmr"
	N_reducers = 500
	opt = rmr.options(backend="hadoop",
	backend.parameters=list(hadoop=list(D=paste0("mapreduce.job.reduces=", N_reducers)))
	)
	mapreduce(input=inputFile, input.format="text",
	output=outputFile, output.format=make.output.format("csv", sep="|"),
	map=map_func,
	reduce=reduce_func,
	vectorized.reduce=FALSE)

On Wednesday, March 4, 2015 at 9:39:13 PM UTC-8, David Wang wrote: