how to specify num.chunks

[Sean McNamara]

i am running into Chunk overflow exceptions.  I am passing hadoop-build-readonly-store.sh -D num.chunks=1000 ... to get around the 2GB chunk size limit.

However, when it runs it always says the num chunks is set to 1:

13/08/20 16:15:34 INFO mr.HadoopStoreBuilder: Number of chunks: 1, number of reducers: 50, save keys: false, reducerPerBucket: false

Am I not specifying it properly, or is it being overridden somewhere?   I am using voldemort 1.3.4

Thanks,

Sean

[Chinmay Soman]

Its calling this class HadoopStoreJobRunner

./contrib/hadoop-store-builder/src/java/voldemort/store/readonly/mr/HadoopStoreJobRunner.java

It suggest using         'chunksize'

parser.accepts("chunksize", "maximum size of a chunk in bytes.").withRequiredArg();

FYI: We dont use this class to do our builds. We use the VoldemortBuildAndPushJob

[Sean McNamara]

Chinmay-

Thank!  I see now, we set our chunk size to be *much* smaller than what we were using and it creates more num.chunks.  The downside is that it creates many more reducer tasks. --reducer-per-bucket should fixed that but it causes the JVM to OOM.  So I will continue to dive in.  We almost have this thing dialed in.

Sean