Overriding array size limit

[Clint Combs]

Hi all,

I'm running up against the arraySizeLimit setting in the Java msgpack implementation.

  org.msgpack.unpacker.SizeLimitException: Size of array (5039767) over limit at 4194304

That value is set in AbstractUnpacker.java:

  https://github.com/msgpack/msgpack-java/blob/a319d80406843c761d7aaeeb2c4c883246240b0e/src/main/java/org/msgpack/unpacker/AbstractUnpacker.java#L32

... and can be updated with a call to setArraySizeLimit(int size) in the same class:

  https://github.com/msgpack/msgpack-java/blob/a319d80406843c761d7aaeeb2c4c883246240b0e/src/main/java/org/msgpack/unpacker/AbstractUnpacker.java#L122

While that seems straightforward, I'm also using msgpack-scala and the array read that blows things up is here:

  https://github.com/msgpack/msgpack-scala/blob/master/src/main/scala/org/msgpack/ScalaMessagePack.scala#L97

Is there a preferred way to overrride the arraySizeLimit parameter with configuration, a call to msgpack-scala, or a direct call to the underlying msgpack-java?

Thanks,

Clint

[Clint Combs]

My current project involves reading large (> 150 GB) msgpack format files containing very large arrays. I'm temporarily working around the array size limit problem by building my own version of msgpack-java and setting the rawSizeLimit, arraySizeLimit, and mapSizeLimit values to Integer.MAX_VALUE:

  https://github.com/msgpack/msgpack-java/blob/a319d80406843c761d7aaeeb2c4c883246240b0e/src/main/java/org/msgpack/unpacker/AbstractUnpacker.java#L30-L34

I'd like to implement a way to set these values at run-time that would be usable from the msgpack-scala layer. Is anyone looking at a good way of doing this?

Thanks,

Clint Combs