Size Limitations

[Albert Azout]

Are there size limitations on values? What are best practices when
creating binary value types.

[Eishay Smith]

I'm using large or even very large binary objects (protobuf) in few of my stores, no problems.
If you use large objects you might want to tune the socketBufferSize to be a larger then your common object to reduce the networking overhead (the default now is 64k, see voldemort.client.ClientConfig.


Eishay

On Sun, Jan 24, 2010 at 10:44 AM, Albert Azout <aaz...@gmail.com> wrote:

Are there size limitations on values? What are best practices when
creating binary value types.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[yufeldman]

Hello here,

I am also interested in the tuning you guys did for large objects. How
large are your objects and what socketBufferSize do you use?

Thanks

On Jan 24, 11:53 am, Eishay Smith <eis...@gmail.com> wrote:
> I'm using large or even very large binary objects (protobuf) in few of my
> stores, no problems.
> If you use large objects you might want to tune the socketBufferSize to be a
> larger then your common object to reduce the networking overhead (the
> default now is 64k, see voldemort.client.ClientConfig.
>
> Eishay
>
> On Sun, Jan 24, 2010 at 10:44 AM, Albert Azout <aaz...@gmail.com> wrote:
> > Are there size limitations on values? What are best practices when
> > creating binary value types.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "project-voldemort" group.
> > To post to this group, send email to project-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > project-voldem...@googlegroups.com<project-voldemort%2Bunsu...@googlegroups.com>

[Rob Adams]

Main caveat is the memory requirements and the fact that the data is copied and not streamed, which is a bit slower for large objects.  So it's just not going to work all that well for really huge objects in the hundreds of megabyte range, especially if you've got a heavy load.

[yufeldman]

We do not have huge binary objects, but for example getting 5 MB (and
max we can have is about 25MB) is taking 10-12 seconds.
So just was wondering if increase of socket buffer could help
somewhat.

Thanks,

[Jay Kreps]

You should really run some profiling to see where this is going (i.e.
run with -hprof options). The normal limits of harddrives and networks
apply, in that if you have a 50MB object and your disk writes at
50MB/sec max then you are not going to see more than 1 req/sec.
However we have not done *any* profiling with objects that large
(large for us is 300k). If the problem is writing to disk then you
need more disks, but if the time is spent in voldemort code somewhere
we will definitely treat it as a bug and try to improve it.

-Jay

> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.