1MB limit on object size

[Sakuntala]

Hi,

I am using
memcached server version 1.4.4 on Linux and
spy memcached client version 2.4.2

I performed a test by storing and retrieving objects of size 1.8MB and
3.6MB. I am able to retrieve the objects successfully without any
error. My load tests show some timeout exceptions when memcached
server is under heavy load, which is expected.

Memcached wiki website states that one of the reasons we should not be
using memcached is if the object size exceeds 1MB. I want to know if
this is a hard limit, or just that memcached is not efficient in that
scenario due to the storage model.

The reason for my question is that I have just one object(which is a
list) which might exceed 1MB in future, but currently, it is not
exceeding the limit. I considered storing it in several chunks and
using bulkGet() to retrieve the multiple small objects. The response
times are not efficient in case of bulkGet(). I should not be using
memcached for this object( and should look into other caching
mechanisms) but dont want to change the implementation till next
release. Please suggest.

Thanks,

[Peter J. Holzer]

On 2010-10-14 11:08:59 -0700, Sakuntala wrote:
> I am using
> memcached server version 1.4.4 on Linux and
> spy memcached client version 2.4.2
>
> I performed a test by storing and retrieving objects of size 1.8MB and
> 3.6MB. I am able to retrieve the objects successfully without any
> error. My load tests show some timeout exceptions when memcached
> server is under heavy load, which is expected.
>
> Memcached wiki website states that one of the reasons we should not be
> using memcached is if the object size exceeds 1MB. I want to know if
> this is a hard limit, or just that memcached is not efficient in that
> scenario due to the storage model.

Yes, it's a hard limit (but you can recompile memcached if you want a
higher limit). But most client libraries can compress the
objects before storing it. If you have enabled compression in spy
memcached client (which might be the default) it is possible that you
can store items much larger than 1MB if they compress well.

hp

--
_ | Peter J. Holzer | Openmoko has already embedded
|_|_) | Sysadmin WSR | voting system.
| | | h...@hjp.at | Named "If you want it -- write it"
__/ | http://www.hjp.at/ | -- Ilja O. on comm...@lists.openmoko.org

[Dustin]

On Oct 15, 9:56 am, "Peter J. Holzer" <h...@hjp.at> wrote:
> Yes, it's a hard limit (but you can recompile memcached if you want a
> higher limit). But most client libraries can compress the
> objects before storing it. If you have enabled compression in spy
> memcached client (which might be the default) it is possible that you
> can store items much larger than 1MB if they compress well.

If you don't want to recompile it, you can just use -I at runtime.


-I Override the size of each slab page. Adjusts max item
size
(default: 1mb, min: 1k, max: 128m)

[Sakuntala]

Peter and Dustin,

Thanks a ton for your replies. The info did help me.

-Sakuntala.

[Abhilash]

Hi,

I have same problem, but starting memcached with -I is causing a
different issue. Can you please help?

Memcached server version: memcached-1.4.5 on Linux 64Bit
spy memcached client version: memcached-2.5

I have two Memcached Servers and they are started with following
command

sudo memcached -d -m 10240 -u root -I 5120 -u root -l [host] -p 11311

I am using 10 instances of the memcached client.

Error Log:

2010-11-10 16:42:17.929 ERROR
net.spy.memcached.protocol.binary.StoreOperationImpl: Error: Too
large.
2010-11-10 16:42:17.931 INFO net.spy.memcached.MemcachedConnection:
Reconnection due to exception handling a memcached operation on {QA
sa=/10.60.291.81:11211, #Rops=1, #Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@54a9a793,
topWop=null, toWrite=0, interested=1}. This may be due to an
authentication failure.
OperationException: SERVER: Too large.
at
net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:
123)
at
net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:
147)
at
net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:
133)
at
net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:
401)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
333)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
219)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1591)
2010-11-10 16:42:17.932 WARN net.spy.memcached.MemcachedConnection:
Closing, and reopening {QA sa=/10.60.291.81:11211, #Rops=1, #Wops=0,
#iq=0,
topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@54a9a793,
topWop=null, toWrite=0, interested=1}, attempt 0.
2010-11-10 16:42:17.933 WARN
net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding
partially completed op:
net.spy.memcached.protocol.binary.StoreOperationImpl@54a9a793
2010-11-10 16:42:17.974 ERROR
net.spy.memcached.protocol.binary.StoreOperationImpl: Error: Too
large.
2010-11-10 16:42:17.983 INFO net.spy.memcached.MemcachedConnection:
Reconnection due to exception handling a memcached operation on {QA
sa=/10.70.181.63:11311, #Rops=1, #Wops=1, #iq=0,
topRop=net.spy.memcached.protocol.binary.StoreOperationImpl@7a36ae66,
topWop=net.spy.memcached.protocol.binary.StoreOperationImpl@7a36ae66,
toWrite=1368, interested=5}. This may be due to an authentication
failure.
OperationException: SERVER: Too large.


Thanks
Abhilash

[Dustin]

On Nov 10, 2:51 pm, Abhilash <abhih...@gmail.com> wrote:

> Memcached server version:  memcached-1.4.5 on Linux 64Bit
> spy memcached client version:  memcached-2.5

> sudo memcached -d -m 10240 -u root -I 5120 -u root -l [host] -p 11311

You've really gone out of your way to run it as root. Why is it so
important to you to do that? I think it's a rather awful idea.

> net.spy.memcached.protocol.binary.StoreOperationImpl:  Error:  Too
> large.

Is the total size of your key and value larger than 5,120 bytes?

[Abhilash]

Hi Dustin, thanks for the quick response. Please see my comments
below.

>You've really gone out of your way to run it as root. Why is it so
>important to you to do that? I think it's a rather awful idea.

Can you please suggest the best way to run memcached, if running it as
root a bad idea.
Correcting my command (sudo memcached -d -m 10240 -u root -I 5120 -u
root -l [host] -p 11311) would help me better understand.

>Is the total size of your key and value larger than 5,120 bytes?

I meant to run it with 5MB, so I have corrected it by changing it from
5120 to 5242880.
It gave me a warning:
#WARNING: Setting item max size above 1MB is not recommended!
#Raising this limit increases the minimum memory requirements, and
will decrease your memory efficiency.

Yes, some of my objects are way too large close to 5 MB!

Even after running memcached with 5MB, I still get the error message -
Caused by: java.lang.IllegalArgumentException: Cannot cache data
larger than 1048576 bytes (1MB) (you tried to cache a 1828343 (1.74
MB) byte object)

Thanks
Abhilash

[Trond Norbye]

On 10. nov. 2010, at 16.21, Abhilash wrote:

> Hi Dustin, thanks for the quick response. Please see my comments
> below.
>
>> You've really gone out of your way to run it as root. Why is it so
>> important to you to do that? I think it's a rather awful idea.
>
> Can you please suggest the best way to run memcached, if running it as
> root a bad idea.
> Correcting my command (sudo memcached -d -m 10240 -u root -I 5120 -u
> root -l [host] -p 11311) would help me better understand.
>

You want to run your processes with as little privileges as possible. Root is the "almighty" user with full access to everything on your system, so you don't want your processes to run as that user unless you have to. If someone finds a bug in memcached, they may compromise more of your system if the process got root privileges.

Memcached tries to warn you about running memcached as root by require you to use the -u option if it detect that it is running as root. What you want to do is to run memcached as an unprivileged user. You might want to create a unique user for memcached, or you can use one of the users your operating system probably had (like noaccess/nobody etc)

sudo memcached -d -m 10240 -u noaccess -I 5120 -l [host] -p 11311

Cheers,

Trond

[Dustin]

On Nov 10, 4:21 pm, Abhilash <abhih...@gmail.com> wrote:
> I meant to run it with 5MB, so I have corrected it by changing it from
> 5120 to 5242880.

Ah, then use -I 5mb

> Even after running memcached with 5MB, I still get the error message -
> Caused by: java.lang.IllegalArgumentException: Cannot cache data
> larger than 1048576 bytes (1MB) (you tried to cache a 1828343 (1.74
> MB) byte object)

The transcoder will prevent you from creating an object that's too
big. You can just instantiate your own instance of
SerializingTranscoder with a larger size and have the connection
factory return yours.

[Abhilash]

Thanks Trond, I changed the way I start memcached, as you suggested.

Can someone please respond to my last point?

>Even after running memcached with 5MB, I still get the error message -
>Caused by: java.lang.IllegalArgumentException: Cannot cache data
>larger than 1048576 bytes (1MB) (you tried to cache a 1828343 (1.74
>MB) byte object)

Why is still erroring if the size is more than 1 MB?

Thanks
Abhilash

[Abhilash]

Thanks Trond, I changed the way I start memcached, as you suggested.

Can someone please respond to my last point?

>Even after running memcached with 5MB, I still get the error message -
>Caused by: java.lang.IllegalArgumentException: Cannot cache data
>larger than 1048576 bytes (1MB) (you tried to cache a 1828343 (1.74
>MB) byte object)

[Alex Miller]

Hi Abhilash,

The Java client performs a size check on requests prior to making the Memcache connection to stop excessive network usage if you accidentally try to store more. You must configure the client to accept larger objects. You may be able to do this in a config file, or it may require recompiling the client. I'm sure someone has that information.

 - Alex

--

Alex Miller
skype | alexmiller.canada
email | alexmill...@gmail.com

[Abhilash]

Thanks All,

So to use the custom Transconder, I am using

public <T> Future<Boolean> set(String key,int exp,T o,Transcoder<T>
tc) of net.spy.memcached.MemcachedClient

and passing 'new SerializingTranscoder(max)' as the Transcoder object,
with the higher limit I need.
instead of
public Future<Boolean> set(String key,int exp,Object o)

This seems to be working. It stored objects with size greater than 1
MB.

However, at some point I am still getting the same error - I can debug
this further
But please respond if you feel that this is not I am supposed to do.

I believe that, this could be because I am storing huge objects
resulting a network latency.

2010-11-10 19:55:11.832 INFO net.spy.memcached.MemcachedConnection:
Reconnecting due to exception on {QA sa=/[host]:11211, #Rops=1,
#Wops=0, #iq=0,
topRop=net.spy.memcached.protocol.binary.GetOperationImpl@5b475b9e,
topWop=null, toWrite=0, interested=1}

java.lang.IllegalArgumentException: Cannot cache data larger than

1048576 bytes (you tried to cache a 2541225 byte object)
at net.spy.memcached.CachedData.<init>(CachedData.java:30)
at net.spy.memcached.MemcachedClient
$4.gotData(MemcachedClient.java:790)
at
net.spy.memcached.protocol.binary.GetOperationImpl.decodePayload(GetOperationImpl.java:
45)
at
net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:
153)

at
net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:
133)
at
net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:
401)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
333)
at
net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:
219)
at net.spy.memcached.MemcachedClient.run(MemcachedClient.java:
1591)

Thanks
Abhilash