Updating memcache frequently?

[mnenchev]

Hi,

I have mdb - message driven bean, that process hundreds messages per
minute. I want to update memcache on every message - this is very
frequently. But if i make my mdb singleton it will process messages
sequentially, so the next update will be done when the previous is
finished. Is this a problem?
Regards.

[Marc Bollinger]

If the scale is actually only hundreds per minute, even sequentially,
I wouldn't worry about memcached writes being the bottleneck, even on
a small machine.

--
Marc Bollinger
mboll...@gmail.com

[Henrik Schröder]

If I remember it correctly, Dustin's SpyMemcached client for Java has both synchronous and asynchronous methods.


/Henrik

[Adam Lee]

Yeah, memcached can do hundreds of writes per seconds, so I wouldn't sweat it.

--
awl

[mariyan nenchev]

Well this is not actually true. I tried to update memcache before with about 10 messages per second every second and it began to throw some IO exceptions. I am using theese classes:
import com.danga.MemCached.MemCachedClient;
import com.danga.MemCached.SockIOPool;
The other code that manages my cached objects is my.

[Dustin]

On Feb 25, 12:39 am, mariyan nenchev <nenchev.mari...@gmail.com>
wrote:

There must be something terribly wrong with your application or
configuration.

Last time I measured, I was doing 90,000 per second on my laptop
(just relative numbers, I was working on a java client improvement
which I brought up from about 45,000/s for the same test).

I don't see problems at 10tps on my old hard drives.

What kinds of problems were you seeing? Do you have code we can
perhaps help straighten out?

[Carlos Alvarez]

We have a few thousands updates per second per box with memcached
(linux with eight core each) and the box hardly reports any activity.

The only problem we had was the terrible slow .net serialization and
the md5 computed when we plug hibernate with memcached.

In my experience, the bottlenecks at the clients arise far more often
than in the server.

--
Sent from my mobile device

Quidquid latine dictum sit, altum sonatur. ;-)

[mariyan nenchev]

Hi,

Before 4 months i had to create cache for data stream messages coming from TCP stream about 400 per second every second. I had to keep the last message only. I implemented MDB that received the messages and updated the memcache but it couldn't take the pressure. May be i didn't configured it correct - please see the code above. So i did put the last message directly in hashmap and it works perfectly. But now i need to put data that is very frequently updated in memcache. So you say this is possible, how do i test it and please have a look at my code, and see if something is wrong.
Thanks.

[mariyan nenchev]

OK,
i wrote simple test to show you what is happening. I create 100 threads, and i start them at the same time. They begin to update the cache.

public class UpdateableObject implements Serializable {
    public int i = 0;
}
    static int i = 0;
    public static void main(String[] args) {
        final List<Thread> ts = new ArrayList<Thread>();
        for (i = 0; i < 100; i++) {
            final Thread t = new Thread(new Runnable() {
                int num = i++;
                @Override
                public void run() {
                    while (true) {
                        final long t = System.currentTimeMillis();
                        final UpdateableObject track = UpdateableObjectMemCache.instance().get(num);
                        track.current = ++track.i;
                        UpdateableObjectMemCache.instance().cache(num, track);
                        final long t1 = System.currentTimeMillis();
                        System.out.println(t1-t);
                    }                   
                }
            });
            ts.add(t);
        }
        for (final Thread thread : ts) {
            thread.start();
        }

If i increase the number of threads >=200
It blows :
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - null
java.net.SocketTimeoutException
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at com.danga.MemCached.SockIOPool$SockIO.readLine(SockIOPool.java:1767)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1285)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1209)
    at claire.mem.cache.Memcached.get(Memcached.java:85)
    at claire.mem.cache.MemCacheManager.cache(MemCacheManager.java:27)
    at claire.web.util.PurchaseTracker$2.run(PurchaseTracker.java:138)
    at java.lang.Thread.run(Thread.java:619)
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - null
java.net.SocketTimeoutException
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at com.danga.MemCached.SockIOPool$SockIO.readLine(SockIOPool.java:1767)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1285)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1209)
    at claire.mem.cache.Memcached.get(Memcached.java:85)
    at claire.mem.cache.MemCacheManager.tryCache(MemCacheManager.java:92)
    at claire.mem.cache.MemCacheManager.get(MemCacheManager.java:105)
    at claire.web.util.PurchaseTracker$2.run(PurchaseTracker.java:135)
    at java.lang.Thread.run(Thread.java:619)
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - null
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - null
java.net.SocketTimeoutException
    at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
    at java.io.DataInputStream.read(DataInputStream.java:132)
    at com.danga.MemCached.SockIOPool$SockIO.readLine(SockIOPool.java:1767)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1285)
    at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1209)
    at claire.mem.cache.Memcached.get(Memcached.java:85)
    at claire.mem.cache.MemCacheManager.tryCache(MemCacheManager.java:92)
    at claire.mem.cache.MemCacheManager.get(MemCacheManager.java:105)
    at claire.web.util.PurchaseTracker$2.run(PurchaseTracker.java:135)
    at java.lang.Thread.run(Thread.java:619)
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - ++++ exception thrown while trying to get object from cache for key: TESTcountpurchase
com.danga.MemCached.MemCachedClient Thu Feb 25 17:10:59 EET 2010 - null
java.net.SocketTimeoutException

Here is simple output of the time taken to update(it is not precise but gives me some idea...):

1
122
120
73
1
0
72
72
70
421
129
0
1
1
0
425
132
130
1
1
0
367
75

[mariyan nenchev]

What will happen if i create 5000 threads(5000 users update simultaniosly) to update the cache just once at the same time?

[Nelz]

First thing I would suggest: use a modern client. The Danga client is
getting old, but the Spy client is up-to-date.

http://code.google.com/p/spymemcached/

- Nelz

[mariyan nenchev]

Hi, it is not up to me to change the client.

[Carlos Alvarez]

That's a problem, because a lot of issues have to do with clients.

Anyway, how healthy is your network?.

Have you tried running memcached with -vv (two 'v' and not a 'w' and)
and see the log in the server?

On Thu, Feb 25, 2010 at 1:56 PM, mariyan nenchev
<nenchev...@gmail.com> wrote:
> Hi, it is not up to me to change the client.
>

--

[mariyan nenchev]

Every thing is one one machine in development, but it will be separate server for the memcache with LAN.
Yes i tried it. What should i see? It stop updating the cache while i get the exceptions above.

[Carlos Alvarez]

I assume you are running memcached in any kind of unix.

So do you had the java virtual machine and the memcached runing in the
box while you got those errors?. If so, forget about the network. But
you could check how busy is your box when you start those threads
(mem, cpu)

On the other hand, see the log in the memcached server. Start it like

memcached <your usual configuration> -vv 2>/tmp/log

and see what appears in the log file.

Also see syslog file.

On Thu, Feb 25, 2010 at 2:35 PM, mariyan nenchev

[Adam Lee]

your problem might indeed be related to the danga client.. if you take
a look at its connection pool (SockIOPool i believe) it has a lot of
synchronized code, which isn't very happy in a highly concurrent
environment. i recommend at least doing a test with the spy client to
see if it actually helps.

On Thursday, February 25, 2010, mariyan nenchev

<nenchev...@gmail.com> wrote:
> What will happen if i create 5000 threads(5000 users update simultaniosly) to update the cache just once at the same time?
>

--
awl

[Dustin]

On Feb 25, 7:21 am, mariyan nenchev <nenchev.mari...@gmail.com> wrote:
> What will happen if i create 5000 threads(5000 users update simultaniosly)
> to update the cache just once at the same time?

Unless your box has 5,000 cores, your application is going to fall
apart in all kinds of other exciting ways here. It's certainly not
going to happen all at the same time anyway.

You're far better off running that work an ExecutorService.

But in general, spymemcached doesn't care how many threads you're
using. There are only practical limitations (e.g. you don't have
5,000 cores so you have a lot of threads waiting for a CPU to become
available for each app thread, spymemcached thread, memcached thread,
etc...) and small contention points while enqueuing commands.

[mariyan nenchev]

Hi, theese threads simulate users. I have jboss AS that manages all user requests and for every request i need to update memcache. I will try spy client tomorow.

[Adam Lee]

Is this for testing purposes? 

I think there is a flaw in your design.  As Dustin said, your machine is not going to be happy about 5000 concurrent IO operations unless you have 5000 cores.  

Java's concurrent package is very easy to use and has low overhead-- take a look at ExecutorService.  Maybe create a thread pool with the same number of threads as you have cores and submit jobs to it for the operations you want to run, for example...

--
awl