I am really curious why spy cannot store up to 6 million...
On Sat, Oct 16, 2010 at 12:59 PM, Shi Yu <she...@gmail.com> wrote:
> Hi,
>
> I have two problems when using memcached java clients the spymemcached
> (http://code.google.com/p/spymemcached/) and the gwhalin java client
> for memcached (http://github.com/gwhalin/Memcached-Java-Client). I
> found that the spymemcached failed to store more than 4.3 million
> records sometimes 3.7 million (please see my code below). There was no
> error no exception, but simply the code automatically stopped at the
> 4.3 million and didn't even hit the final line. In contrast, the
> gwhalin java client was able to insert 6 million records without
> problem, however, comparing the speed of inserting the first 4 million
> records that the gwhalin client is much slower than the spymemcached.
> The memcached server is set up using the following command
> "./memcached -d -m 4000 127.0.0.1 -p 11211" and I think there is no
> problem at the server side. What is the problem here, should I adjust
> any settings? Thanks?
>
> -Shi
>
>
> //spymemcached code
> public static void main(String[] args) throws Exception {
> MemcachedClient mc=new MemcachedClient(new
> InetSocketAddress("ocuic32.research", 11211));
> mc.flush();
> System.out.println("Memchaced flushed ...");
> int count = 0;
> for(int i=0;i<6000000;i++){
> String a = "String"+i;
> String b = "Value"+i;
>
> mc.add(a,i,(String) b);
> count ++;
> if (String.valueOf(count).endsWith("00000"))
> System.out.println(count+ " elements added.");
> }
>
> System.out.println("done "+ count +" records inserted");
> //spymemcached aint able to get this line
> }
>
>
>
> //gwhalin memcached code
> public static void main(String[] args) throws Exception {
> BasicConfigurator.configure();
> String[] servers = { "ocuic32.research:11211" };
> SockIOPool pool = SockIOPool.getInstance();
> pool.setServers( servers );
> pool.setFailover( true );
> pool.setInitConn( 10 );
> pool.setMinConn( 5 );
> pool.setMaxConn( 250 );
> pool.setMaintSleep( 30 );
> pool.setNagle( false );
> pool.setSocketTO( 3000 );
> pool.setAliveCheck( true );
> pool.initialize();
>
> MemcachedClient mcc = new MemcachedClient();
> mcc.flushAll();
> int count = 0;
> int maxlength = 0;
> //while((line=br.readLine())!=null){
>
> for(int i=0;i<6000000;i++){
> String a = "String"+i;
> String b = "Value"+i;
> String sha1_ad1 = AeSimpleSHA1.SHA1(a);
> mcc.set(sha1_ad1,(String) b);
> count ++;
> if (String.valueOf(count).endsWith("00000"))
> System.out.println(count+ " elements added.");
>
> }
>
> System.out.println("done "+ count +" records
> inserted"); //gwhalin 's client is able to get this line, but very
> slow
> }
>
I'd definitely spend some more time analyzing what's going on if I
were you before going down that road. Turn up gc logging
(-verbose:gc), see if the app is heavily gc-ing when the program
"stops"; dump threads (kill -3 pid) etc....
Shi
public static void mapload() throws Exception{
MemcachedClient mc=new MemcachedClient(new
InetSocketAddress("ocuic32.research", 11211));
mc.flush();
System.out.println("Memchaced flushed ...");
CacheLoader cl = new CacheLoader(mc);
System.out.println("Cache loader created ...");
Map<String,String> map1 = new HashMap<String,String>();
Map<String,String> map2 = new HashMap<String,String>();
Map<String,String> map3 = new HashMap<String,String>();
for (int i=0;i<1999999;i++){
map1.put("key"+i,"value"+i);
}
try{
cl.loadData(map1);
System.out.println("map1 loaded");
}catch(Exception e1){
e1.printStackTrace();
}
map1=null;
for (int i=2000000;i<3999999;i++){
map2.put("key"+i,"value"+i);
}
try{
cl.loadData(map2);
System.out.println("map2 loaded");
}catch(Exception e2){
e2.printStackTrace();
}
map2=null;
for (int i=4000000;i<5999999;i++){
map3.put("key"+i,"value"+i);
}
try{
cl.loadData(map3);
System.out.println("map3 loaded");
}catch(Exception e3){
e3.printStackTrace();
}
map3=null;
System.out.println("All done");
}
And I run with the following java command on a 64-bit Unix machine
which has 8G memory. I separate the Map into three parts, still
failed. TBH I think there is some bug in the spymemcached input
method. With Whalin's API there is no any problem with only 2G heap
size, just a little bit slower but thats definitely better than being
stuck for 6 hours on a bugged API.
java -Xms4G -Xmx4G -classpath ./lib/spymemcached-2.5.jar Memcaceload
Here is the error output:
2010-10-16 22:40:50.959 INFO net.spy.memcached.MemcachedConnection:
Added {QA sa=ocuic32.research/192.168.136.36:11211, #Rops=0, #Wops=0,
#iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect
queue
Memchaced flushed ...
Cache loader created ...
2010-10-16 22:40:50.989 INFO net.spy.memcached.MemcachedConnection:
Connection state changed for sun.nio.ch.SelectionKeyImpl@25fa1bb6
map1 loaded
map2 loaded
java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:51)
at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:215)
at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:207)
at java.lang.StringCoding.encode(StringCoding.java:266)
at java.lang.String.getBytes(String.java:947)
at net.spy.memcached.KeyUtil.getKeyBytes(KeyUtil.java:20)
at net.spy.memcached.protocol.ascii.OperationImpl.setArguments(OperationImpl.java:86)
at net.spy.memcached.protocol.ascii.BaseStoreOperationImpl.initialize(BaseStoreOperationImpl.java:48)
at net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:601)
at net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:582)
at net.spy.memcached.MemcachedClient.addOp(MemcachedClient.java:277)
at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:314)
at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:691)
at net.spy.memcached.util.CacheLoader.push(CacheLoader.java:92)
at net.spy.memcached.util.CacheLoader.loadData(CacheLoader.java:61)
at net.spy.memcached.util.CacheLoader.loadData(CacheLoader.java:75)
at MemchacedLoad.mapload(MemchacedLoad.java:90)
at MemchacedLoad.main(MemchacedLoad.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Shi
On Sun, Oct 17, 2010 at 12:36 AM, Jonathan Leech <jona...@gmail.com> wrote:
> Sounds like bug 125 to me. Your thread inserting the records can write them
> to the queue faster than they are written from the queue to memcached, the
> queue fills up with more and more records, and each one takes longer than
> the last to get written. If you are also bouncing up against the upper limit
> of the heap, then the VM will compound the issue spending a lot of time
> garbage collecting.
>
>
>
> On Oct 16, 2010, at 6:53 PM, Boris Partensky <boris.p...@gmail.com>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "spymemcached" group.
>> To post to this group, send email to spymem...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> spymemcached...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/spymemcached?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "spymemcached" group.
> To post to this group, send email to spymem...@googlegroups.com.
> To unsubscribe from this group, send email to
> spymemcached...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/spymemcached?hl=en.
>
>
Be careful when you start calling it a buggy API, especially as you
present the quality of code that you did in your initial test case. Your
bugs-per-LOC was pretty high.
However, it seems that you did in fact stumble into a bug in the Spy client,
but only because you did no error checking at all.
Dustin,
while trying to re-create this problem and point out the various errors in
his code, I found that, in his test case, if I did not call Future.get() to
verify the result of the set, the spyMemcached client leaked memory. Given
that the Spymemcached wiki says that fire-and-forget is a valid mode of
usage, this appears to be a bug.
Here's my testcase against spymemcached-2.5.jar:
'java -cp .:./memcached-2.5.jar FutureResultLeak true' leaks memory and will
eventually die OOM.
' java -cp .:./memcached-2.5.jar FutureResultLeak false' does not leak and
runs to completion.
Here's the code. It's based on Shi's testcase so he and I now share the
blame for code quality :)
----------------------
import net.spy.memcached.*;
import java.lang.*;
import java.net.*;
import java.util.concurrent.*;
public class FutureResultLeak {
public static void main(String[] args) throws Exception {
boolean leakMemory = false;
if (args.length >= 1) {
leakMemory = Boolean.valueOf(args[0]);
}
System.out.println("Testcase will " + (leakMemory ? "leak memory" : "not
leak memory"));
MemcachedClient mc=new MemcachedClient(new
InetSocketAddress("localhost", 11211));
mc.flush();
System.out.println("Memcached flushed ...");
int count = 0;
int logInterval = 100000;
int itemExpiryTime = 600;
long intervalStartTime = System.currentTimeMillis();
for(int i=0;i<6000000;i++){
String a = "String"+i;
String b = "Value"+i;
Future<Boolean> f =mc.add(a,itemExpiryTime, b);
if (!leakMemory) {
f.get();
}
count++;
if (count % logInterval == 0) {
long elapsed = System.currentTimeMillis() - intervalStartTime;
double itemsPerSec = logInterval*1.0/elapsed;
System.out.println(count+ " elements added in " + elapsed + " (" +
itemsPerSec + " per sec).");
intervalStartTime = System.currentTimeMillis();
}
}
System.out.println("done "+ count +" records inserted");
mc.shutdown(60, TimeUnit.SECONDS);
}
}
----------------------
Regards,
Kelvin
This is year 2010 and computer programs should not be that fragile.
And I believe my code is just a fast simple toy problem trying to find
out why I failed too many times in my real problem. Before I post my
problem, I checked and searched many documents, I read through the API
and there is no clear instruction telling me what should I do to
prevent such an error. I don't have time to bug an API on purpose, I
am doing NLP pos tagging and I have exactly 6 million stemmed word to
store. Fortunately or unlucky to me, that number exactly triggers the
failure so I had to spend 6 hours finding out the reason. Actually spy
client is the first API I tried, as I pointed out in my first post, it
is fast, however, there is an error. I don't think for a normal
end-product API, the memory leak issue should be considered by the
user.
Shi
Shi