Memory Meter Uninitialized exception on server start

[oz]

Our Titan Db (Cassandra Backended) is definitely slow. 

We are getting message about Memory Meter.. We have 15GB Free Ram.

Also we are getting lots of OutOfMemory errors.

Something is wrong but we have no idea about it.

Any idea?

Slow means: For empty database, local db, simple gremlins are 70-250 ms..

[Pavel Yaskevich]

Hey, 

  You most certainly need to tune your GC settings and adjust Cassandra memtable size.

Best Regards, Pavel.

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[oz]

I'm getting this errors also,

How can I tune  GC settings and adjust Cassandra memtable size?

Only settings I can found in cassandra.yaml is "memtable_total_space_in_mb: 2048"

Jul 02, 2013 8:50:11 PM org.glassfish.grizzly.filterchain.DefaultFilterChain exe

cute

WARNING: Exception during FilterChain execution

java.lang.OutOfMemoryError: PermGen space

        at java.lang.ClassLoader.defineClass1(Native Method)

        at java.lang.ClassLoader.defineClass(Unknown Source)

        at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.define(Cl

assLoaderForClassArtifacts.java:42)

        at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(Cla

ssLoaderForClassArtifacts.java:86)

        at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts$1.run(Cla

ssLoaderForClassArtifacts.java:84)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.codehaus.groovy.reflection.ClassLoaderForClassArtifacts.defineCla

ssAndGetConstructor(ClassLoaderForClassArtifacts.java:84)

        at org.codehaus.groovy.runtime.callsite.CallSiteGenerator.compilePojoMet

hod(CallSiteGenerator.java:229)

        at org.codehaus.groovy.reflection.CachedMethod.createPojoMetaMethodSite(

CachedMethod.java:257)

        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.createCachedM

ethodSite(PojoMetaMethodSite.java:159)

        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.createPojoMet

aMethodSite(PojoMetaMethodSite.java:148)

        at groovy.lang.MetaClassImpl.createPojoCallSite(MetaClassImpl.java:3023)

        at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPojoSite(Cal

lSiteArray.java:116)

        at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(Cal

lSiteArray.java:150)

        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSi

teArray.java:42)

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa

llSite.java:108)

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa

llSite.java:112)

        at com.tinkerpop.gremlin.groovy.loaders.GraphLoader$_load_closure10.doCa

ll(GraphLoader.groovy:90)

        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:

90)

        at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(Closur

eMetaMethod.java:80)

        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMetho

dSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)

        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMeta

MethodSite.java:53)

        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSi

teArray.java:42)

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa

llSite.java:108)

        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa

llSite.java:120)

        at com.tinkerpop.gremlin.groovy.loaders.GraphLoader$_load_closure11.doCa

ll(GraphLoader.groovy:96)

        at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

13/07/02 20:51:28 INFO service.GCInspector: GC for PS MarkSweep: 1084 ms for 4 c

ollections, 56888704 used; max is 1028128768

13/07/02 20:53:01 INFO service.GCInspector: GC for PS MarkSweep: 9697 ms for 48

collections, 56694640 used; max is 1073479680

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler i

n thread "OptionalTasks:1"

Jul 02, 2013 9:00:05 PM org.glassfish.grizzly.filterchain.DefaultFilterChain exe

cute

WARNING: Exception during FilterChain execution

java.lang.OutOfMemoryError: PermGen space

[Pavel Yaskevich]

It might be that you are doing too big of transactions or your heap is too small.

You can try setting "memtable_total_space_in_mb" to 512 and try increasing total heap size and setting young gen to 1GB.

[Oğuz Karadenizli]

I've set memtable_total_space_in_mb:2048  in cassandra.yaml

I've also changed titan.bat

set JAVA_OPTIONS=-Xms1G^

 -Xmx1G^

 -Dcom.sun.management.jmxremote.port=7199^

 -Dcom.sun.management.jmxremote.ssl=false^

 -Dcom.sun.management.jmxremote.authenticate=false

 -javaagent:"D:\Db\Titan\lib\jamm-0.2.5.jar"                         <-------- I've added this so Memory Metter warning messages gone

but I could not find the correct syntax for total heap size and young gen 

will I add this parameters to cassandra.yaml? or titan.bat?

JRE:  

Platform 1.7

Product: 1.7.0_17

Is it normal also?

Windows 2008 R2

--
You received this message because you are subscribed to a topic in the Google Groups "Aurelius" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/aureliusgraphs/LMUOBP8NbAY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to aureliusgraph...@googlegroups.com.

[Pavel Yaskevich]

1GB for "mx" is too small, you can do -Xms1GB -Xmx4GB -XX:NewSize=768MB

[Oğuz Karadenizli]

I got  Unrecognized VM option  error for -XX:NewSize=768MB

I've added two files as attachment.

titan.bat (titan.bak) and cassandra.yaml

[Pavel Yaskevich]

I'm sorry, it actually should be -XX:NewSize=768M (not MB)

[Oğuz Karadenizli]

Thank Pavel,

It works, performance is a little better.

But still 

50ms-250ms for g.v.has('Type','Article')

50-80ms for g.v(1)

I suspect that RexterPro..

I will investigate tomorrow.

Do you have any benchmark for empty database? 

I'm waiting 0ms,2ms..

Thanks again.

[Pavel Yaskevich]

No problem, you can actually investigate if it's GC issue by adding following options 

-XX:+PrintGCDetails -XX:+PrintGCDateStamps  -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+PrintPromotionFailure 

It could be a RexterPro problem but, unfortunately, I don't know how to benchmark it, maybe somebody else would help you with that.

[Oğuz Karadenizli]

I will check RexterPro, but with these problems we could not use Cassandra or other db for Titan backend.. 

Our project is web application, we have no chance with this performance :(

These are from console:

{Heap before GC invocations=2 (full 0):

 PSYoungGen      total 688128K, used 607916K [0x00000007aaab0000, 0x00000007daab

0000, 0x0000000800000000)

  eden space 589824K, 100% used [0x00000007aaab0000,0x00000007ceab0000,0x0000000

7ceab0000)

  from space 98304K, 18% used [0x00000007ceab0000,0x00000007cfc5b220,0x00000007d

4ab0000)

  to   space 98304K, 0% used [0x00000007d4ab0000,0x00000007d4ab0000,0x00000007da

ab0000)

 ParOldGen       total 262144K, used 8K [0x0000000700000000, 0x0000000710000000,

 0x00000007aaab0000)

  object space 262144K, 0% used [0x0000000700000000,0x0000000700002000,0x0000000

710000000)

 PSPermGen       total 38336K, used 38270K [0x00000006fae00000, 0x00000006fd3700

00, 0x0000000700000000)

  object space 38336K, 99% used [0x00000006fae00000,0x00000006fd35f8c8,0x0000000

6fd370000)

2013-07-03T00:24:11.345+0000: [GC

Desired survivor size 100663296 bytes, new threshold 7 (max 15)

 [PSYoungGen: 607916K->32195K(688128K)] 607924K->32219K(950272K), 0.0432010 secs

] [Times: user=0.09 sys=0.01, real=0.05 secs]

Heap after GC invocations=2 (full 0):

 PSYoungGen      total 688128K, used 32195K [0x00000007aaab0000, 0x00000007daab0

000, 0x0000000800000000)

  eden space 589824K, 0% used [0x00000007aaab0000,0x00000007aaab0000,0x00000007c

eab0000)

  from space 98304K, 32% used [0x00000007d4ab0000,0x00000007d6a20e78,0x00000007d

aab0000)

  to   space 98304K, 0% used [0x00000007ceab0000,0x00000007ceab0000,0x00000007d4

ab0000)

 ParOldGen       total 262144K, used 24K [0x0000000700000000, 0x0000000710000000

, 0x00000007aaab0000)

  object space 262144K, 0% used [0x0000000700000000,0x0000000700006010,0x0000000

710000000)

 PSPermGen       total 38336K, used 38270K [0x00000006fae00000, 0x00000006fd3700

00, 0x0000000700000000)

  object space 38336K, 99% used [0x00000006fae00000,0x00000006fd35f8c8,0x0000000

6fd370000)

}

[Pavel Yaskevich]

That looks ok but it's pretty small sample tho. I think it would be useful to check RexterPro, that might be a problem.

[Oğuz Karadenizli]

Ok. so see you tomorrow :)