turning off gremlin class cache

[Peter Musial]

Hi all,

I am using JansuGraph in a NodeJS environment and connect to it using gremlin client.  It was observed that gremlin will cache compiled queries as classes, which eventually puts stress on memory and affects latency of query execution.  At my end there is effort to change queries to be parametrized, but that is a significant code change, hence the desire to simply turn off cache all together.  

From the gremlin documentation, we found this flag:

#jsr223.groovy.engine.keep.globals : phantom

Which is supposed to accomplish the task, but having trouble to understand how it works (if it does at all).  Here is a sample code.

'use strict';

const Gremlin = require('gremlin');

const GremlinClient = require('./GremlinClient.js');

const client = Gremlin.createClient(port, host, {path: '/gremlin'});

client.execute(script, {'#jsr223.groovy.engine.keep.globals':'phantom'}, (err, results) => { /* process response */ });

Specific environment information: gremlin 2.7.0 node module, Janus 0.2.1 with Cassandra 3.10

An example of a query from gremlin logs where you can see that bindings include the phantom flag: 

45805 [gremlin-server-worker-1] DEBUG log-aggregator-encoder  - [id: 0x1279ba5c, L:/127.0.0.1:8182 - R:/127.0.0.1:51764] READ: RequestMessage{, requestId=2393b6e0-ba85-11e8-9194-87a8c590e3d7, op='eval', processor='', args={gremlin=g.V().has(...).as(...).optional(__.in(...).as(...).select(...), bindings={#jsr223.groovy.engine.keep.globals=phantom}, accept=application/json, language=gremlin-groovy}}

However, in VisualVM monitor, class unloading is not very predictable and in general the number of loaded classes keeps accumulating.  Perhaps this is the desired behavior, or perhaps I should use other flag value such as 'weak' or 'soft'.  

When the flag is not provided as class cache increases monotonically and so does latency of query execution.  We did a POC with parametrized queries and latency remains constant regardless of the duration of the run.

Some more resources.

Tried several variants of setting it, but nothing worked.

1. added to scripts/empty-sample.groovy

2. added directly to our driver code

3. set Java args -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled to clean up perm heap

some more links:

https://stackoverflow.com/questions/712187/troubleshooting-grails-groovy-memory-leaks/779221#779221

https://stackoverflow.com/questions/5815952/locating-code-that-is-filling-permgen-with-dead-groovy-code

https://stackoverflow.com/questions/24169976/understanding-groovy-grails-classloader-leak

https://bugster.forgerock.org/jira/browse/OPENAM-6024

There is no tenable evidence that the flag works as advertised.

Question to the community. Is my approach to disabling individual queries correct (meaning that indeed there is no caching and each query will have some constant compilation time)?  

Regards,

Peter

[Stephen Mallette]

I think your expectation for what this is supposed to do is a little off. First of all, you can see that we do have a test for phantom reference cleanup and it does seem to work:

https://github.com/apache/tinkerpop/blob/f9668cc621a5e664760dcf48c47d8feca10ad7d2/gremlin-server/src/test/java/org/apache/tinkerpop/gremlin/server/GremlinServerIntegrateTest.java#L1006

but, note that it's cleaning up the global function cache not the cache that holds your scripts. that's a different cache all together and you don't have a lot of control over that. It's basically configured by default to use "soft" cache values which means that cached compiled classes should release in a least-used manner on GC in response to memory demand. There is currently no way to change that behavior:

https://github.com/apache/tinkerpop/blob/f9668cc621a5e664760dcf48c47d8feca10ad7d2/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/jsr223/GremlinGroovyScriptEngine.java#L155

I wouldn't be against offering more flexibility around cache configuration options - perhaps that would be helpful to folks, especially since soft isn't super predictable.

--
You received this message because you are subscribed to the Google Groups "Gremlin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gremlin-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/94f1adda-8b70-4022-b059-17cf589b4d7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Dale]

Peter, just curious, have you tried turning on stats?  I wonder if that shows anything useful for this particular issue.

I think caches should always be tunable as different use cases have different needs.  Caffeine also recommends not using 'soft' and instead setting a max value for more predictable behavior [1].  Gremlin Server does this for the side-effects cache.  I think we should do the same here.  I've created https://issues.apache.org/jira/browse/TINKERPOP-1644

1. https://github.com/ben-manes/caffeine/wiki/Eviction#reference-based

Robert Dale

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CAA-H43-C2iTqo3VWEjqHihqo36Y_ohVM_k7uBh%2BD-7Xk1-v43Q%40mail.gmail.com.

[Stephen Mallette]

i think you meant to reference this: https://issues.apache.org/jira/browse/TINKERPOP-2038

To view this discussion on the web visit https://groups.google.com/d/msgid/gremlin-users/CABed_4owFFoykMuppPAYi4tQWkBvDAnmCPDdpRceiAEuAAPcXg%40mail.gmail.com.