Please HELP! Slow commit and OutOfMemory in 2.1.0 with transaction w average 100 vertices/edges

[Chris Gomes]

Hello,
I really hope someone can shed some light on the performance issues that we  have been experiencing. Thank you much in advance.

Regards,
-Chris

I have attached 2 html files - one is heap dump of OrientDB server and the other is App 2 VM.
Below is also partial stack trace of App 2 and some code snippet where main transaction boundaries are.

We are getting long commit time and eventually runs out of memory, currently Orient is becoming unusable for what we are trying to do with client App 2 as described below. 

Background:
Database Server instance using Java 8
1) We run OrientDB using server.sh/bat as a standalone and with 1G of heap space and default Disk Cache
Databases 
2) We have 2 databases created as plocal and graph, let's name them, Expansion & Universe.
3) 2 have 2 client applications, both are spring boot project with an embedded jetty container, run as executable jars

Java Client Applications - Spring Boot using Java 8
4) App 1 - reads a large xml file (670 Meg and process each business unit of at a time) and create Vertices and Edges based on the
business concepts described in the xml. NOTE, in both app we define the schema for Vertices and Edge Type definitions. 
For APP 1, Expansion DB barely has any  index as we mostly write blindly as an expansion of elements from XML. BUT, we will need to read the data
from APP 2. To load the file and expand the xml int vertex and edge representation it takes anywhere from 55 to 2 hours, depending on SSD vs regular disk, this is on a brand new clean DB. 

One thing that is clear, as data grows, same data file loaded again will take longer to load, seems like linear growth. I noticed something in the forum about this already. 
This file has about 35,000 business unit worth of data. The raw size in DB is  2,610,075 edges,  2,610,076 vertices.  We have defined about 16 Vertices (vertex types) and 23 Edges (edge types) in Expansion and Universe DB.  
This is just one file and we will get incremental update everyday and will grow over the time. So, we would need to be able to handle large amount of data in MANY millions in both databases.

App 1 runs fine with 256Meg of heap space and though it gets slower over the time, it runs.

5) App 2 - All edge types have non unique index as we need to look for existing of vertices in an edge as well as we keep history of relationships as in what point a relationship is no longer current. 
We also have index for some of the Vertex where we have some business key we need to query on. Overall # of indices are certainly much more than Expansion DB, 
but not overwhelming as we would be doing lot of bulk writing (create/update vertices, create/update edges) as well as reading to look for existing vertices or edges.

6) NOTE, we use connection pool and transaction. Our transaction boundary would process a Business unit worth of data, which translates into average of 100 vertices/edges. 
However some extreme cases we could end up creating/updating about 1500 vertices/edges. 
7) App 2 has 2 Graph connection Factory one for each DB (Expansion and Universe). App reads one Business unit worth of vertices/edges from Expansion DB and in a transaction writes (create/update/lookup) into Universe DB. 
Average for this varies from 2 sec to 4 secs depending on SSD vs non-SSD disks. 

8) With 1G of heap for the App2, it will eventually get out of memory exception. I have only been able to process 30,000 business units read from Expansion DB and it took over 11 hours before getting out of memory. 
This is on a SSD drive. At this performance, the app and DB becomes pretty much unusable. 

9) Tried few suggestions based on forum and documentation - disabling transaction log, but I don't think it made any real difference. 

Here's the stack trace from App 2. I have also heap dumps from both OrientDB server and App 2.

Would love some insights as to what the options are. I am curious if anyone else is using OrientDB for something like the way I described. I don't think this usecase is anything out of the ordinary.

--- Stack Trace from Client App 2: Note, Caused by: java.lang.OutOfMemoryError: Java heap space---

2015-09-11 04:11:16.858 DEBUG 1132 --- [qtp174969681-23] .b.u.u.UniverseMergeRequestServiceFacade : Processed 30555 statements with an average of 1332 milliseconds
2015-09-11 04:17:52.406 DEBUG 1132 --- [qtp174969681-23] c.b.u.u.StatementUniverseMergeService    : {"delivery-id":"1e556f40-7e00-d9ba-9464-22000b9447eb","log-type":"universe-merge-statement","universe-merge-request-uuid":"c0fbbf7d-358e-4a01-a5c1-44007a311aab","statement-uid":"1e4ed31a-3631-df77-b997-22000a4590b6"}
2015-09-11 04:17:55.268 DEBUG 1132 --- [qtp174969681-23] c.b.u.u.StatementUniverseMergeService    : {"delivery-id":"1e556f40-7e00-d9ba-9464-22000b9447eb","log-type":"universe-merge-statement","universe-merge-request-uuid":"c0fbbf7d-358e-4a01-a5c1-44007a311aab","statement-uid":"1e4ed31a-3f9b-d96a-b997-22000a4590b6"}
2015-09-11 04:18:01.435 DEBUG 1132 --- [qtp174969681-23] c.b.u.u.StatementUniverseMergeService    : {"delivery-id":"1e556f40-7e00-d9ba-9464-22000b9447eb","log-type":"universe-merge-statement","universe-merge-request-uuid":"c0fbbf7d-358e-4a01-a5c1-44007a311aab","statement-uid":"1e4ed31a-3fe7-d518-b997-22000a4590b6"}
2015-09-11 04:34:02.328 ERROR 1132 --- [qtp174969681-23] .b.u.u.UniverseMergeRequestServiceFacade : Universe merge request for deliveries failed for extraction date > 2001-01-01 00:00:00.0

com.bps.commons.exception.StateDumpException: Exception thrown when creating the statement due to Exception during response processing.
	at com.bps.urjanet.universe.StatementUniverseMergeService.merge(StatementUniverseMergeService.java:97)
	at com.bps.urjanet.universe.UniverseMergeRequestServiceFacade.processMergeRequest(UniverseMergeRequestServiceFacade.java:87)
	at com.bps.urjanet.universe.resource.UniverseMergeResource.requestUniverseMerge(UniverseMergeResource.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:172)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:152)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:384)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:342)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:254)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1030)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:377)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:381)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:344)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:221)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:224)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:85)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:497)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Unknown Source)
Caused by: com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException: Exception during response processing.
	at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:445)
	at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:400)
	at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:282)
	at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:171)
	at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2136)
	at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:1364)
	at com.orientechnologies.orient.client.remote.OStorageRemoteThread.commit(OStorageRemoteThread.java:448)
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:596)
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:156)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2578)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2547)
	at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:161)
	at com.bps.urjanet.universe.StatementUniverseMergeService.merge(StatementUniverseMergeService.java:85)
	... 95 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space

2015-09-11 04:34:02.349 ERROR 1132 --- [qtp174969681-23] c.b.u.u.r.StateDumpExceptionMapper       : StateDumpExceptionMapper>>toResponse:Error Occured: Exception thrown when updating the StatementsDelivery due to Transaction was rolled back more times than it was started.
Transaction was rolled back more times than it was started.


com.bps.commons.exception.StateDumpException: Exception thrown when updating the StatementsDelivery due to Transaction was rolled back more times than it was started.
	...
Caused by: com.orientechnologies.orient.core.exception.OTransactionException: Transaction was rolled back more times than it was started.
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.rollback(OTransactionOptimistic.java:188)
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.rollback(OTransactionOptimistic.java:169)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.endResponse(ONetworkProtocolBinary.java:1764)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.readRecord(ONetworkProtocolBinary.java:1662)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:353)
	at com.orientechnologies.orient.server.network.protocol.binary.OBinaryNetworkProtocolAbstract.execute(OBinaryNetworkProtocolAbstract.java:223)
	at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

///
Code snippet of where the main transaction processing happens. This method is called from another service to loop for each Business unit of work.
As you can see, each business unit of work is where transactions are happening. 

public TreeVertex merge(params...){
		
		List<Reference> references = new ArrayList<>();
		List<MeasurementPeriod> measurementPeriods = new ArrayList<>();
		List<SummaryData> summaryDatas = new ArrayList<>();
		List<RateComponents> rateComponentsList = new ArrayList<>();
		Map<String, OrientVertex> vertexMap = new HashMap<>();	//<UID, Vertex> Only objects that have a UID that are created in process.

		OrientGraph mergedGraph = this.orientUniverseGraphFactory.getTx();	//Exception should flow through.  This is a problem.
		OrientVertex statement = null;

		try {
			mergedGraph.begin();
			mergedGraph.getRawGraph().getTransaction().setUsingLog(false);

			statementProcessor.endEdgesWithStatementUid(mergedGraph, uid);	//If any edges are found with the statement UID, then end those relationships.

			statement = process(mergedGraph, expandedTreeVertex, deliveryId, uid, vertexMap,
					references, measurementPeriods, summaryDatas, rateComponentsList
			);

			mergedGraph.commit();

		} catch (Exception e) {
			mergedGraph.rollback();
			
			throw new StateDumpException("Exception thrown when creating the statement due to " + e.getMessage(), e)
				.addStateDump("universeMergeRequestUuid", universeMergeRequestUuid)
				.addStateDump("deliveryId", deliveryId)
				.addStateDump("statement-uid", uid);
		} finally {
			if (mergedGraph != null) {
				mergedGraph.shutdown();
			}
		}

		mergedGraph = this.orientUniverseGraphFactory.getTx();	//Exception should flow through.  This is a problem.
		try {
			mergedGraph.begin();
			mergedGraph.getRawGraph().getTransaction().setUsingLog(false);

			childProcessor.postProcess(mergedGraph, statement, references, measurementPeriods, summaryDatas, rateComponentsList);
			mergedGraph.commit();

		} catch (Exception e) {
			mergedGraph.rollback();
			
			throw new StateDumpException("Exception thrown when creating the statement due to " + e.getMessage(), e)
					.addStateDump("universeMergeRequestUuid", universeMergeRequestUuid)
					.addStateDump("deliveryId", deliveryId)
					.addStateDump("uid", uid);
		} finally {
			if (mergedGraph != null) {
				mergedGraph.shutdown();
			}
		}
		
		
	}
----

[Luigi Dell'Aquila]

Hi,

first of all I suggest you to update to 2.1.2 (released a few days ago) and try again.

Do you think you can provide a simple test case to reproduce the issue? I would help a lot on our side

Thanks

Luigi

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[José Cornado]

My two cents:

When I experience these performance drops, it has always been the case that I left too many graph operations accumulate. My ballpark threshold is 48K add/remove ops. It seems to do the job.

[Chris Gomes]

Hi Luigi,
Thank you for your reply.

Actually, we already tried with 2.1.2 and with 2G of max heap and with 1GIG of diskcache for OrientDB.
We had to bump up max heap for application to 2 GIG, which I feel is huge.
With these settings, we were able to process
in 20 hours in a lunix docker container.

Application heap space was a trial and error and with 512M of heap, I was able to run for 12 hours before it quit.

We were hoping that our application would not need such huge heap as it's only one thread that processes our data and one business unit of data at a time. We will be doing query in the future, but it
was not really used by any other thread.

20 hours created 2,610,078 edges and 765,397 vertices. Average time for each business unit of work is 2 secs. Essentially we have processed about 47 entities per second.

I hope there is room for improvement.

Unfortunately, it's hard to reproduce this with simple case.

Any general ideas on how many vertices/edges create/update/read is a good practice in one transaction and any other settings suggestion would be appreciated.

Thanks,
-Chris

[Chris Gomes]

Hi Jose,
Actually in our case, our transaction does not deal with more than at most 3-4K add/update/remove/read operations at a time. Do you mean # of operations or size of objects?

Thanks
-Chris

[José Cornado]

I meant vertices, edges, properties, labels. When I have ran into this it was caused by not doing the bean counting. I work with a relatively simple graph that runs into supernodes quickly. But as luigi said, maybe try to reproduce it outside all that noise

[José Cornado]

I also have found out that if I place a break point before invoking any orientdb method and step over it (with eclipse debugger) I also fall into these performance drops.

Ok. Busy now

[Chris Gomes]

Thank you. I will have to take a close look at # of vertices, edges, properties per transaction. Does not seem like they are huge, but forming  relationships perhaps ends up reading and modifying large edge collections.

On Wed, Sep 16, 2015 at 2:33 PM, José Cornado <jose.c...@gmail.com> wrote:

I meant vertices, edges, properties, labels. When I have ran into this it was caused by not do the bean counting. I work with a realtively simple graph that run into supernodes quickly. But as luigi said, maybe try to reproduce it outside all that noise

--

---
You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/aVUbokCskEs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.