Deserialization issue (InvalidClassException) during communication with remote storage

52 views
Skip to first unread message

Simon Erhardt

unread,
Sep 13, 2018, 8:16:25 AM9/13/18
to OrientDB
Hello community,

we are using OrientDB in our product, and since the update from version 2.1.15 to version 2.2.34, we have the problem described below, which occurs sporadically.
Is there anything we can do to mitigate it or to identify the actual error source?
We are grateful for every kind of help.

- Simon

OrientDB release?
2.2.34

What steps will reproduce the problem?
Unknown - our attempts to reproduce the problem have been very unsatisfying yet. Nonetheless, the problem occurs again and again (without a recognizable pattern), in different runtime environments of different customers, and in different situations. It seems to happen in random situations when the database is accessed. It happens that one query is successful, and the next one fails (with the error below), and then every subsequent query also fails.

Symptoms:
An InvalidClassException occurs sometimes when querying the database:
Caused by: com.orientechnologies.common.io.OIOException: com.o; serializable and externalizable flags conflict
at com.orientechnologies.orient.client.remote.OStorageRemote.handleIOException(OStorageRemote.java:321)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:292)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetry(OStorageRemote.java:204)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:215)
at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1187)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.tinkerpop.blueprints.impls.orient.OrientGraphCommand.execute(OrientGraphCommand.java:49)
at com.mycompany.BaseDAOImpl.query(OrientConnection.java:188)
... 13 common frames omitted
Caused by: java.io.InvalidClassException: com.o; serializable and externalizable flags conflict
at java.base/java.io.ObjectStreamClass.readNonProxy(Unknown Source)
at java.base/java.io.ObjectInputStream.readClassDescriptor(Unknown Source)
at java.base/java.io.ObjectInputStream.readNonProxyDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:436)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:400)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:283)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2365)
at com.orientechnologies.orient.client.remote.OStorageRemote$27.execute(OStorageRemote.java:1211)
at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:207)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:252)
... 19 common frames omitted
 
The BaseDAOImpl.query looks like:
public Iterable<Vertex> query(String query, Object... params) throws PersistenceException {
OrientBaseGraph graph = OrientGraph.getActiveGraph();
try {
OCommandSQL command = new OCommandSQL(query);
graph.clearCache();
return graph.command(command).execute(params);
} catch (Exception e) {
throw new PersistenceException("Query triggered an exception", e);
}
}

It is called in the following context:
OrientGraphFactory factory = new OrientGraphFactory("remote:localhost/mydb", user, password).setupPool(10, 50);
setRequireTransaction(true);
setAutoStartTx(false);
OrientGraph db = factory.getTx();
...
baseDAO.query("select from MyClass");
...
db.shutdown();


If you're using custom settings please provide them below (to dump all the settings run the application using the JVM argument -Denvironment.dumpCfgAtStartup=true):
OrientDB 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x) configuration dump:
- ENVIRONMENT
  + environment.dumpCfgAtStartup = true
  + environment.concurrent = true
  + environment.lockManager.concurrency.level = 64
  + environment.allowJVMShutdown = true
- SCRIPT
  + script.pool.maxSize = 20
- MEMORY
  + memory.useUnsafe = true
  + memory.chunk.size = 2147483647
  + memory.directMemory.safeMode = true
  + memory.directMemory.trackMode = false
  + memory.directMemory.onlyAlignedMemoryAccess = true
- JVM
  + jvm.gc.delayForOptimize = 600
- STORAGE
  + storage.openFiles.limit = 512
  + storage.componentsLock.cache = 10000
  + storage.diskCache.pinnedPages = 20
  + storage.diskCache.bufferSize = 4054
  + storage.diskCache.writeCachePart = 15
  + storage.diskCache.writeCachePageTTL = 86400
  + storage.diskCache.writeCachePageFlushInterval = 25
  + storage.diskCache.writeCacheFlushInactivityInterval = 60000
  + storage.diskCache.writeCacheFlushLockTimeout = -1
  + storage.diskCache.diskFreeSpaceCheckInterval = 5
  + storage.diskCache.diskFreeSpaceCheckIntervalInPages = 2048
  + storage.diskCache.keepState = true
  + storage.diskCache.checksumMode = StoreAndSwitchReadOnlyMode
  + storage.configuration.syncOnUpdate = true
  + storage.compressionMethod = nothing
  + storage.encryptionMethod = nothing
  + storage.encryptionKey = <hidden>
  + storage.makeFullCheckpointAfterCreate = false
  + storage.makeFullCheckpointAfterOpen = false
  + storage.makeFullCheckpointAfterClusterCreate = true
  + storage.trackChangedRecordsInWAL = false
  + storage.useWAL = true
  + storage.wal.syncOnPageFlush = true
  + storage.wal.cacheSize = 3000
  + storage.wal.fileAutoCloseInterval = 10
  + storage.wal.maxSegmentSize = 128
  + storage.wal.maxSize = 4096
  + storage.wal.commitTimeout = 1000
  + storage.wal.shutdownTimeout = 10000
  + storage.wal.fuzzyCheckpointInterval = 300
  + storage.wal.reportAfterOperationsDuringRestore = 10000
  + storage.wal.restore.batchSize = 50000
  + storage.wal.readCacheSize = 1000
  + storage.wal.fuzzyCheckpointShutdownWait = 600
  + storage.wal.fullCheckpointShutdownTimeout = 600
  + storage.wal.path = null
  + storage.diskCache.pageSize = 64
  + storage.diskCache.diskFreeSpaceLimit = 256
  + storage.lowestFreeListBound = 16
  + storage.lockTimeout = 0
  + storage.record.lockTimeout = 2000
  + storage.useTombstones = false
- RECORD
  + record.downsizing.enabled = true
- OBJECT
  + object.saveOnlyDirty = false
- DB
  + db.pool.min = 50
  + db.pool.max = 100
  + db.pool.idleTimeout = 0
  + db.pool.idleCheckDelay = 0
  + db.mvcc.throwfast = false
  + db.validation = true
- NONTX
  + nonTX.recordUpdate.synch = false
  + nonTX.clusters.sync.immediately = manindex
- TX
  + tx.trackAtomicOperations = false
  + tx.pageCacheSize = 12
- INDEX
  + index.embeddedToSbtreeBonsaiThreshold = 40
  + index.sbtreeBonsaiToEmbeddedThreshold = -1
- HASHTABLE
  + hashTable.slitBucketsBuffer.length = 1500
- INDEX
  + index.auto.synchronousAutoRebuild = true
  + index.auto.lazyUpdates = 10000
  + index.flushAfterCreate = true
  + index.manual.lazyUpdates = 1
  + index.durableInNonTxMode = true
  + index.ignoreNullValuesDefault = false
  + index.txMode = FULL
  + index.cursor.prefetchSize = 10000
- SBTREE
  + sbtree.maxDepth = 64
  + sbtree.maxKeySize = 10240
  + sbtree.maxEmbeddedValueSize = 40960
- SBTREEBONSAI
  + sbtreebonsai.bucketSize = 2
  + sbtreebonsai.linkBagCache.size = 100000
  + sbtreebonsai.linkBagCache.evictionSize = 1000
  + sbtreebonsai.freeSpaceReuseTrigger = 0.5
- RIDBAG
  + ridBag.embeddedDefaultSize = 4
  + ridBag.embeddedToSbtreeBonsaiThreshold = 40
  + ridBag.sbtreeBonsaiToEmbeddedToThreshold = -1
- COLLECTIONS
  + collections.preferSBTreeSet = false
- FILE
  + file.trackFileClose = false
  + file.lock = true
  + file.deleteDelay = 10
  + file.deleteRetry = 50
- SECURITY
  + security.userPasswordSaltIterations = 65536
  + security.userPasswordSaltCacheSize = 500
  + security.userPasswordDefaultAlgorithm = PBKDF2WithHmacSHA256
- NETWORK
  + network.maxConcurrentSessions = 1000
  + network.socketBufferSize = 0
  + network.lockTimeout = 15000
  + network.socketTimeout = 15000
  + network.requestTimeout = 3600000
  + network.retry.strategy = auto
  + network.retry = 5
  + network.retryDelay = 500
  + network.binary.loadBalancing.enabled = false
  + network.binary.loadBalancing.timeout = 2000
  + network.binary.maxLength = 16384
  + network.binary.readResponse.maxTimes = 20
  + network.binary.debug = false
  + network.http.installDefaultCommands = true
  + network.http.serverInfo = OrientDB Server v.2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
  + network.http.maxLength = 1000000
  + network.http.streaming = false
  + network.http.charset = utf-8
  + network.http.jsonResponseError = true
  + network.http.jsonp = false
  + network.http.sessionExpireTimeout = 300
  + network.http.useToken = false
  + network.token.secretKey = 
  + network.token.encryptionAlgorithm = HmacSHA256
  + network.token.expireTimeout = 60
- PROFILER
  + profiler.enabled = false
  + profiler.config = null
  + profiler.autoDump.interval = 0
  + profiler.autoDump.type = full
  + profiler.maxValues = 200
  + profiler.memoryCheckInterval = 120000
- SEQUENCE
  + sequence.maxRetry = 100
  + sequence.retryDelay = 200
- STORAGEPROFILER
  + storageProfiler.intervalBetweenSnapshots = 100
  + storageProfiler.cleanUpInterval = 5000
- LOG
  + log.console.level = info
  + log.file.level = fine
- CLASS
  + class.minimumClusters = 0
- LOG
  + log.console.ansi = auto
- CACHE
  + cache.local.impl = com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs
- COMMAND
  + command.timeout = 0
  + command.cache.enabled = false
  + command.cache.evictStrategy = PER_CLUSTER
  + command.cache.minExecutionTime = 10
  + command.cache.maxResultsetSize = 500
- QUERY
  + query.useSoftReferencesInResultSet = false
  + query.parallelAuto = false
  + query.parallelMinimumRecords = 300000
  + query.parallelResultQueueSize = 20000
  + query.scanPrefetchPages = 20
  + query.scanBatchSize = 1000
  + query.scanThresholdTip = 50000
  + query.limitThresholdTip = 10000
  + query.live.support = true
  + query.timeout.defaultStrategy = EXCEPTION
- LUCENE
  + lucene.query.pageSize = 10000
- STATEMENT
  + statement.cacheSize = 100
- SQL
  + sql.graphConsistencyMode = tx
- CLIENT
  + client.channel.maxPool = 200
  + client.connectionPool.waitTimeout = 5000
  + client.channel.dbReleaseWaitTimeout = 10000
  + client.ssl.enabled = false
  + client.ssl.keyStore = null
  + client.ssl.keyStorePass = null
  + client.ssl.trustStore = null
  + client.ssl.trustStorePass = null
- SERVER
  + server.openAllDatabasesAtStartup = false
  + server.channel.cleanDelay = 5000
  + server.cache.staticFile = false
  + server.log.dumpClientExceptionLevel = FINE
  + server.log.dumpClientExceptionFullStackTrace = false
- DISTRIBUTED
  + distributed.dumpStatsEvery = 0
  + distributed.crudTaskTimeout = 10000
  + distributed.maxStartupDelay = 10000
  + distributed.commandTaskTimeout = 120000
  + distributed.commandQuickTaskTimeout = 5000
  + distributed.commandLongTaskTimeout = 86400000
  + distributed.deployDbTaskTimeout = 1200000
  + distributed.deployChunkTaskTimeout = 60000
  + distributed.deployDbTaskCompression = 7
  + distributed.asynchQueueSize = 0
  + distributed.asynchResponsesTimeout = 15000
  + distributed.purgeResponsesTimerDelay = 15000
  + distributed.conflictResolverRepairerChain = quorum,content,majority,version
  + distributed.conflictResolverRepairerCheckEvery = 5000
  + distributed.conflictResolverRepairerBatch = 50
  + distributed.txAliveTimeout = 30000
  + distributed.requestChannels = 1
  + distributed.responseChannels = 1
  + distributed.queueTimeout = 15000
  + distributed.heartbeatTimeout = 10000
  + distributed.checkHealthCanOfflineServer = false
  + distributed.checkHealthEvery = 10000
  + distributed.autoRemoveOfflineServers = 0
  + distributed.publishNodeStatusEvery = 10000
  + distributed.localQueueSize = 10000
  + distributed.dbWorkerThreads = 0
  + distributed.queueMaxSize = 10000
  + distributed.backupDirectory = ../backup/databases
  + distributed.backupTryIncrementalFirst = true
  + distributed.checkIntegrityLastTxs = 16
  + distributed.concurrentTxMaxAutoRetry = 15
  + distributed.atomicLockTimeout = 1000
  + distributed.concurrentTxAutoRetryDelay = 10
- DB
  + db.document.serializer = ORecordSerializerBinary
- CLIENT
  + client.krb5.config = null
  + client.krb5.ccname = null
  + client.krb5.ktname = null
  + client.credentialinterceptor = null
  + client.ci.keyalgorithm = AES
  + client.ci.ciphertransform = AES/CBC/PKCS5Padding
  + client.ci.keystore.file = null
  + client.ci.keystore.password = null
- SECURITY
  + security.createDefaultUsers = true
- SERVER
  + server.security.file = null
- ORIENT
  + orient.initInServletContextListener = true
- JNA
  + jna.disable.system.library = true
- DB
  + db.makeFullCheckpointOnIndexChange = true
  + db.makeFullCheckpointOnSchemaChange = true
- CLIENT
  + client.session.tokenBased = true
- OAUTH2
  + oauth2.secretkey = 
- STORAGE
  + storage.cluster.usecrc32 = false
- LAZYSET
  + lazyset.workOnStream = true
- DB
  + db.mvcc = true
  + db.use.distributedVersion = false
- MVRBTREE
  + mvrbtree.timeout = 0
  + mvrbtree.nodePageSize = 256
  + mvrbtree.loadFactor = 0.7
  + mvrbtree.optimizeThreshold = 100000
  + mvrbtree.entryPoints = 64
  + mvrbtree.optimizeEntryPointsFactor = 1.0
  + mvrbtree.entryKeysInMemory = false
  + mvrbtree.entryValuesInMemory = false
  + mvrbtree.ridBinaryThreshold = -1
  + mvrbtree.ridNodePageSize = 64
  + mvrbtree.ridNodeSaveMemory = false
- TX
  + tx.commit.synch = false
  + tx.autoRetry = 1
  + tx.log.fileType = classic
  + tx.log.synch = false
  + tx.useLog = true
- INDEX
  + index.auto.rebuildAfterNotSoftClose = true
- CLIENT
  + client.channel.minPool = 50
- STORAGE
  + storage.keepOpen = true
- CACHE
  + cache.local.enabled = true
- DISTRIBUTED
  + distributed.shutdownTimeout = 2000

Luigi Dell'Aquila

unread,
Sep 14, 2018, 1:47:37 AM9/14/18
to orient-...@googlegroups.com
Hi Simon,

Are you using the exact same OrientDB version on both the client and the server?

Thanks

Luigi

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Simon Erhardt

unread,
Sep 14, 2018, 3:29:00 AM9/14/18
to orient-...@googlegroups.com
Hi Luigi,

yes, although we use the remote protocol, client and server are run in the same process. Our classpath contains the following libraries:

orientdb-client-2.2.34.jar
orientdb-core-2.2.34.jar
orientdb-graphdb-2.2.34.jar
orientdb-server-2.2.34.jar
orientdb-tools-2.2.34.jar

Simon

You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/8UE029Na4jE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

Luigi Dell'Aquila

unread,
Sep 14, 2018, 4:56:11 AM9/14/18
to orient-...@googlegroups.com
Hi Simon,

Ok, so it must be a bug. Could you please file an issue here https://github.com/orientechnologies/orientdb/issues

Thanks

Luigi

Simon Erhardt

unread,
Sep 14, 2018, 6:02:07 AM9/14/18
to orient-...@googlegroups.com
Thank you Luigi, for looking at it. I filed this one: https://github.com/orientechnologies/orientdb/issues/8537.

Simon
Reply all
Reply to author
Forward
0 new messages