master, replica distributed role IllegalStateException

[defmi...@gmail.com]

I am experimenting with the master/replica distributed roles in https://github.com/orientechnologies/orientdb/issues/3772 in v2.1-rc6. I am basically trying to follow the setup at https://orientdb.com/docs/last/Tutorial-Setup-a-distributed-database.html with modified settings of default-distributed-db-config.json as described in the github ticket.  On the 2nd node, I have only gotten something like

2015-08-04 00:23:50:728 INFO  [testnode2] Publishing online status for database testnode2.MovieRatings... [OHazelcastDistributedDatabase]Exception in thread "main" java.lang.IllegalStateException: Current database instance (com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@6d1310f6) is not active on current thread (Thread[main,5,main]). Current active database is: com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@3228d990

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.checkIfActive(ODatabaseDocumentTx.java:3133)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.close(ODatabaseDocumentTx.java:1128)

at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1057)

at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:924)

at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1337)

at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:184)

at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:964)

at com.orientechnologies.orient.server.OServer.activate(OServer.java:333)

at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)

On the 2nd node, when I try to connect locally with the console i get

Connecting to database [remote:localhost/MovieRatings] with user 'yourUsername'...Error during acquiring of connection to URL 127.0.0.1:2424/MovieRatings

com.orientechnologies.common.io.OIOException: Error on connecting to 127.0.0.1:2424/MovieRatings

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:244)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:80)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:77)

at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:91)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:101)

at com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(OStorageRemote.java:2096)

at com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1839)

at com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:224)

at com.orientechnologies.orient.client.remote.OStorageRemoteThread.open(OStorageRemoteThread.java:89)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:249)

at com.orientechnologies.orient.console.OConsoleDatabaseApp.connect(OConsoleDatabaseApp.java:232)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:375)

at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:240)

at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:129)

at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:64)

Caused by: com.orientechnologies.orient.enterprise.channel.binary.ONetworkProtocolException: Cannot read protocol version from remote server /127.0.0.1:2424: java.io.EOFException

at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.<init>(OChannelBinaryAsynchClient.java:98)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:231)

... 18 more

Can not open database with url 127.0.0.1:2424/MovieRatings

com.orientechnologies.common.io.OIOException: Cannot open a connection to remote server: 127.0.0.1:2424/MovieRatings

at com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(OStorageRemote.java:2107)

at com.orientechnologies.orient.client.remote.OStorageRemote.openRemoteDatabase(OStorageRemote.java:1839)

at com.orientechnologies.orient.client.remote.OStorageRemote.open(OStorageRemote.java:224)

at com.orientechnologies.orient.client.remote.OStorageRemoteThread.open(OStorageRemoteThread.java:89)

at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:249)

at com.orientechnologies.orient.console.OConsoleDatabaseApp.connect(OConsoleDatabaseApp.java:232)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:375)

at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:240)

at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:129)

at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:64)

Caused by: com.orientechnologies.common.io.OIOException: Error on connecting to 127.0.0.1:2424/MovieRatings

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:244)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:80)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager$1.createNewResource(ORemoteConnectionManager.java:77)

at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:91)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:101)

at com.orientechnologies.orient.client.remote.OStorageRemote.getAvailableNetwork(OStorageRemote.java:2096)

... 13 more

Caused by: com.orientechnologies.orient.enterprise.channel.binary.ONetworkProtocolException: Cannot read protocol version from remote server /127.0.0.1:2424: java.io.EOFException

at com.orientechnologies.orient.enterprise.channel.binary.OChannelBinaryAsynchClient.<init>(OChannelBinaryAsynchClient.java:98)

at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.createNetworkConnection(ORemoteConnectionManager.java:231)

... 18 more

Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot create a connection to remote server address(es): [127.0.0.1:2424]

Any tips on getting this to work?

[Giulia Brignoli]

Hi.

I tried to recreate your problem, but it works for me.

It might be a wrong configuration of your file hazelcast.xml, I've configured that:

?xml version="1.0" encoding="UTF-8"?>

<!-- ~ Copyright (c) 2008-2012, Hazel Bilisim Ltd. All Rights Reserved. ~ 

~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may 

not use this file except in compliance with the License. ~ You may obtain 

a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ 

~ Unless required by applicable law or agreed to in writing, software ~ distributed 

under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES 

OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for 

the specific language governing permissions and ~ limitations under the License. -->

<hazelcast

xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.0.xsd"

xmlns="http://www.hazelcast.com/schema/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<group>

<name>orientdb</name>

<password>orientdb</password>

</group>

<network>

    <port auto-increment="true">2434</port>

    <join>

      <multicast enabled="false">

        <multicast-group>235.1.1.1</multicast-group>

        <multicast-port>2434</multicast-port>

      </multicast>

      <tcp-ip enabled="true">

        <member>10.0.22.157:2434</member>

        <member>10.0.22.146:2434</member>

      </tcp-ip>

     </join>

  </network>

<executor-service>

<pool-size>16</pool-size>

</executor-service>

</hazelcast>

Regards,

Giulia

[Mike Koh]

Perhaps there is a subtle difference in what we're doing.  Below is the order of what I'm doing. Let me know if there is something you are doing differently.

2 instances, ubuntu 14.04

- On both, install ubuntu 14.04, install Oracle Java 1.8.0_51 64bit

- On both, extract orientdb-community-2.1-rc6.tar.gz (which includes GratefulDeadConcerts db).

- On both, modify hazelcast.xml as shown in your example (i.e. 2 manually specified ips)

- On both, modify default-distributed-db-config.json to be similar to the example in the github ticket https://github.com/orientechnologies/orientdb/issues/3772

{
    "autoDeploy": true,
    "hotAlignment": false,
    "offlineMsgQueueSize" : 0,
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "servers": {
      "testnode1" : "master",
      "*": "replica"
     },
    "clusters": {
        "internal": {
        },
        "index": {
        },
        "*": {
            "servers" : [ "<NEW_NODE>" ]
        }
    }
}

- On first instance, cd /opt/orientdb-community-2.1-rc6/bin . Then, ./dserver.sh . specify a root password. Interactively set nodename to "testnode1".  /opt/orientdb-community-2.1-rc6/databases/GratefulDeadConcerts/distributed-config.json updates to new settings and
INFO  [testnode1] received updated status testnode1.GratefulDeadConcerts=ONLINE [OHazelcastPlugin]

- On second instance, cd /opt/orientdb-community-2.1-rc6/bin . Then, ./dserver.sh . specify a root password. Interactively set nodename to "testnode2".

testnode1 creates backup of GratefulDeadConcerts and sends it to testnode2. testnode2 installs it, and then I get the exception

[testnode2] Publishing online status for database testnode2.GratefulDeadConcerts... [OHazelcastDistributedDatabase]Exception in thread "main" java.lang.IllegalStateException: Current database instance (com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@3f3c966c) is not active on current thread (Thread[main,5,main]). Current active database is: com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@11ee02f8

    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.checkIfActive(ODatabaseDocumentTx.java:3133)
    at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.close(ODatabaseDocumentTx.java:1128)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabaseFromNetwork(OHazelcastPlugin.java:1057)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installDatabase(OHazelcastPlugin.java:924)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabases(OHazelcastPlugin.java:1337)
    at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:184)
    at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:964)
    at com.orientechnologies.orient.server.OServer.activate(OServer.java:333)
    at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)

These are AWS instances if that makes a difference (don't think it should because we are manually specifying IP).

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[defmi...@gmail.com]

I see the same problem on orientdb community 2.1.0.

To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscribe@googlegroups.com.

[defmi...@gmail.com]

Has anybody gotten this feature to work? If so, any tips?

[Luca Garulli]

Please could you try with last 2.1.1-SNAPSHOT?

https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/2.1.1-SNAPSHOT/orientdb-community-2.1.1-20150809.182628-3-distribution.tar.gz

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

On 10 August 2015 at 21:42, <defmi...@gmail.com> wrote:

Has anybody gotten this feature to work? If so, any tips?

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.

[defmi...@gmail.com]

Same error with orientdb-community-2.1.1-20150809.182628-3-distribution.tar.gz .

testnode1 backs up, transfer to testnode2

2015-08-10 23:48:32:375 INFO  {db=GratefulDeadConcerts} [testnode2] installed database 'GratefulDeadConcerts' [OHazelcastPlugin]

2015-08-10 23:48:32:379 INFO  {db=GratefulDeadConcerts} [testnode2] found no previous messages in queue orientdb.node.testnode2.GratefulDeadConcerts.request [OHazelcastDistributedMessageService]

2015-08-10 23:48:32:772 INFO  [testnode2] Publishing online status for database testnode2.GratefulDeadConcerts... [OHazelcastDistributedDatabase]Exception in thread "main" java.lang.IllegalStateException: Current database instance (com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@7561db12) is not active on current thread (Thread[main,5,main]). Current active database is: com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx@3301500b

[defmi...@gmail.com]

I did the same process without modifying the default-distributed-db-config.json which worked. So more specifically,

2 instances, ubuntu 14.04

- On both, install ubuntu 14.04, install Oracle Java 1.8.0_51 64bit

- On both, extract orientdb-community-2.1.1-20150809.182628-3-distribution.tar.gz (which includes GratefulDeadConcerts db).

- On both, modify hazelcast.xml as shown in your example (i.e. 2 manually specified ips)

- On first instance, cd /opt/orientdb-community-2.1.1-SNAPSHOT/bin . Then, ./dserver.sh . specify a root password. Interactively set nodename to "testnode1".  /opt/orientdb-community-2.1-rc6/databases/GratefulDeadConcerts/distributed-config.json updates to new settings and 

INFO  [testnode1] received updated status testnode1.GratefulDeadConcerts=ONLINE [OHazelcastPlugin]

- On second instance, cd /opt/orientdb-community-2.1.1-SNAPSHOT/bin . Then, ./dserver.sh . specify a root password. Interactively set nodename to "testnode2".

testnode1 creates backup of GratefulDeadConcerts and sends it to testnode2. testnode2 installs db.  Both nodes have GratefulDeadConcerts=ONLINE with no java.lang.IllegalStateException.

[Luca Garulli]

Hi,

So the problem was in default-distributed-db-config.json file?

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

--

[defmi...@gmail.com]

The unmodified default-distributed-db-config.json works (i.e. without "replica" role enabled).

{

  "autoDeploy": true,

  "hotAlignment": false,

  "executionMode": "undefined",

  "readQuorum": 1,

  "writeQuorum": 2,

  "failureAvailableNodesLessQuorum": false,

  "readYourWrites": true,

  "servers": {

    "*": "master"

  },

  "clusters": {

    "internal": {

    },

    "index": {

    },

    "*": {

      "servers": ["<NEW_NODE>"]

    }

  }

}




Enabling the replica role like below and shown at https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Forientechnologies%2Forientdb%2Fissues%2F3772&sa=D&sntz=1&usg=AFQjCNGUPw8mYKa3GfIN3Sc5nvSXMvUZtQ results in java.lang.IllegalStateException on testnode2 after the db autodeploys.

{
    "autoDeploy": true,
    "hotAlignment": false,
    "offlineMsgQueueSize" : 0,
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "servers": {
      "testnode1" : "master",
      "*": "replica"
     },
    "clusters": {
        "internal": {
        },
        "index": {
        },
        "*": {
            "servers" : [ "<NEW_NODE>" ]
        }
    }
}




[Luca Garulli]

Ok, this is definitely a bug. Just opened https://github.com/orientechnologies/orientdb/issues/4775 to track it.

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

[Luca Garulli]

Fixed in last "2.1.x" and "develop" branches, respectively 2.1.1-SNAPSHOT and 2.2.0-SNAPSHOT.

Thanks for reporting it.

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

[defmi...@gmail.com]

I downloaded orientdb-community-2.1.1-20150812.003537-6-distribution.tar.gz and no longer get the IllegalStateException. Yay!

I'm wondering if the autodeploy logic should be changed also. That is, a "replica" shouldn't autodeploy to a "master". I can't think of a case where that would be desirable.  I was able to make this happen by:

- turn on testnode1 (master)

- turn on testnode2 (replica). testnode1 (master) autodeploys db to testnode2 (replica).

- turn off testnode1

- turn on testnode1 (master). testnode2 (replica) autodeploys db to testnode1 (master).