unable to start JG after upgrading elasticsearch

285 views
Skip to first unread message

Haroon Qureshi

unread,
Apr 6, 2020, 4:03:20 PM4/6/20
to JanusGraph users
I recently upgraded to JG 0.5.1 and had to upgrade ElasticSearch as well.  I upgraded ES to 6.8.6.  Now, I'm not able to start JG.  I'm running on Kubernetes and I've attached the log file as well.  The error says it cannot connect to the URL for ES.  I'm able to curl to that IP address and port and get a successful response.  So I'm not sure what is going on.  Any help is appreciated.

Thanks
Haroon

6736 [main] WARN  org.janusgraph.diskstorage.es.rest.RestElasticSearchClient  - Unable to determine Elasticsearch server version. Default to SEVEN.
java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.254.99:9200]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.getMajorVersion(RestElasticSearchClient.java:137)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.<init>(RestElasticSearchClient.java:117)
at org.janusgraph.diskstorage.es.rest.RestClientSetup.getElasticSearchClient(RestClientSetup.java:107)
at org.janusgraph.diskstorage.es.rest.RestClientSetup.connect(RestClientSetup.java:75)
at org.janusgraph.diskstorage.es.ElasticSearchSetup$1.connect(ElasticSearchSetup.java:51)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.interfaceConfiguration(ElasticSearchIndex.java:437)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:324)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:427)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:150)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1359)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:146)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:77)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:69)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:103)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.254.99:9200]
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:628)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:894)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:184)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:214)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
7745 [main] WARN  org.apache.tinkerpop.gremlin.server.GremlinServer  - Graph [graph] configured at [/janusgraph-config/janusgraph.properties] could not be instantiated and will not be available in Gremlin Server.  GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:81)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:69)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:103)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.lambda$new$0(DefaultGraphManager.java:57)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager.<init>(DefaultGraphManager.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:80)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:122)
at org.apache.tinkerpop.gremlin.server.GremlinServer.<init>(GremlinServer.java:86)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:77)
... 13 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:440)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:427)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:150)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1359)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:146)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:161)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:132)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:112)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
... 26 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.254.99:9200]
at org.janusgraph.diskstorage.es.ElasticSearchIndex.checkClusterHealth(ElasticSearchIndex.java:341)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:326)
... 31 more
Caused by: java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.254.99:9200]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.clusterHealthRequest(RestElasticSearchClient.java:156)
at org.janusgraph.diskstorage.es.ElasticSearchIndex.checkClusterHealth(ElasticSearchIndex.java:339)
... 32 more
Caused by: java.net.ConnectException: Timeout connecting to [janusgraph-051-elasticsearch-client/10.63.254.99:9200]
at org.apache.http.nio.pool.RouteSpecificPool.timeout(RouteSpecificPool.java:169)
at org.apache.http.nio.pool.AbstractNIOConnPool.requestTimeout(AbstractNIOConnPool.java:628)
at org.apache.http.nio.pool.AbstractNIOConnPool$InternalSessionRequestCallback.timeout(AbstractNIOConnPool.java:894)
at org.apache.http.impl.nio.reactor.SessionRequestImpl.timeout(SessionRequestImpl.java:184)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processTimeouts(DefaultConnectingIOReactor.java:214)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:158)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)
logs.out

Oleksandr Porunov

unread,
Apr 7, 2020, 2:39:47 AM4/7/20
to JanusGraph users
Hi,

Did you use multitype indexes in ElasticSearch? If so, you are required to reindex your data because JanusGraph doesn't support EOL multitype indexes.

Haroon Qureshi

unread,
Apr 7, 2020, 7:57:39 AM4/7/20
to janusgra...@googlegroups.com
This is a new installation and an empty database.

Haroon A. Qureshi

From: janusgra...@googlegroups.com <janusgra...@googlegroups.com> on behalf of Oleksandr Porunov <alexandr...@gmail.com>
Sent: Tuesday, April 7, 2020 1:39:47 AM
To: JanusGraph users <janusgra...@googlegroups.com>
Subject: Re: unable to start JG after upgrading elasticsearch
 
--
You received this message because you are subscribed to the Google Groups "JanusGraph users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/f2ec08a5-eb9c-4379-8b1c-2d3ef6188680%40googlegroups.com.

Oleksandr Porunov

unread,
Apr 7, 2020, 8:26:52 PM4/7/20
to JanusGraph users
Following logs, your JanusGraph instance can't connect to your ElasticSearch instance. The connection step timeouts.
What does your ElasticSearch logs show?
Did you set index.[X].elasticsearch.health-request-timeout or storage.connection-timeout?
Are you shure that your JanusGraph instance isn't blocked for IN or OUT requests by any firewall or iptables? Does you ES port is open for JanusGraph request? Also, you may want to check SELINUX settings as well.
I can't reproduce your scenario with the same JanusGraph and ElasticSearch version. In my situation it connects without any problems in both private and public networks.


On Tuesday, April 7, 2020 at 4:57:39 AM UTC-7, Haroon Qureshi wrote:
This is a new installation and an empty database.

Haroon A. Qureshi

To unsubscribe from this group and stop receiving emails from it, send an email to janusgra...@googlegroups.com.

Haroon Qureshi

unread,
Apr 8, 2020, 3:03:45 PM4/8/20
to JanusGraph users
I've attached the logs from ES.  It appears there is an error with ES Discovery.

Haroon A. Qureshi
--
M: +1.312.927.0276

From: janusgra...@googlegroups.com <janusgra...@googlegroups.com> on behalf of Oleksandr Porunov <alexandr...@gmail.com>
Sent: Tuesday, April 7, 2020 7:26 PM
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/cffe0e0b-384a-411e-8fe9-a87fb8e6688b%40googlegroups.com.
es.log

Oleksandr Porunov

unread,
Apr 8, 2020, 3:30:51 PM4/8/20
to JanusGraph users
Yes. As you can see, ElasticSearch client cannot discover other master nodes. Most likely, your ElasticSearch cluster state has some problems. You should resolve your problems with ElasticSearch before atempting to use it for JanusGraph. When your ElasticSearch is up and running and has no discovery problems, try again connecting JanusGraph to ElasticSearch. You may get help with your ES deployment here https://discuss.elastic.co/
Also, if you need just to play with JanusGraph by using CQL + ElasticSearch, you can download JanusGraph 0.5.1 full release and start janusgraph.sh. It has a properly configured ES and CQL for testing purposes.

On Wednesday, April 8, 2020 at 12:03:45 PM UTC-7, Haroon Qureshi wrote:
I've attached the logs from ES.  It appears there is an error with ES Discovery.

Haroon A. Qureshi
--
M: +1.312.927.0276

Haroon Qureshi

unread,
Apr 8, 2020, 3:43:32 PM4/8/20
to janusgra...@googlegroups.com
What version of ES is part of the 0.5.1 release?  

Haroon A. Qureshi

From: janusgra...@googlegroups.com <janusgra...@googlegroups.com> on behalf of Oleksandr Porunov <alexandr...@gmail.com>
Sent: Wednesday, April 8, 2020 2:30:51 PM
To unsubscribe from this group and stop receiving emails from it, send an email to janusgraph-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-users/a2bd5d53-ae38-4b39-a065-3ba74fbe2f98%40googlegroups.com.

Oleksandr Porunov

unread,
Apr 8, 2020, 3:51:29 PM4/8/20
to JanusGraph users
We use ES 7.6.1 client version in JanusGraph 0.5.1. Both 6.x and 7.x ElasticSearch versions are supported and tested. Your ES version should work as well as newer ES versions.


On Wednesday, April 8, 2020 at 12:43:32 PM UTC-7, Haroon Qureshi wrote:
What version of ES is part of the 0.5.1 release?  

Haroon A. Qureshi

Reply all
Reply to author
Forward
0 new messages