sgadmin hangs at "wait for YELLOW clusterstate"

[Chintan Patel]

Hello,

I'm trying to implement search guard in ES 5.0.2

So i installed ES 5.0.2 with apt and install plugin with 

• sudo bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:5.0.2-8

I created certificate through pki scripts and put those in /etc/elasticsearch. Here is my es config

http://pastie.org/private/7p8egfx8mpkpcotmpsirw

I started es and this is log

http://pastie.org/private/q38zdouqiw8a7g2rv0hkpg

https://localhost:9200 gives me "Search Guard not initialized (SG11). See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md"

So then i run sgadmin like

sudo ./sgadmin.sh -cd ../sgconfig/ -ks /opt/lampp/htdocs/search-guard-ssl/example-pki-scripts/kirk-keystore.jks -ts /opt/lampp/htdocs/search-guard-ssl/example-pki-scripts/truststore.jks -tspass changeit -kspass changeit -nhnv

But it stuck at 

Search Guard Admin v5

Will connect to localhost:9300 ... done

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

What can be the issue ? Because same elasticsearch.yml, certificate and sgadmin tool command is working with search guard bundle.

Thanks,

Chintan Patel

[Jochen Kressin]

If you do not use the standard cluster name "elasticsearch", either specify your cluster name with the -cn option, or use the -icl option to ignore the cluster name altogether

[Search Guard]

try adding -icl to the sgadmin command

[Chintan Patel]

I tried that too. But it's not working. As i said, same elasticsearch.yml working in search guard bundle.

[Chintan Patel]

I just found that if i run ES from source code and do all configurations with key then sgadmin is working. So the problem is when i install ES from package.

[Search Guard]

thats sounds strange

Can you post (or mail) your source code?

[Chintan Patel]

I mean from zip file. :)

[Seldon Sun]

I have observed the same error on RHEL7. ES 5.0.2 is installed from the official repo (RPM) package. sgadmin.sh will loop with these errors:

Search Guard Admin v5

Will connect to x.x.x.x:9300 ... done

Contacting elasticsearch cluster 'ss-es-test-cluster' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{Dx4yUSaKRpS7Ke5tSOqwwA}{x.x.x.x}{x.x.x.x:9300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{Dx4yUSaKRpS7Ke5tSOqwwA}{x.x.x.x}{x.x.x.x:9300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{Dx4yUSaKRpS7Ke5tSOqwwA}{x.x.x.x}{x.x.x.x:9300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{Dx4yUSaKRpS7Ke5tSOqwwA}{x.x.x.x}{x.x.x.x:9300}]. This is not an error, will keep on trying ...

in ES log it says:

[2017-01-03T15:12:03,156][WARN ][o.e.t.n.Netty4Transport  ] [ssestest01] exception caught on transport layer [[id: 0xe16c2ce0, L:/x.x.x.x:9300 ! R:/x.x.x.x:54242]], closing connection

io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)

        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:375) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325) ~[netty-codec-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:234) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:234) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1329) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:255) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:908) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:744) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) [netty-common-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440) [netty-transport-4.1.5.Final.jar:4.1.5.Final]

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.5.Final.jar:4.1.5.Final]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)

        at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1103) ~[elasticsearch-5.0.2.jar:5.0.2]

        at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]

        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]

        ... 20 more

Any thoughts?

[SG]

Please make sure that

1) the JVM Version you run your Elasticsearch nodes on
- are exactly the same on all nodes
- are exactly the same you run sgadmin with

2) The version of sgadmin fits the Elasticsearch version.

sgadmin talks to elasticsearch with elasticsearchs own binary protocol (so called "Transport protocol").
This only works if JVM versions are the same and the libraries match. If unsure you can also download sgadmin standalone from here for ES 5.0.2:
http://search.maven.org/remotecontent?filepath=com/floragunn/search-guard-5/5.0.2-9/search-guard-5-5.0.2-9-sgadmin-standalone.zip

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4d3e736a-a381-40f7-9a34-27d1b2d711dc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Seldon Sun]

Thanks, SG.

I have 4 ES nodes, all of them are version 5.0.2-1 running on RHEL 7.3. The JVM is java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64 and they are the same across all 4 nodes. 

SG is installed via command "./elasticsearch-plugin install -b com.floragunn:search-guard-5:5.0.2-9", so I believe it is the right version for ES 5.0.2-1.

I ran sgadmin.sh on one of the nodes against that node's IP. What's the supported java version for SG?

[Search Guard]

Please use one of the standalone sgadmins below. They will write a diagnostic trace file which should provide more info.

When running sgadmin you should see something like "Diagnostic trace written to: xxx" Please post this file (or send it via mail directly because it can contain sensitive informations). The email address you can find here: https://github.com/floragunncom

https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-5/5.0.2-10-SNAPSHOT/search-guard-5-5.0.2-10-20170104.103345-5-sgadmin-standalone.zip

https://oss.sonatype.org/content/repositories/snapshots/com/floragunn/search-guard-5/5.0.2-10-SNAPSHOT/search-guard-5-5.0.2-10-20170104.103345-5-sgadmin-standalone.tar.gz