Caused by: ElasticsearchException[Is a directory Expected file!]

[Anthony Cleaves]

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version

* Used enterprise modules, if any

* JVM version and operating system version

* Search Guard configuration files

* Elasticsearch log messages on debug level

Hello, I am trying to add this code into ansible to make it a bit easier to deploy in future.

I have ran into an issue, when I run the sgadmin I get the following error:

root@ip-172-31-27-116:/usr/share/elasticsearch/plugins/search-guard-5/tools# ./sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-5/sgconfig -cn actual-cluster -cert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -cacert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -nhnv --diagnose
WARNING: JAVA_HOME not set, will use /usr/bin/java
Search Guard Admin v5
Will connect to localhost:9300 ... done


### LICENSE NOTICE Search Guard ###


If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)


* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging


In case of any doubt mail to <sales@floragunn.com>
###################################
ERR: An unexpected ElasticsearchException occured: Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:
ElasticsearchException[Failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[Is a directory: /usr/share/elasticsearch/plugins/search-guard-5/tools Expected file!];
 at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:434)
 at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:103)
 at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:101)
 at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:126)
 at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:254)
 at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:715)
 at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:370)
 at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:109)
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:423)
 ... 7 more
Caused by: ElasticsearchException[Is a directory: /usr/share/elasticsearch/plugins/search-guard-5/tools Expected file!]
 at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkStorePath(DefaultSearchGuardKeyStore.java:686)
 at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:271)
 at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:150)
 at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:237)
 ... 12 more

Does anyone understand why this is happening? I am trying to use SSL certificates rather than keystores that's the only noticable change I have done, the crts are readable.

I don't understand what it's doing in order to try and use that folder instead of a file, if I change location on the machine the folder in the error will change to the current destination

So if I cd /home/blah, the error would be "home/blah Expected file!"

[SG]

you're missing the -key option.

We know already that the error message is misleading, this will be fixed in the next version

> In case of any doubt mail to <sa...@floragunn.com>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/4d177704-82e6-455a-8696-de261cbddfe2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Anthony Cleaves]

Now you mention it, that is blindly obvious haha. Thanks.

Can search guard be installed on a single node or does it always expect a cluster? Just of curiosity for my testing (I will run it in a cluster on production)

[SG]

Works also on a single node (but you need configure SSL/TLS anyway)

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/e4517ca2-185b-4735-a20d-1b1d44df452e%40googlegroups.com.

[Anthony Cleaves]

Ah yes, I am seeing different errors now. Can you elaborate on this error for me?

[2017-07-26T15:18:44,134][WARN ][c.f.s.a.BackendRegistry  ] Transport authentication finally failed for CN=*.x,O=x,L=x,x=x,C=GB
[2017-07-26T15:18:44,134][ERROR][c.f.s.t.SearchGuardRequestHandler] Cannot authenticate null

(I removed sensitive data)

[Anthony Cleaves]

Scrap that, I think I found the problem. Thanks for everything!

[SG]

no problem

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/97e1a23f-229e-42e0-b355-4f956c29d087%40googlegroups.com.

[Anthony Cleaves]

I seem to be having issues with two clients clustering. The master is currently saying the following:

[2017-07-26T16:39:53,456][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating header

Where as the node is saying

Caused by: org.elasticsearch.transport.RemoteTransportException: [34.248.89.180-x-x.x-x.com][172.31.27.116:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: bad header found

[SG]

pls provide your elasticsearch.yml

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/f461c3f5-dce0-484d-8937-a2ca63fe6f43%40googlegroups.com.

[Anthony Cleaves]

Sure, I have x'd out some sensitive info, i can always private message you if you prefer with the complete file.

root@ip-172-31-27-116:~# cat /etc/elasticsearch/elastic-master.x-x.com/elasticsearch.yml 


cluster.name: actual-cluster
discovery.zen.ping.unicast.hosts:
- 172.31.27.116:9300
- 172.31.22.225:9300
http.port: 9200
node.data: false
node.master: true
transport.tcp.port: 9300






node.name: 34.248.89.180-elastic-master.x-x.com


network.host: 0.0.0.0




searchguard.ssl.transport.pemkey_filepath: x-x.com.key
searchguard.ssl.transport.pemcert_filepath: x-x.com.crt
searchguard.ssl.transport.pemtrustedcas_filepath: x-x.com.crt
searchguard.ssl.http.pemkey_filepath: x-x.com.key
searchguard.ssl.http.pemcert_filepath: x-x.com.crt
searchguard.ssl.http.pemtrustedcas_filepath: x-x.com.crt
searchguard.authcz.admin_dn:
        - CN=*.x-x.com,O=x x PLC,L=x,ST=x,C=GB
searchguard.ssl.transport.enforce_hostname_verification: false
#################################### Paths ####################################


# Path to directory containing configuration (this file and logging.yml):
path.conf: /etc/elasticsearch/elastic-master.x-x.com


path.data: /var/lib/elasticsearch/34.248.89.180-elastic-master.x-x.com


path.logs: /var/log/elasticsearch/34.248.89.180-elastic-x.x-x.com

[Anthony Cleaves]

If I remove

searchguard.ssl.transport.enforce_hostname_verification: false

I see

" SSL Problem Received fatal alert: certificate_unknon"

[Search Guard]

How did you create your certificates? Make sure you either have the OID in your node certificates or nodes_dn is defined in elasticsearch.yml

More on that you will find here: https://github.com/floragunncom/search-guard-docs/blob/master/tls_node_certificates.md

"Bad Header" means that one node is not trusting your others and that is because of ssl is not configured properly.

[Anthony Cleaves]

Interesting, I use a standard wildcard certificate from globalsign. It's been converted to pkcs8.

I guess I will run some openssl commands on it to find out what's going on. The nodes use the same cert as the master.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/TmyxeWwCuYA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/53df2662-b436-4caf-b845-c403424f9422%40googlegroups.com.

[Anthony Cleaves]

Ok, so that is fixed.

You were spot on (as usual) the issue was I was only specifying a master dn, instead of using nodes as it's a wildcard all must use node.

Now when running sgadmin on the final run, I get this:

Clustername: actual-cluster

Clusterstate: GREEN

Number of nodes: 2

Number of data nodes: 1

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Trace:

ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]

at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)

at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)

at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)

at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)

at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)

at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:177)

at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:191)

at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:139)

at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:336)

at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)

at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)

at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)

at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)

at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)

at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)

at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

at java.lang.Thread.run(Thread.java:748) 

The command I am running is below:

/bin/bash /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-5/sgconfig -cn actual-cluster -cert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -cacert /etc/elasticsearch/elastic-master.x-x.com/x-x.com.crt -key /etc/elasticsearch/elastic-master.x-x.com/x-x.com.key -nhnv

[SG]

Also SSL related, this time its about the client admin certificate:

https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
https://github.com/floragunncom/search-guard/issues/366

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/77862480-428c-4e82-a808-1ae1814c6733%40googlegroups.com.

[Anthony Cleaves]

I'm a little confused, does the master need both a master and a node definition?

Currently in es.yml for master I have a master dn, and for my node a node dn.

[SG]

What you mean with "master" and "node"?
In terms of Search Guard your ES elected master (and all master eligible nodes) are not different from any other node (data node, ingest node, client node ...)

That means that the searchguard related configuration for all nodes is typically identical in elasticsearch.yml and you can point sgadmin against any node
(wheter its the elected master, master eligible, data node, ingest node, client node ...).

BTW: For a two node cluster you should not specify a dedicated master node, this makes no sense. For a typical production setup you will normally have 3 dedicated master eligible nodes and a minimum of 2 data nodes.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9d4eeabd-009e-452d-83e8-250b48a97fc2%40googlegroups.com.

[Anthony Cleaves]

Ref the node numbers, this is purely dev. I am just trying to ansible this whole installation.

So in each configuration for my elected masted and my elected node, I have the following in my elasticsearc.yml file:

searchguard.authcz.admin_dn:

      - CN=*.x-x.com,O=x x PLC,L=x,ST=x,C=GB

searchguard.nodes_dn:

      - CN=*.x-x.com,O=x x PLC,L=x,ST=x,C=GB

Both are identical, as both are using the wildcard certificate.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/9d4eeabd-009e-452d-83e8-250b48a97fc2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/TmyxeWwCuYA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/23993B7D-7A39-4E40-B513-B69BA12A9139%40search-guard.com.

[Anthony Cleaves]

Using the above gives me the error described earlier:

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

Heres some openssl for the cert

        Subject: C=GB, ST=x, L=x, O=x x PLC, CN=*.x-x.com

Which matches the yml

[Anthony Cleaves]

The reason I asked about what needs to be in the node elasticsearch.ym was because of this:

"All certificate DNs listed here are considered valid node certificates. Wildcards and regular expressions are supported. If you use this approach, please make sure to list only node certificates."

So is this documentation wrong, or am I not explaining this very well?

At the moment, I have the following on both of my nodes:

searchguard.ssl.transport.pemkey_filepath: globalsign_x-x.com.key
searchguard.ssl.transport.pemcert_filepath: globalsign_x-x.com.crt
searchguard.ssl.transport.pemtrustedcas_filepath: globalsign_x-x.com.crt
searchguard.ssl.http.pemkey_filepath: globalsign_x-x.com.key
searchguard.ssl.http.pemcert_filepath: globalsign_x-x.com.crt
searchguard.ssl.http.pemtrustedcas_filepath: globalsign_x-x.com.crt
searchguard.authcz.admin_dn:
      - CN=*.x-x.com,O=x x x,L=x,ST=x,C=GB
searchguard.nodes_dn:

      - CN=*.x-x.com,O=x x x,L=x,ST=x,C=GB

[Anthony Cleaves]

Do I need to use these parameters?

https://github.com/floragunncom/search-guard-docs/blob/master/tls_openssl.md

• 
searchguard.ssl.transport.enable_openssl_if_available: true
• searchguard.ssl.http.enable_openssl_if_available: true