Unable to do Cross Cluster Search

[Venkata Naresh]

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version

6.4.3

* Installed and used enterprise modules, if any

* JVM version and operating system version

Java 8, Linux 18

* Search Guard configuration files

##################Schema################################

searchguard.enterprise_modules_enabled: true

searchguard.ssl.transport.pemcert_filepath: esnode.pem

searchguard.ssl.transport.pemkey_filepath: esnode-key.pem

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemcert_filepath: esnode.pem

searchguard.ssl.http.pemkey_filepath: esnode-key.pem

searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem

searchguard.allow_unsafe_democertificates: true

searchguard.allow_default_init_sgindex: true

searchguard.authcz.admin_dn:

  - CN=kirk,OU=client,O=client,L=test, C=de

searchguard.audit.type: internal_elasticsearch

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: ["sg_all_access"]

cluster.routing.allocation.disk.threshold_enabled: false

cluster.name: searchguard_demo

discovery.zen.minimum_master_nodes: 1

node.max_local_storage_nodes: 3

xpack.security.enabled: false

#####################################################################

* Elasticsearch log messages on debug level

org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster [cluster_two]]

Caused by: org.elasticsearch.transport.ConnectTransportException: [][x.x.x.x:9300] general node connection failure

at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:688) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:124) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:348) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:458) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:455) [elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:443) [elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.3.jar:6.4.3]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.3.jar:6.4.3]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

Caused by: java.lang.IllegalStateException: handshake failed

at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1680) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:654) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:124) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:348) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.lambda$collectRemoteNodes$2(RemoteClusterConnection.java:458) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.common.util.CancellableThreads.executeIO(CancellableThreads.java:105) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler.collectRemoteNodes(RemoteClusterConnection.java:455) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.RemoteClusterConnection$ConnectHandler$1.doRun(RemoteClusterConnection.java:443) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.4.3.jar:6.4.3]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_191]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_191]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.4.3.jar:6.4.3]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]

at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]

Caused by: org.elasticsearch.transport.TransportException: connection reset

at org.elasticsearch.transport.TcpTransport.cancelHandshakeForChannel(TcpTransport.java:1717) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.transport.TcpTransport.lambda$openConnection$12(TcpTransport.java:651) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.action.ActionListener.lambda$wrap$0(ActionListener.java:82) ~[elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) [elasticsearch-6.4.3.jar:6.4.3]

at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:96) ~[elasticsearch-6.4.3.jar:6.4.3]

at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_191]

at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_191]

at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_191]

at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) ~[?:1.8.0_191]

at org.elasticsearch.transport.netty4.NettyTcpChannel.lambda$new$0(NettyTcpChannel.java:42) ~[?:?]

at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) ~[?:?]

at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) ~[?:?]

at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) ~[?:?]

at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) ~[?:?]

at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]

at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) ~[?:?]

at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1148) ~[?:?]

at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:764) ~[?:?]

at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:740) ~[?:?]

at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:611) ~[?:?]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:85) ~[?:?]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:142) ~[?:?]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[?:?]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]

at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]

* Other installed Elasticsearch or Kibana plugins, if any

I'm trying to do cross cluster search from my server elastic search which is secured by search guard to normal http elastic search service which is running in another server.

I'm able to do the search between two unsecured elastic search, but I'm unable to do https to http elastic search

Below were my search guard elastic search Cluster Settings:

{

  "persistent": {

    "search": {

      "remote": {

        "cluster_one": {

          "seeds": [

            "x.x.x.x:9300"

          ]

        }

      }

    }

  },

  "transient": {}

}

when I curl the cluster_one transport port 9300, I'm getting This is not a HTTP port, but still I'm unable to connect the server.

Please help me to solve this.

[SG]

That looks like a SSL problem (handshake failed). You need to make sure that the clusters can (SSL wise) talk to each other which means sharing the same root certificate typically.

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2b1e03b2-72e1-4971-bb14-e17560ef6f1b%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Venkata Naresh]

Hi,

As I said one cluster is secure with SSL, where as another is not secure with SSL. How we can share the SSL root certificate to normal server 

[SG]

Both clusters need to be SSL and Search Guard secured!

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/63d82437-a491-405a-bfd0-f2e3f367da60%40googlegroups.com.

[Venkata Naresh]

Is it not possible to keep one cluster secure and another not for communication?

My Unsecured cluster is giving below error when its getting request from SG Cluster

Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)

es_1       |    at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1226) ~[elasticsearch-5.2.2.jar:5.2.2]

es_1       |    at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]

es_1       |    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]

My use case deals with communication btw secure ES and unsecure ES. Is there any work around to do this process?

[SG]

> Am 06.02.2019 um 18:28 schrieb Venkata Naresh <divi.v...@gmail.com>:
>
> Is it not possible to keep one cluster secure and another not for communication?

No. Search Guard relies on SSL so all nodes/clusters who want to talk to eachother must use SSL

>
> My Unsecured cluster is giving below error when its getting request from SG Cluster
>
> Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
> es_1 | at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1226) ~[elasticsearch-5.2.2.jar:5.2.2]
> es_1 | at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]
> es_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[?:?]

Yes, expected because aSSL secured cluster can not talk to an not SSL secured cluster

>
> My use case deals with communication btw secure ES and unsecure ES. Is there any work around to do this process?

I'am afraid not (at least not with cross cluster search). Your client can deal separately with a secured and unsecured cluster using two different HTTP connections.
But on transport layer (that is how clusters/nodes talk to each other) it is not possible with Search Guard. If it would be possible then this would be a security issue :-)

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/23fab6ce-e60a-40f3-91e5-cf88a377d71b%40googlegroups.com.

[Venkata Naresh]

In that case I will make my other ES server also SSL enadled, but then I have a problem with data sink between mongodb to elastic search. As of now I'm using abcimport plugin to import data into ES. For abcimport there is no option to add SSL certificate of ES due to which the data import is getting failed.

Is there any option to sink data between mongo to secure ES?