Elasticsearch on SO can't connect to itself?

[Kevin Branch]

I am not sure why but my SO Elasticsearch can't connect to itself on 9300 which naturally prevents Kibana from being able to fetch any records at all.

The ES instance reports a green state but throws a connection exception when it attempts "fetching nodes from external cluster".  When I encountered this, my first response was to run a fresh sudo soup, but that did not change this behavior.  When that did not help, I also did a so-elastic-stop followed by a so-elastic-configure-stack, but that also did not fix it.  Please advise where I might look next.  All other facilities appear to be working.  100% green OKs in response to so-status.  I think this is likely a simple Elasticsearch/Docker plumbing issue but I'm not sure where to look next.  Thanks!

GET /_cluster/settings

{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "my-so-standalone" : {
          "seeds" : [
            "172.17.0.2:9300"
          ]
        }
      }
    }
  },
  "transient" : { }
}

root@NSM:~# curl 127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "nsm.domain.org",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 117,
  "active_shards" : 117,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

# ps auxw | grep docker-proxy | grep 9300
root     14164  0.0  0.0   9412  3740 ?        Sl   01:22   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 9300 -container-ip 172.17.0.2 -container-port 9300

from logstash log file:

[2019-07-02T01:23:19,627][INFO ][org.elasticsearch.discovery.DiscoveryModule] using discovery type [zen] and host providers [settings]
[2019-07-02T01:23:21,419][INFO ][org.elasticsearch.node.Node] initialized
[2019-07-02T01:23:21,420][INFO ][org.elasticsearch.node.Node] starting ...
[2019-07-02T01:23:21,600][INFO ][org.elasticsearch.transport.TransportService] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2019-07-02T01:23:24,985][INFO ][org.elasticsearch.cluster.service.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {CTVW34H}{CTVW34HtQiaUTghnCWwmkw}{KFgXBOBEQLuQ-qVPu6iF3A}{127.0.0.1}{127.0.0.1:9300}
[2019-07-02T01:23:24,991][INFO ][org.elasticsearch.cluster.service.ClusterApplierService] new_master {CTVW34H}{CTVW34HtQiaUTghnCWwmkw}{KFgXBOBEQLuQ-qVPu6iF3A}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {CTVW34H}{CTVW34HtQiaUTghnCWwmkw}{KFgXBOBEQLuQ-qVPu6iF3A}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-07-02T01:23:25,011][INFO ][org.elasticsearch.http.netty4.Netty4HttpServerTransport] publish_address {172.17.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2019-07-02T01:23:25,012][INFO ][org.elasticsearch.node.Node] started
[2019-07-02T01:23:27,335][INFO ][org.elasticsearch.common.settings.ClusterSettings] updating [cluster.remote.my-so-standalone.seeds] from [[]] to [["172.17.0.2:9300"]]
[2019-07-02T01:23:27,353][WARN ][org.elasticsearch.transport.RemoteClusterConnection] fetching nodes from external cluster [my-so-standalone] failed
org.elasticsearch.transport.ConnectTransportException: [][172.17.0.2:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1309) ~[elasticsearch-6.7.2.jar:6.7.2]
        at org.elasticsearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:100) ~[elasticsearch-6.7.2.jar:6.7.2]
        at org.elasticsearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:42) ~[elasticsearch-core-6.7.2.jar:6.7.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159) ~[?:?]
        at org.elasticsearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:57) ~[elasticsearch-core-6.7.2.jar:6.7.2]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$new$1(Netty4TcpChannel.java:72) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.2:9300
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        ... 6 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327) ~[?:?]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?]
        ... 6 more

[Doug Burks]

Hi Kevin,

Did you manually set your _cluster/settings to map my-so-standalone to "172.17.0.2:9300" at some point?  I think our scripts would normally set that to "127.0.0.1:9300".

From https://github.com/Security-Onion-Solutions/securityonion-elastic/blob/master/usr/sbin/so-elastic-configure-kibana-config#L74-L77:

 echo "Applying cross cluster search config..."
curl -s -XPUT http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_cluster/settings \
    -H 'Content-Type: application/json' \
    -d "{\"persistent\": {\"search\": {\"remote\": {\"$HOSTNAME\": {\"seeds\": [\"127.0.0.1:9300\"]}}}}}"

--
Follow Security Onion on Twitter!
https://twitter.com/securityonion
---
You received this message because you are subscribed to the Google Groups "security-onion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at https://groups.google.com/group/security-onion.
To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CA%2BdGL9Hdq%3DwPtK%3DxZiRb5a7Dy5FF-%3DVe49nzDu40JxnATdjGmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Doug Burks
CEO
Security Onion Solutions, LLC

[Kevin Branch]

Indeed I confess I had fiddled with that and thought I'd put it back to default.  Thanks for pointing it out.  The system is working now :)

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/security-onion/CAJ%2BhwWDV31Mkg4iOBm2zxK1GN%2B_EekBcHNhFz9nxy3LKbUvckA%40mail.gmail.com.