latest Searchguard bundle not working on a single node
413 views
Skip to first unread message
rocky
Aug 4, 2016, 9:58:25 AM8/4/16
to Search Guard
Hi All,
Can anyone help me with the below issue I am facing
I used tarfile which was said to be the easiest way to installtion searchguard which also includes elasticsearch package. Please refer to below link
https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle.
After extracting tarfile I have started elasticsearch using the steps as mentioned in link and later ran sgadmin.sh script and evrything was looking great.
Once I have installed Kibana and Logstash as root user , installed beats on client machine . Currently beats are able to push logs to logstash whereas logstash is not able to comminucate to elasticsearch.
When I found reason it says "Search Guard not initialized (SG11)" and searchguard index not healthy (timeout: truewhich is really strange.
Note: I deployed search guard tarfile using non rootuser.I also installed Kibana logastah and Elasticsearch with search guard on same server. beats on client machines.
Can anyone guide me here please as I am going crazy with this searchguard installation?
SG
Aug 4, 2016, 3:50:11 PM8/4/16
to search...@googlegroups.com
pls do not cross post, this one is already logged here https://github.com/floragunncom/search-guard/issues/182#issuecomment-237533599
Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.
Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a "Search Guard not initialized (SG11)" afterwards.
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Can you please provide error messages and your elasticsearch log files as well as your configuration? Otherwise its hard to help.
Start easy just with the Search Guard bundle and curl. Once you initialized Search Guard by runiing sgadmin you normally should not see a "Search Guard not initialized (SG11)" afterwards.
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/68a3415e-eaee-439d-ad0e-9e2759295549%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
rocky
Aug 4, 2016, 8:32:05 PM8/4/16
to Search Guard
*******************************I thought previous post was closed hence I opened a new one . Sorry for this confusion*************************************************************************
When I installed SG+ES bundle for first time I was able to successfully executed sgadmin script without any issues. Once I added Logstash and Kibana and when I tried to start ES I got message as searchguard not initialized SG11.
I tried below steps to fix this issue as per previous blogs but no luck. Can you please help on this?
*************************************ES logs when disabled both searchguard SSL and SG**********************************************************************************
./elasticsearch-2.3.4-localhost/bin/elasticsearch
[2016-08-04 18:52:41,343][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,
CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-08-04 18:52:41,437][INFO ][node ] [localhost] version[2.3.4], pid[20117], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-04 18:52:41,437][INFO ][node ] [localhost] initializing ...
[2016-08-04 18:52:41,769][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin not available
[2016-08-04 18:52:41,789][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-
guard-2], sites [kopf]
[2016-08-04 18:52:41,803][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],
net total_space [9.7gb], spins? [possibly], types [ext4]
[2016-08-04 18:52:41,803][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]
[2016-08-04 18:52:41,803][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at
least [65536]
[2016-08-04 18:52:41,830][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK
SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,
netty-tcnative]
[2016-08-04 18:52:42,011][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from
there the key- and truststore files are resolved relatively
Exception in thread "main" ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]
at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)
at com.floragunn.searchguard.ssl.SearchGuardKeyStore.<init>(SearchGuardKeyStore.java:139)
at com.floragunn.searchguard.ssl.SearchGuardSSLModule.<init>(SearchGuardSSLModule.java:29)
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)
at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)
at org.elasticsearch.node.Node.<init>(Node.java:179)
at org.elasticsearch.node.Node.<init>(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
*******************************************************************************************************************************************************
ES logs after uncommenting SG SSL only. Restarted ES and below are logs
******************************************************************************************************************************************************
./elasticsearch-2.3.4-localhost/bin/elasticsearch
[2016-08-04 18:59:00,447][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,
CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-08-04 18:59:00,554][INFO ][node ] [localhost] version[2.3.4], pid[20193], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-04 18:59:00,554][INFO ][node ] [localhost] initializing ...
[2016-08-04 18:59:00,892][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available
[2016-08-04 18:59:00,913][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-
guard-2], sites [kopf]
[2016-08-04 18:59:00,927][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],
net total_space [9.7gb], spins? [possibly], types [ext4]
[2016-08-04 18:59:00,927][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]
[2016-08-04 18:59:00,927][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at
least [65536]
[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK
SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,
netty-tcnative]
[2016-08-04 18:59:00,954][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK
SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,
netty-tcnative]
[2016-08-04 18:59:01,114][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from
there the key- and truststore files are resolved relatively
Exception in thread "main" ElasticsearchException[searchguard.ssl.transport.keystore_filepath must be set if transport ssl is reqested.]
at com.floragunn.searchguard.ssl.SearchGuardKeyStore.initSSLConfig(SearchGuardKeyStore.java:188)
at com.floragunn.searchguard.ssl.SearchGuardKeyStore.<init>(SearchGuardKeyStore.java:139)
at com.floragunn.searchguard.ssl.SearchGuardSSLModule.<init>(SearchGuardSSLModule.java:29)
at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.nodeModules(SearchGuardSSLPlugin.java:126)
at org.elasticsearch.plugins.PluginsService.nodeModules(PluginsService.java:263)
at org.elasticsearch.node.Node.<init>(Node.java:179)
at org.elasticsearch.node.Node.<init>(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
***************************ES logs after SG . Restarted ES and below are logs*******************************************************************
./elasticsearch-2.3.4-localhost/bin/elasticsearch
[2016-08-04 19:04:26,014][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel,
CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
[2016-08-04 19:04:26,112][INFO ][node ] [localhost] version[2.3.4], pid[20249], build[e455fd0/2016-06-30T11:24:31Z]
[2016-08-04 19:04:26,112][INFO ][node ] [localhost] initializing ...
[2016-08-04 19:04:26,444][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available
[2016-08-04 19:04:26,464][INFO ][plugins ] [localhost] modules [reindex, lang-expression, lang-groovy], plugins [search-guard-ssl, kopf, search-
guard-2], sites [kopf]
[2016-08-04 19:04:26,479][INFO ][env ] [localhost] using [1] data paths, mounts [[/home (/dev/mapper/vg_sda-lv_home)]], net usable_space [8.9gb],
net total_space [9.7gb], spins? [possibly], types [ext4]
[2016-08-04 19:04:26,479][INFO ][env ] [localhost] heap size [989.8mb], compressed ordinary object pointers [true]
[2016-08-04 19:04:26,479][WARN ][env ] [localhost] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at
least [65536]
[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK
SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,
netty-tcnative]
[2016-08-04 19:04:26,506][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK
SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty-tcnative-linux-x86_64, netty-tcnative-linux-x86_64-fedora,
netty-tcnative]
[2016-08-04 19:04:26,661][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /home/test/test/elasticsearch-2.3.4-localhost/config/, from
there the key- and truststore files are resolved relatively
[2016-08-04 19:04:26,674][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit.. That is not an issue, it
just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files'
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers
[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers
[TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-08-04 19:04:26,678][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-08-04 19:04:26,785][INFO ][http ] [localhost] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by
[search-guard2]
[2016-08-04 19:04:26,831][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve bound
***************************************************
Search Guard Audit Log is not free software
for commercial use in production.
You have to obtain a license if you
use it in production.
***************************************************
[2016-08-04 19:04:26,833][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog available (AuditLogImpl)
[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service,
overridden by [search-guard2]
[2016-08-04 19:04:26,874][INFO ][transport ] [localhost] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport,
overridden by [search-guard-ssl]
[2016-08-04 19:04:27,340][INFO ][com.floragunn.searchguard.auditlog.impl.AuditLogImpl] Audit Log class: ESAuditLog
[2016-08-04 19:04:27,755][INFO ][node ] [localhost] initialized
[2016-08-04 19:04:27,755][INFO ][node ] [localhost] starting ...
[2016-08-04 19:04:27,816][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [localhost] publish_address {127.0.0.1:9301}, bound_addresses
{[::1]:9301}, {127.0.0.1:9301}
[2016-08-04 19:04:27,820][INFO ][discovery ] [localhost] elasticsearch/14mtfOfFR-yfWh3ZCijeqw
[2016-08-04 19:04:27,823][DEBUG][action.admin.cluster.health] [localhost] no known master node, scheduling a retry
[2016-08-04 19:04:31,080][INFO ][cluster.service ] [localhost] detected_master {localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300}, added
{{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{localhost}{ux5kIQD6QZCseHxfpN7vcQ}{127.0.0.1}
{127.0.0.1:9300}])
[2016-08-04 19:04:31,306][INFO ][http ] [localhost] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[2016-08-04 19:04:31,307][INFO ][node ] [localhost] started
***************************************************
Searchguard DLS/FLS(+) Security is not free software
for commercial use in production.
You have to obtain a license if you
use it in production.
(+) Document-/Fieldlevel
***************************************************
[2016-08-04 19:04:31,437][INFO ][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS enabled
[2016-08-04 19:05:01,288][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:05:34,289][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:06:07,291][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:06:40,292][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:07:13,293][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
*********************************************************************************************************************************************************
Finally started sgadmin and it throws timeout error error
./sgadmin.sh
Connect to localhost:9300
Cluster state timeout
********************************************************************************************************************************************
Logs in ES after starting sgadmin
[2016-08-04 19:13:08,012][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:10,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:13,046][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:14,567][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:13:15,564][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:16,304][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:13:18,075][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:20,593][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:23,104][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:25,620][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:28,133][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:30,649][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:33,164][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:35,697][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:38,212][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:40,728][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:43,244][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:45,760][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:47,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:13:48,274][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:49,305][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:13:50,790][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:53,305][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:55,821][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:13:58,333][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:00,850][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:03,363][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:05,881][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:08,393][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:10,923][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:13,437][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:15,952][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:18,467][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:20,568][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:14:20,983][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:22,306][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [localhost] searchguard index not healthy (timeout: true)
[2016-08-04 19:14:23,498][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:26,014][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:28,530][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:31,045][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-08-04 19:14:33,628][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialize
****************************************************************************************************************************************************************
My Infradetails : Logstash, Kibana, SG+ES bundle installed in one server and installed filebeats on client server
Note: I am going with default setup ad per searchguardbundle for testing phase I didnot create any new admin certificates
ES configuration
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
# cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: node-1
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: x.x.x.x
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, see the documentation at:
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#
#
# For more information, see the documentation at:
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true
node.name: localhost
##################################################
# Search Guard 2 configuration
#
#Host: localhost
#Generated: Sun Jul 31 17:50:58 UTC 2016
#Git Hash: 51fced7dba388267d30ca7826cebcba8fb9edfb8
#ES-Version: 2.3.4
#SG-Version: 2.3.4.4
#SGSSL-Version: 2.3.4.14
#NettyNative-Version: 1.1.33.Fork17
#
#CA_PASS: f3c45ab4d999bc1f0ad8967ff9b42e39980eec58
#CL_ADM_PASS: 553281dad7802f1d6018
#CL_DEMOUSER_PASS: d88242ef83b1c5e48e0e
##################################################
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.transport.keystore_password: xxxx
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: xxxx
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.http.keystore_password: xxxxx
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: xxxx
searchguard.kerberos.krb5_filepath: /Users/temp/kerberos_ldap_environment/krb5.conf
searchguard.kerberos.acceptor_keytab_filepath: http_srv.keytab
searchguard.audit.type: internal_elasticsearch
searchguard.authcz.admin_dn:
- CN=sgadmin
rocky
Aug 5, 2016, 4:17:36 AM8/5/16
to Search Guard
Currently I tried to run sgadmin successfully but still I am not able to push logs from logstash to elasticsearch.
My architecture flow is beats(clientmachine) -> Logstash -> Elasticsearch
Note: Logstash, Kibana are installed with root user and searchguard bundle +ES with non-root user...
Please see additional details which can help you to fix my issue
************logstash configuration ***************************************************************
input {
beats {
port => "5044"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/xxx"
ssl_key => "/etc/pki/tls/private/xxx"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?:
%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "logstash"
password => "xxxx"
ssl => true
ssl_certificate_verification => true
truststore => "/home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks"
truststore_password => "xxxx"
manage_template => false
index => "%{[@metadata][beat][filebeat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
Logstash errors
{:timestamp=>"2016-08-05T02:56:20.048000-0500", :message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target", :class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-
java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-
java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-
transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",
"org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-
java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-
java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-
java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-
java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-
java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}
ES log errors
2016-08-05 03:06:19,735][ERROR][com.floragunn.searchguard.http.SearchGuardHttpServerTransport] [localhost] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
rocky
Aug 27, 2016, 4:08:45 AM8/27/16
to Search Guard
Hi all,
Can any on help me in fixing this issue as I have added new users and roles and when I try to run sgadmin.sh script to update users and roles it give below message.
./sgadmin.sh
Connect to localhost:9300
Cluster state timeout
***************************************************************************************************************************************************************************************************
rocky
Aug 27, 2016, 4:11:52 AM8/27/16
to Search Guard
Hi all,
Can any on help me in fixing this issue as I have added new users and roles and when I try to run sgadmin.sh script to update users and roles it give below message.
I am using one node only
rocky
Aug 28, 2016, 1:11:56 AM8/28/16
to Search Guard
Can anyone help me as I am not able to update users and roles I am getting below error and please see the elasticsearch logs
./sgadmin.sh
Connect to localhost:9300
Cluster state timeout
tail -10 /opt/elastic/elasticsearch-2.3.4-localhost/logs/elasticsearch.log
UnavailableShardsException[[auditlog][4] primary shard is not active Timeout: [1m], request: [index {[auditlog][auditlog][AVbPiINBVOSyDPZ_5rxd], source[{"audit_remote_address":"127.0.0.1:48532","audit_request_headers":"[]","audit_request_class":"class org.elasticsearch.action.admin.cluster.health.ClusterHealthRequest","audit_principal":null,"audit_details":"null","audit_date":"Sun Aug 28 00:04:05 CDT 2016","audit_reason":"cluster:monitor/health","audit_request_context":"[_sg_ssl_cipher=>TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, _sg_remote_address=>127.0.0.1:48532, _sg_user=>User [name=kibanaserver, roles=[]], _sg_ssl_protocol=>TLSv1.2]","audit_category":"AUTHENTICATED","audit_request_user":"kibanaserver"}]}]]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:596)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:465)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:558)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
heapsize details
java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
intx CompilerThreadStackSize = 0 {pd product}
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 130023424 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 2053111808 {product}
intx ThreadStackSize = 1024 {pd product}
intx VMThreadStackSize = 1024 {pd product}
java version "1.8.0_73"
Java(TM) SE Runtime Environment (build 1.8.0_73-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode)
blith...@gmail.com
Aug 24, 2017, 3:58:49 AM8/24/17
to Search Guard Community Forum
How to resolve this question? My es cluster has 5 nodes, and met this question, too.
在 2016年8月5日星期五 UTC+8下午4:17:36，rocky写道：
在 2016年8月5日星期五 UTC+8下午4:17:36，rocky写道：
Reply all
Reply to author
Forward
0 new messages