logstash can't send message to es that installed search guard

683 views

Skip to first unread message

blith...@gmail.com

unread,

Aug 22, 2017, 6:06:59 AM8/22/17

to Search Guard Community Forum

es version: 5.5.1

search guard: 5-5.5.1-15

jdk version: 1.8.0_131-b11

logstash version: 2.4.1

My es cluster installed search guard has 5 nodes, logstash can't send message to es, and error info is: host name does not match certificate subject. Why?

elasticsearch.yml

cluster.name: honeycomb-es-guard-5.5.1

# node 1 to 5

node.name: node-1

path.data: /data01/elasticsearch-guard

path.logs: /opt/yrd_logs/elasticsearch-guard

network.host: 0.0.0.0

http.cors.enabled: true

http.cors.allow-origin: "*"

thread_pool.bulk.queue_size: 50

searchguard.ssl.transport.keystore_filepath: node-1-keystore.jks

searchguard.ssl.transport.keystore_password: node_es

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.truststore_password: r_ca_honeycomb

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.enable_openssl_if_available: true

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: node-1-keystore.jks

searchguard.ssl.http.keystore_password: node_es

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.truststore_password: r_ca_honeycomb

searchguard.authcz.admin_dn:

- CN=*,OU=client,O=client,L=test, C=de

- CN=kirk,OU=client,O=client,L=test,C=DE

searchguard info, as follows

[elk@orz-core-elk-05 config]$ curl -k -u admin:admin 'https://10.134.84.33:9200/_searchguard/authinfo?pretty'

{

"user" : "User [name=admin, roles=[]]",

"user_name" : "admin",

"user_requested_tenant" : null,

"remote_address" : "10.134.84.36:49843",

"sg_roles" : [

"sg_all_access",

"sg_own_index",

"sg_public"

"sg_tenants" : {

"test_tenant_ro" : true,

"admin" : true,

"adm_tenant" : true

"principal" : null,

"peer_certificates" : "0"

}

cluster info, as follows

curl -k -u admin:admin 'https://10.134.84.33:9200/_cat/health?v'

epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent

1503395742 17:55:42 honeycomb-es-guard-5.5.1 green 5 5 5 1 0 0 0 0 - 100.0%

logstash configuration

output {

elasticsearch {

hosts => ["https://10.134.84.32:9200"]

index => "%{log_project}-%{+YYYY-MM-dd}"

ssl => true

ssl_certificate_verification => true

truststore => "/opt/yrd_soft/elasticsearch-5.5.1-guard/config/truststore.jks"

truststore_password => r_ca_honeycomb

user => logstash

password => logstash

}

start logstash

[elk@orz-core-elk-02 conf]$ /opt/yrd_soft/logstash-2.4.1-guard/bin/logstash -f /opt/yrd_soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf &

[1] 1691

[elk@orz-core-elk-02 conf]$ Settings: Default pipeline workers: 56

Host name '10.134.84.32' does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE) {:class=>"Manticore::UnknownException", :level=>:error}

Pipeline main started

Attempted to send a bulk request to Elasticsearch configured at '["https://10.134.84.32:9200"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error_message=>"Host name '10.134.84.32' does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE)", :error_class=>"Manticore::UnknownException", :backtrace=>["/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

Host name '10.134.84.32' does not match the certificate subject provided by the peer (CN=*, OU=SSL, O=Test, L=Test, C=DE) {:class=>"Manticore::UnknownException", :backtrace=>["/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}

SG

unread,

Aug 22, 2017, 4:47:57 PM8/22/17

to search...@googlegroups.com

set
ssl_certificate_verification => false
in logstash config, see https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ssl_certificate_verification

or (the better and more secure option) create ssl certificates which match your hostname (http://apetec.com/support/generatesan-csr.htm)

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2ab4e468-b5e7-4b00-ab9d-5350e1d6d8f7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Wei Feng

unread,

Aug 22, 2017, 11:15:08 PM8/22/17

to search...@googlegroups.com

when set ssl_certificate_verification => false, failed to start logstash

logstash info as follows:

[elk@orz-core-elk-02 conf]$ /opt/yrd_soft/logstash-2.4.1-guard/bin/logstash -f /opt/yrd_soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf &

[5] 27508

[elk@orz-core-elk-02 conf]$ Settings: Default pipeline workers: 56

** WARNING ** Detected UNSAFE options in elasticsearch output configuration!

** WARNING ** You have enabled encryption but DISABLED certificate verification.

** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true {:level=>:warn}

Pipeline aborted due to error {:exception=>"Java::JavaIo::EOFException", :error=>"", :backtrace=>["java.io.DataInputStream.readInt(java/io/DataInputStream.java:392)", "sun.security.provider.JavaKeyStore.engineLoad(sun/security/provider/JavaKeyStore.java:653)", "sun.security.provider.JavaKeyStore$JKS.engineLoad(sun/security/provider/JavaKeyStore.java:56)", "sun.security.provider.KeyStoreDelegator.engineLoad(sun/security/provider/KeyStoreDelegator.java:224)", "sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(sun/security/provider/JavaKeyStore.java:70)", "java.security.KeyStore.load(java/security/KeyStore.java:1445)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "RUBY.get_store(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:667)", "org.jruby.RubyKernel.tap(org/jruby/RubyKernel.java:1858)", "RUBY.get_store(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:665)", "RUBY.setup_trust_store(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:609)", "RUBY.ssl_socket_factory_from_options(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:602)", "RUBY.pool_builder(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:394)", "RUBY.pool(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:402)", "RUBY.initialize(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/client.rb:208)", "RUBY.build_client(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:58)", "RUBY.initialize(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:49)", "RUBY.initialize(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:118)", "RUBY.new(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport.rb:26)", "RUBY.build_client(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:129)", "RUBY.initialize(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:20)", "RUBY.build(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:44)", "RUBY.build_client(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch.rb:134)", "RUBY.register(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:14)", "RUBY.register(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:75)", "RUBY.start_workers(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:181)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.start_workers(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:181)", "RUBY.run(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:136)", "RUBY.start_pipeline(/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/agent.rb:491)", "java.lang.Thread.run(java/lang/Thread.java:748)"], :level=>:error}

stopping pipeline {:id=>"main"}

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2ab4e468-b5e7-4b00-ab9d-5350e1d6d8f7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/wNUym4hQCyY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/97A1EBF2-0BED-48B7-8598-1CA105B82E55%40search-guard.com.

SG

unread,

Aug 23, 2017, 4:16:46 AM8/23/17

to search...@googlegroups.com

This looks like a logstash bug, i recommend you ask the question here https://discuss.elastic.co
You can also try to remove truststore and truststore_password setting cause you do not validate the server cert anyhow

Beside that i wonder why you use a 2.x logstash version and a 5.x elasticsearch version.
Maybe you have more luck with a more recent logstash (Version 5.5.2).

> > To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> > To post to this group, send email to search...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2ab4e468-b5e7-4b00-ab9d-5350e1d6d8f7%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/wNUym4hQCyY/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/97A1EBF2-0BED-48B7-8598-1CA105B82E55%40search-guard.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/CAEv-g5hpAkUbfzrYQ8ijbtSb9ee1f9C8Pd%2Bjii_iv0hm%2BPP9AQ%40mail.gmail.com.

Wei Feng

unread,

Aug 23, 2017, 11:03:51 PM8/23/17

to search...@googlegroups.com

Thanks for your answer!

1. When I deleted truststore and truststore_password setting, logstash failed too

logstash error info

[elk@orz-core-elk-02 conf]$ /opt/yrd_soft/logstash-2.4.1-guard/bin/logstash -f /opt/yrd_soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf &

[1] 4890

[elk@orz-core-elk-02 conf]$ Settings: Default pipeline workers: 56

** WARNING ** Detected UNSAFE options in elasticsearch output configuration!

** WARNING ** You have enabled encryption but DISABLED certificate verification.

** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true {:level=>:warn}

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:class=>"Manticore::ClientProtocolException", :level=>:error}

Pipeline main started

Attempted to send a bulk request to Elasticsearch configured at '["https://localhost:9200"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error_message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/yrd_soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

And now, es log as follows:

[2017-08-24T10:39:20,834][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]

at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

[2017-08-24T10:39:21,663][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown