Having an issue with fluentd to connect to Elasticsearch using SSL key and pem

Krishna

unread,

Mar 29, 2019, 12:39:59 PM3/29/19

to Fluentd Google Group

Hi,

I am having an issue to connecting elasticsearch using Client key, pem and cacert file, please help me.

Error:

2019-03-29 16:30:28 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-03-29 16:30:29 +0000 chunk="5853e2fffab210b9005cf3f23c1cf4b1" error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"test1-elastic..net\", :port=>443, :scheme=>\"https\", :user=>\"admin\", :password=>\"obfuscated\"})!"

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:351:in `client'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:638:in `rescue in send_bulk'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:625:in `send_bulk'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:530:in `block in write'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:529:in `each'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.12.0/lib/fluent/plugin/out_elasticsearch.rb:529:in `write'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1111:in `try_flush'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:1394:in `flush_thread_run'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin/output.rb:445:in `block (2 levels) in start'

2019-03-29 16:30:28 +0000 [warn]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.2.6/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Fluentd Config:

apiVersion: v1

kind: Namespace

metadata:

name: logging

---

apiVersion: rbac.authorization.k8s.io/v1beta1

@type elasticsearch

hosts test1-elastic.net:443

scheme https

ca_file "/home/certs/cacert"

client_cert "/home/certs/test1-elastic-client.net.pem"

client_key "/home/certs/test1-elastic-client.net.key"

user "admin"

password changeme

index log-nprod

reload_connections true

Certs file permissions on docker container

root@fluentd-47g2r:/home/certs# ls -latr

total 44

-rw-r--r--. 1 fluent fluent 6753 Mar 29 16:11 test1-elastic-client.net.pem

-rw-r--r--. 1 fluent fluent 2097 Mar 29 16:11 cacert

-rw-r--r--. 1 fluent fluent 1704 Mar 29 16:13 test1-elastic-client.net.key

drwxr-xr-x. 1 root root 4096 Mar 29 16:16 ..

drwxr-xr-x. 1 fluent fluent 4096 Mar 29 16:16 .

Thomas Goeller

unread,

Mar 29, 2019, 12:51:19 PM3/29/19

to flu...@googlegroups.com

Is it reachable at all? Right port? 9200 instead of 443?

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Krishna

unread,

Mar 29, 2019, 5:43:18 PM3/29/19

to Fluentd Google Group

yes, i am able to connect to elasticsearch on port 443

i ran curl command from same fluentd pod where it's failing.

curl -XGET "https://test1-elastic.net:443/_cluster/health?pretty" --key /home/certs/test1-elastic-client.net.key --cert /home/certs/test1-elastic-client.net.pem --cacert /home/certs/cacert

{

"cluster_name" : "escluster-test1",

"status" : "green",

"timed_out" : false,

"number_of_nodes" : 4,

"number_of_data_nodes" : 4,

"active_primary_shards" : 4,

"active_shards" : 10,

"relocating_shards" : 0,

"initializing_shards" : 0,

"unassigned_shards" : 0,

"delayed_unassigned_shards" : 0,

"number_of_pending_tasks" : 0,

"number_of_in_flight_fetch" : 0,

"task_max_waiting_in_queue_millis" : 0,

"active_shards_percent_as_number" : 100.0

}

On Friday, March 29, 2019 at 9:51:19 AM UTC-7, TomG wrote:

Is it reachable at all? Right port? 9200 instead of 443?

To unsubscribe from this group and stop receiving emails from it, send an email to flu...@googlegroups.com.

Mr. Fiber

unread,

Mar 30, 2019, 3:20:34 AM3/30/19

to Fluentd Google Group

> :host=>\"test1-elastic..net\",

From the log, host value is "test1-elastic..net", not "test1-elastic.net".

Is this typo?

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

Krishna

unread,

Mar 30, 2019, 4:49:31 PM3/30/19

to Fluentd Google Group

It was a typo while I was posting here.

Krishna

unread,

Mar 30, 2019, 8:12:06 PM3/30/19

to Fluentd Google Group

After upgrading Elasticsearch plugin fluentd is trying to connect but it says ssl version error.

2019-03-31 00:07:18 +0000 [error]: #0 unexpected error error_class=Faraday::SSLError error="SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate unknown (OpenSSL::SSL::SSLError)"

this is the variable defined in fluentd config

ssl_version TLSv1_2

updated with ssl_version TLSv1, it has same issue.

Krishna

unread,

Mar 31, 2019, 2:24:30 PM3/31/19

to Fluentd Google Group

Fluentd sucks, getting SSL error from past 2 days.

2019-03-31 18:21:27 +0000 [info]: starting fluentd-1.4.1 pid=1 ruby="2.3.3"

2019-03-31 18:21:27 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/usr/local/bin/fluentd", "-c", "/fluentd/etc/fluentd.conf", "-p", "/fluentd/plugins", "--under-supervisor"]

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.4.0'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.12.0'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-kubernetes-log-level' version '0.2.3'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.27.0'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.1'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.1'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-splunk-http-eventcollector' version '0.3.0'

2019-03-31 18:21:27 +0000 [info]: gem 'fluent-plugin-systemd' version '0.3.0'

2019-03-31 18:21:27 +0000 [info]: gem 'fluentd' version '1.4.1'

2019-03-31 18:21:27 +0000 [info]: gem 'fluentd' version '1.2.6'

2019-03-31 18:21:27 +0000 [info]: gem 'fluentd' version '0.14.19'

2019-03-31 18:21:27 +0000 [info]: gem 'fluentd' version '0.12.33'

2019-03-31 18:21:27 +0000 [info]: adding filter in @INPUT pattern="kubernetes.**" type="kubernetes_metadata"

2019-03-31 18:21:28 +0000 [info]: adding match in @INPUT pattern="**" type="relabel"

2019-03-31 18:21:28 +0000 [info]: adding match in @RETAG pattern="**" type="rewrite_tag_filter"

2019-03-31 18:21:28 +0000 [info]: #0 adding rewrite_tag_filter rule: @severity [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x007f8b7dc5da40 @keys="@severity">, /(DEBUG|TRACE|WARN)/, "", "elasticsearch.${tag}"]

2019-03-31 18:21:28 +0000 [info]: #0 adding rewrite_tag_filter rule: @severity [#<Fluent::PluginHelper::RecordAccessor::Accessor:0x007f8b7cbefe88 @keys="@severity">, /(INFO|ERROR)/, "", "splunk.${tag}"]

2019-03-31 18:21:28 +0000 [info]: adding filter in @OUTPUT pattern="splunk.**" type="record_transformer"

2019-03-31 18:21:28 +0000 [info]: adding match in @OUTPUT pattern="splunk.**" type="splunk-http-eventcollector"

2019-03-31 18:21:28 +0000 [info]: adding match in @OUTPUT pattern="elasticsearch.**" type="elasticsearch"

2019-03-31 18:21:30 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=unknown state: wrong version number (OpenSSL::SSL::SSLError)

2019-03-31 18:21:34 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=unknown state: wrong version number (OpenSSL::SSL::SSLError)

2019-03-31 18:21:42 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=unknown state: wrong version number (OpenSSL::SSL::SSLError)

2019-03-31 18:21:58 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=unknown state: wrong version number (OpenSSL::SSL::SSLError)

i have tried with all theses versions: SSLv23,TLSv1,TLSv1_1,TLSv1_2

also upgraded fluentd version, no luck, someone please help me.

On Saturday, March 30, 2019 at 12:20:34 AM UTC-7, repeatedly wrote:

Krishna

unread,

Apr 1, 2019, 2:08:46 PM4/1/19

to Fluentd Google Group

Fluentd is able to connect to Elasticsearch if i use "cacert, username and password".

it's failing to connect to elasticsearch when i use only `"client key, pem and cacert" without user and password.

Mr. Fiber

unread,

Apr 1, 2019, 8:01:17 PM4/1/19

to Fluentd Google Group

Hmm... I see.

It seems a bug of elasticsearch-ruby or backend HTTP library usage.

https://github.com/uken/fluent-plugin-elasticsearch/blob/38c8d2267b9598c6bebcdd813f9699becc1615b2/lib/fluent/plugin/out_elasticsearch.rb#L310

https://github.com/uken/fluent-plugin-elasticsearch/blob/38c8d2267b9598c6bebcdd813f9699becc1615b2/lib/fluent/plugin/out_elasticsearch.rb#L394

Here is plugin code. You can check elasticsearch-ruby behaviour on you side.

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.

Suat Toksoz

unread,

Feb 27, 2020, 8:01:02 AM2/27/20

to Fluentd Google Group

Hi,

How can I get the fluentd to connect/senda data to elasticsearch cluster?

Thanks

Balaji K

unread,

Feb 28, 2020, 5:39:39 AM2/28/20

to flu...@googlegroups.com

#Setting here <quickstart -> namespace> key to username/password elastic/elastic
kubectl create secret generic quickstart-es-elastic-user --from-literal=elastic=elastic --> Set static password before starting operator.
#docker.elastic.co/eck/eck-operator:1.0.1
kubectl apply -f all-in-one.yaml
#Above will start the operator
kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
#Above is to observer logs of operator
kubectl apply -f elasticsearch_auto.yaml
#Above Creates object of the CRD and deploys
kubectl port-forward service/quickstart-es-http 9200

kubectl port-forward service/quickstart-kb-http 5601

https://github.com/helm/charts/tree/master/stable/fluentd-elasticsearch

set sslVerify to false

enable auth and set user and password.

Above is the set of procedures we followed and works fine. Hope this helps..

rgds

Balaji Kamal Kannadassan

To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/7a969080-635f-420e-8ba1-c367d6201950%40googlegroups.com.

--

Balaji Kamal Kannadassan
plot #70, Door #28,
Majestic Colony Main Road,
Vlasaravakkam,
Chennai-87

Balaji K

unread,

Feb 28, 2020, 5:41:48 AM2/28/20

to flu...@googlegroups.com

Reference:

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.htm

Reply all

Reply to author

Forward