fluent-bit -> fluentd tls

1,741 views
Skip to first unread message

Samit Pal

unread,
Jun 15, 2018, 2:45:18 AM6/15/18
to Fluent-Bit
Hi,

I'm working on the following setup where I'm sending logs using fluent-bit to fluentd via secure forward output plugin.

fluent-bit (using tls) -> fluentd -> s3

I'm using a public certificate on the fluentd. I then enable the following configs on the fluent-bit
-----------
tls on
tls.verfiy on
-----------
fluent-bit complains whenever tls.verfiy on is turned on. 

The following are my questions

Q1. Does fluent-bit need setting an explicit ca certificate file via tls.ca_file config?

Q2. I also plan to use tls mutual auth between fluentd and fluent-bit. Can I use a public CA certificate on the fluentds but use a private CA signed certificate on the fluent-bits and have tls mutual auth? Or do the certificates on the fluentd and fluent-bit need to be signed by the same CA authority?

Thanks
Samit


Samit Pal

unread,
Jun 15, 2018, 5:40:39 AM6/15/18
to Fluent-Bit
BTW, regarding question 1, when I try to set 'tls.verify' to 'On' I get the following error

flb_io_tls.c:305 X509 - Certificate verification failed, e.g. CRL, CA or signature check

Samit Pal

unread,
Jun 15, 2018, 7:17:17 AM6/15/18
to Fluent-Bit
Also when I try to set the ca file as follows in the config,
-----
tls.ca_file "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"
-----

I get a weird error (below)


Error
=====
Fluent-Bit v0.13.4
Copyright (C) Treasure Data

[2018/06/15 10:42:56] [ info] [engine] started (pid=26123)
[2018/06/15 10:42:56] [error] [io_tls] flb_io_tls.c:109 PK - Read/write of file failed
[2018/06/15 10:42:56] [error] [TLS] Invalid CA file: "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"
[2018/06/15 10:42:56] [error] [output forward.0] error initializing TLS context

I can use the same ca file with openssl command and it works fine there. What am I missing?

Thanks
Samit

s...@getpolymorph.com

unread,
Jun 15, 2018, 10:56:43 AM6/15/18
to Fluent-Bit
I would highly recommend not using the secure forward plug-in. Instead handle the encryption with a VPN Tunel if using a hybrid cloud or AWS cross region, if not then wrapping it with nginx/haproxy TCP TLS upstream.

fluent-bit => nginx/haproxy TCP => encrypted public internet => nginx/haproxy TCP => fluentd

Samit Pal

unread,
Jun 15, 2018, 11:44:48 PM6/15/18
to s...@getpolymorph.com, fluen...@googlegroups.com

Interesting! I would be interested to know why. How about using the secure http output fluent-bit plugin? If that is also not recommended I guess I could consider using fluentd at the client side too. So essentially fluentd (client side, secure forward) -> fluentd (server running tls) -> s3

Thanks Sam for your response!

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/5896071f-cfb9-45d0-b66e-d75390ae4667%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Regards
Samit

Eduardo Silva

unread,
Jun 16, 2018, 12:03:17 AM6/16/18
to sam...@gmail.com, s...@getpolymorph.com, Fluent-Bit
which commands are you using to generate your certificates ?


For more options, visit https://groups.google.com/d/optout.


--
Eduardo Silva
Open Source, Treasure Data
http://www.treasuredata.com/opensource

http://twitter.com/edsiper
  http://www.linkedin.com/in/edsiper

Samit Pal

unread,
Jun 16, 2018, 12:19:03 AM6/16/18
to edu...@treasure-data.com, s...@getpolymorph.com, fluen...@googlegroups.com
1. As I said, I'm using public CA signed certificate (Entrust intermediate) on the fluentd side. On the fluent-bit host I'm just trying the following config

-----------
tls on
tls.verfiy on
-----------

I get "flb_io_tls.c:305 X509 - Certificate verification failed, e.g. CRL, CA or signature check" error from fluent-bit. Note that the error goes away if i turn off tls.verify. I have used "openssl s_client  -connect <host>:<port> command from the fluent-bit machine to verify whether the certs are correct on the fluentd server and it works fine.

2. The next attempt that I made was with the following config

-----
tls on
tls.verfiy on
tls.ca_file "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"
-----
This is when I get the following error

[2018/06/15 10:42:56] [ info] [engine] started (pid=26123)
[2018/06/15 10:42:56] [error] [io_tls] flb_io_tls.c:109 PK - Read/write of file failed
[2018/06/15 10:42:56] [error] [TLS] Invalid CA file: "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"
[2018/06/15 10:42:56] [error] [output forward.0] error initializing TLS context

I can again confirm that when I use the same ca file with openssl command on the fluent-bit host it just works

$ openssl s_client -CAfile /etc/ssl/certs/Entrust_Root_Certification_Authority.pem -connect  <host>:<port>
--
Regards
Samit

s...@getpolymorph.com

unread,
Jun 16, 2018, 1:33:47 AM6/16/18
to Fluent-Bit
We used to use the secure forward but we swapped away about two years ago. The problem was that it doesn't dynamically resolve DNS. This meant that we couldn't autoscale using an ELB. Also, ssl termination in ruby isn't as efficient as nginx or haproxy.

One caveat, Nginx also wont resolve upstream DNS unless you pay for it. This means that when the ELB updates its ip addresses you lose connection. If i were to do it all over again i'd set up VPN peering since network capacity managed by aws and skip the whole encryption discussion. Then you can just use the internal DNS entry from each region and forget about it.

Sándor Guba

unread,
Jun 16, 2018, 3:35:37 AM6/16/18
to s...@getpolymorph.com, Fluent-Bit
If you have trouble setting up this deployment I suggest you to read this blog article: https://banzaicloud.com/blog/k8s-logging-tls/
This is a comprehensive working set-up for the described fluent-bit, fluentd tls. I hope you find it useful

To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.

To post to this group, send email to fluen...@googlegroups.com.

Samit Pal

unread,
Jun 18, 2018, 12:29:07 AM6/18/18
to guba...@gmail.com, s...@getpolymorph.com, fluen...@googlegroups.com
Thanks Sandor. I had referenced the doc that you mention earlier during my design phase.

@Eduardo,
Could the error from fluent-bit be because of the ssl certificate chain? I have the .pem cert in the following manner

-----BEGIN CERTIFICATE----- 
(My Primary SSL certificate: your_domain_name.crt) 
-----END CERTIFICATE----- 

-----BEGIN CERTIFICATE----- 
(My Intermediate certificate: entrust.crt) 
-----END CERTIFICATE-----


I have also tried the following without success

-----BEGIN CERTIFICATE----- 
(My Primary SSL certificate: your_domain_name.crt) 
-----END CERTIFICATE----- 

-----BEGIN CERTIFICATE----- 
(My Intermediate certificate: entrust.crt) 
-----END CERTIFICATE----- 

-----BEGIN CERTIFICATE----- 
(My Root certificate: entrust.crt) 
-----END CERTIFICATE-----



We had a similar issue with our nginx setup proxypassing to aws hosted kibana. We solved it by following https://distinctplace.com/2017/04/19/nginx-resolver-explained/ . Search for "Free Nginx resolver alternative".

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Regards
Samit
Reply all
Reply to author
Forward
0 new messages