fluent-bit -> fluentd tls

[Samit Pal]

Hi,

I'm working on the following setup where I'm sending logs using fluent-bit to fluentd via secure forward output plugin.

fluent-bit (using tls) -> fluentd -> s3

I'm using a public certificate on the fluentd. I then enable the following configs on the fluent-bit

-----------

tls on

tls.verfiy on

-----------

fluent-bit complains whenever tls.verfiy on is turned on. 

The following are my questions

Q1. Does fluent-bit need setting an explicit ca certificate file via tls.ca_file config?

Q2. I also plan to use tls mutual auth between fluentd and fluent-bit. Can I use a public CA certificate on the fluentds but use a private CA signed certificate on the fluent-bits and have tls mutual auth? Or do the certificates on the fluentd and fluent-bit need to be signed by the same CA authority?

Thanks

Samit

[Samit Pal]

BTW, regarding question 1, when I try to set 'tls.verify' to 'On' I get the following error

flb_io_tls.c:305 X509 - Certificate verification failed, e.g. CRL, CA or signature check

[Samit Pal]

Also when I try to set the ca file as follows in the config,

-----

tls.ca_file "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"

-----

I get a weird error (below)

Error

=====

Fluent-Bit v0.13.4

Copyright (C) Treasure Data

[2018/06/15 10:42:56] [ info] [engine] started (pid=26123)

[2018/06/15 10:42:56] [error] [io_tls] flb_io_tls.c:109 PK - Read/write of file failed

[2018/06/15 10:42:56] [error] [TLS] Invalid CA file: "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"

[2018/06/15 10:42:56] [error] [output forward.0] error initializing TLS context

I can use the same ca file with openssl command and it works fine there. What am I missing?

Thanks

Samit

[s...@getpolymorph.com]

I would highly recommend not using the secure forward plug-in. Instead handle the encryption with a VPN Tunel if using a hybrid cloud or AWS cross region, if not then wrapping it with nginx/haproxy TCP TLS upstream.

fluent-bit => nginx/haproxy TCP => encrypted public internet => nginx/haproxy TCP => fluentd

[Samit Pal]

Interesting! I would be interested to know why. How about using the secure http output fluent-bit plugin? If that is also not recommended I guess I could consider using fluentd at the client side too. So essentially fluentd (client side, secure forward) -> fluentd (server running tls) -> s3

Thanks Sam for your response!

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/5896071f-cfb9-45d0-b66e-d75390ae4667%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards

Samit

[Eduardo Silva]

which commands are you using to generate your certificates ?

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/CAAY7zxsY9%2BL2yQLrZZGXKRM0pZE-W%3DNcrVnw6cYJ0a85Obtqog%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eduardo Silva

Open Source, Treasure Data

http://www.treasuredata.com/opensource

 

[Samit Pal]

1. As I said, I'm using public CA signed certificate (Entrust intermediate) on the fluentd side. On the fluent-bit host I'm just trying the following config

-----------

tls on

tls.verfiy on

-----------

I get "flb_io_tls.c:305 X509 - Certificate verification failed, e.g. CRL, CA or signature check" error from fluent-bit. Note that the error goes away if i turn off tls.verify. I have used "openssl s_client  -connect <host>:<port> command from the fluent-bit machine to verify whether the certs are correct on the fluentd server and it works fine.

2. The next attempt that I made was with the following config

-----

tls on

tls.verfiy on

tls.ca_file "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"

-----

This is when I get the following error

[2018/06/15 10:42:56] [ info] [engine] started (pid=26123)

[2018/06/15 10:42:56] [error] [io_tls] flb_io_tls.c:109 PK - Read/write of file failed

[2018/06/15 10:42:56] [error] [TLS] Invalid CA file: "/etc/ssl/certs/Entrust_Root_Certification_Authority.pem"

[2018/06/15 10:42:56] [error] [output forward.0] error initializing TLS context

I can again confirm that when I use the same ca file with openssl command on the fluent-bit host it just works

$ openssl s_client -CAfile /etc/ssl/certs/Entrust_Root_Certification_Authority.pem -connect  <host>:<port>

--

Regards

Samit

[s...@getpolymorph.com]

We used to use the secure forward but we swapped away about two years ago. The problem was that it doesn't dynamically resolve DNS. This meant that we couldn't autoscale using an ELB. Also, ssl termination in ruby isn't as efficient as nginx or haproxy.

One caveat, Nginx also wont resolve upstream DNS unless you pay for it. This means that when the ELB updates its ip addresses you lose connection. If i were to do it all over again i'd set up VPN peering since network capacity managed by aws and skip the whole encryption discussion. Then you can just use the internal DNS entry from each region and forget about it.

[Sándor Guba]

If you have trouble setting up this deployment I suggest you to read this blog article: https://banzaicloud.com/blog/k8s-logging-tls/

This is a comprehensive working set-up for the described fluent-bit, fluentd tls. I hope you find it useful

To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+unsubscribe@googlegroups.com.

To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/e7cd94ce-e122-4fee-8ab4-22238e108ae1%40googlegroups.com.

[Samit Pal]

Thanks Sandor. I had referenced the doc that you mention earlier during my design phase.

@Eduardo,

Could the error from fluent-bit be because of the ssl certificate chain? I have the .pem cert in the following manner

-----BEGIN CERTIFICATE----- 
(My Primary SSL certificate: your_domain_name.crt) 
-----END CERTIFICATE----- 
-----BEGIN CERTIFICATE----- 
(My Intermediate certificate: entrust.crt) 
-----END CERTIFICATE-----

I have also tried the following without success

-----BEGIN CERTIFICATE----- 
(My Primary SSL certificate: your_domain_name.crt) 
-----END CERTIFICATE----- 
-----BEGIN CERTIFICATE----- 
(My Intermediate certificate: entrust.crt) 
-----END CERTIFICATE----- 
-----BEGIN CERTIFICATE----- 
(My Root certificate: entrust.crt) 
-----END CERTIFICATE-----

@s...@getpolymorph.com,

We had a similar issue with our nginx setup proxypassing to aws hosted kibana. We solved it by following https://distinctplace.com/2017/04/19/nginx-resolver-explained/ . Search for "Free Nginx resolver alternative".

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/e7cd94ce-e122-4fee-8ab4-22238e108ae1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fluent-Bit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluent-bit+...@googlegroups.com.
To post to this group, send email to fluen...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fluent-bit/CAFWQzYpuR-14z65ndtNrv%2B5W3TKRw7jcVDedX7vSJ8B2gvg_%2Bw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards

Samit