Draining loggregator logs via syslog into LogStash

[david...@labs.cityindex.com]

Dear loggregator gurus,

James mentioned (https://groups.google.com/a/cloudfoundry.org/forum/#!topic/vcap-dev/lVLLvnmXG_g) that loggregator will allow you to drain your application logs into a 3rd party log analysis tool via syslog.

I'm working on adding support for this CF syslog drain for the logsearch (logstash/elasticsearch/kibana3) log analysis tool my team works on - https://github.com/cityindex/logsearch/issues/224

I have a couple of questions I'd like to ask concerning the origin and "security" of the loggregator syslog drain.

Specifically:

• For the purposes of opening firewall ports, where does a CF cluster's loggregator syslog drain originate?  Is it a fixed IP / cluster?  What is it for run.pivotal.io?
• Is the syslog data flow encrypted?  If so how?

Apologies if these questions are premature.  Answers along the lines of "we don't know yet, here are the options we are considering" would be really appreciated too.

Thanks!

David

[Tammer Saleh]

• For the purposes of opening firewall ports, where does a CF cluster's loggregator syslog drain originate?  Is it a fixed IP / cluster?  What is it for run.pivotal.io?

Depends on your installation.  run.pivotal.io uses a set of NAT (network address translation, not NATS) boxes, so they would be the source address.  Otherwise, it'd be the address of your loggregator servers. 

• Is the syslog data flow encrypted?  If so how?

No, it's not.  Is there a standard syslog encryption?  If so (and if the syslog partners in the wild support it), then I'd love to hear about it.

[David Laing]

Thanks Tammer.

I've bumped into syslog over TCP secured with TLS in the LogStash and the Loggly docs.  I'll dig into the documentation and see if this is a standard

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[david...@labs.cityindex.com]

Tammer,

Syslog over TCP with TLS seems to be the "standard" to ship syslogs securely - see http://tools.ietf.org/html/rfc5425 

Loggly gives the following instructions for syslog with TLS

• syslog-ng - http://community.loggly.com/customer/portal/articles/1225987-syslog-ng-configuration#tls
• nglog - http://community.loggly.com/customer/portal/articles/1266344-nxlog-windows-configuration 

Papertrail has these instructions:

• rsyslog - http://help.papertrailapp.com/kb/configuration/encrypting-remote-syslog-with-tls-ssl#rsyslog 
• syslog-ng - http://help.papertrailapp.com/kb/configuration/encrypting-remote-syslog-with-tls-ssl#syslog-ng

LogStash supports syslog via a TCP endpoint, which also supports TLS/SSL

• http://logstash.net/docs/1.2.2/inputs/tcp#ssl_cacert

Seems to me that TLS is the "standard" encryption used for syslog.

Do you agree?

D

[James Bayer]

tammer is out of the office for a few weeks.

it turns out that syslog over TSL or SSL is not implemented yet.

i agree that this is a very desirable capability. the loggregator team will discuss it next week.

--

Thank you,

James Bayer

[Tammer Saleh]

Thanks for doing the research on that, David!  Looks like a pretty standard solution, so I'm looking forward to seeing the loggregator team discuss it while I'm out.

Cheers,

Tammer Saleh

[Sreekanth Iyer]

Just checking back if there is any update on whether the Loggregator for CF currently supports syslog TSL and SSL?

[Christopher Ferris]

Sreek,

I believe so. Please see the docs [1]

[1] http://docs.cloudfoundry.org/devguide/services/log-management.html

Chris

[David Lee]

Yes, loggregator supports syslog-tls drains.

-Dave

On Sun, Apr 13, 2014 at 11:41 PM, Sreekanth Iyer <sreek...@gmail.com> wrote:

Just checking back if there is any update on whether the Loggregator for CF currently supports syslog TSL and SSL?

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.