Shipping application logs via loggregator syslog drain to a logstash endpoint

[da...@castlelaing.com]

I'm interested in shipping my CF app logs via a loggregator syslog drain to a logstash/elasticsearch/kibana endpoint.

(specifically a logstash 1.2 syslog endpoint [1] and a logstash 1.2 parser [2] )

I have a loggregator syslog drain defined in my space, and I bind my app to that service, causing its loggregator logs to be forwarded:

gcf create-user-provided-service mylogcluster-syslog0 -l syslog://syslog-endpoint.mylogcluster.com
gcf bind-service monitor-bot-dev mylogcluster-syslog0

The logs arrive at my syslog endpoint looking like this:

353 <14>1 2013-12-03T09:41:11+00:00 loggregator 66dcf62e-703d-49d4-b478-319e80f47fd8 App - - {"@timestamp":"2013-12-03T09:41:11.040Z","Name":"Latency CIAPI.PriceStream","Value":"0.085928","@source.name":"pivotal-us-east-1-push1","@source.lonlat":[-78.450065,38.130112],"logger":"Monitors.Common.LatencyMonitor","level":"INFO"}

The bit output by my app is the JSON object at the end ({"@timestamp": ... "level":"INFO"})

Note that currently the loggregator syslog drain doesn't support encryption.  I think it should support RFC5425 (syslog over TCP with TLS) - see related discussion [3]

Doug asks:

Are all possible application/service and platform logs for a given unique customer CF environment available via loggregator?

Everything that your app writes to stdout + stderr is available.  I've also seen logs from the router and the stager.  Beyond that I'm not sure.  Anyone know if loggregator does / can ship service logs?

Are there differences in what types of logs available via loggregator versus a syslog drain? (or are they the same thing?)

No.  A loggregator syslog drain is just a way of getting at existing loggregator logs.

What scenarios would require having a log shipping agent (e.g. logstash-forwarder) actually within a client's CF app/service environment? (why did you add logstash-forwarder to your buildpack?)

Once loggregator syslog drains can be secured, I don't think there is any reason to have a log shipping agent "inside" your app 

What meta data is available for each log source/log record emitted from a given loggregator/syslog drain output? Will they all be uniquely identifiable to a specific app/service/client/group/org, etc. or will this data need to be added to a log stream externally? (e.g I need to be able to uniquely identify one type of log record from another for a given client app and act upon them differently)

Good question.  I know that you get an app's guid but not its instance id.  The component that loggregator got the log event from is identified (eg, APP, STG, RTR).  Obviously your app can write whatever it wants to stdout - I have mine add  "@source.name" and a "@source.lonlat" which is useful for drawing maps of where the logs are coming from. I'm not really sure how to lookup more details of the  app/service/client/group/org given the app guid; or how to integrate this information into your log store.

Finally, 

What tests can we work on?

How about these - https://github.com/cityindex/logsearch/pull/249

:D

[1] https://github.com/cityindex/logsearch/blob/master/srv/logstash/config/syslog_to_redis.conf.erb

[2] https://github.com/cityindex/logsearch/blob/master/srv/logstash/config/filter.d/syslog.conf.erb

[3] https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/3hXUl4szWI8/i4mHdjkcacQJ

On Tuesday, 3 December 2013, Doug McClure wrote:

David,

I'm interested in learning & experimenting!  I'm very much still trying to wrap my head around all the terminology, components, etc so definitely a noob here still on the CF front.

I'm looking for the best way to plumb a given environment's loggregator feed(s) and get them all shipped outbound towards some smarter thing that gives me more flexibility for mediating the log stream (breaking into unique flows by application/service component, routing to unique destinations, archiving, etc.)  In my case, there could be hundreds of unique customer CF application/service environments within the various CF environments so I'll need to be able to handle each unique environment's logs individually, securely, reliably, etc. without any impact on performance, etc.

We make use of Logstash today as a mediation tool so my first experiment would be to find the best way to ship logs out of each unique app/service environment that allows for scale, performance, reliability, flexibility, security, etc. This might be using logstash-forwarder, logstash, rsyslog, some hybrid thing, etc. Not sure yet - many open questions like these.

Are all possible application/service and platform logs for a given unique customer CF environment available via loggregator?
Are there differences in what types of logs available via loggregator versus a syslog drain? (or are they the same thing?)
What scenarios would require having a log shipping agent (e.g. logstash-forwarder) actually within a client's CF app/service environment? (why did you add logstash-forwarder to your buildpack?)
What meta data is available for each log source/log record emitted from a given loggregator/syslog drain output? Will they all be uniquely identifiable to a specific app/service/client/group/org, etc. or will this data need to be added to a log stream externally? (e.g I need to be able to uniquely identify one type of log record from another for a given client app and act upon them differently)

What tests can we work on?

Doug

[Tammer Saleh]

Great responses! The only thing I’ll add is that we aren’t yet emitting loggregator messages from services. You can see all of the places we emit by looking for Loggregator.emit:

 ~/code/cloudfoundry/cloud_controller_ng master $ ag Loggregator.emit 
./app/controllers/runtime/app_bits_controller.rb
39:        Loggregator.emit_error(guid, "Could not find package for #{guid}")

./app/controllers/runtime/apps_controller.rb
73:      Loggregator.emit(app.guid, "Created app with guid #{app.guid}")
87:      Loggregator.emit(app.guid, "Updated app with guid #{app.guid}")

./app/controllers/runtime/stagings_controller.rb
116:      Loggregator.emit_error(app_guid, "Did not find #{name} for app with guid: #{app_guid}")

./app/models/runtime/app_stop_event.rb
38:          Loggregator.emit(app.guid, "Tried to stop app that never received a start event")

./lib/cloud_controller/app_stager_task.rb
127:      Loggregator.emit_error(@app.guid, "exception handling first response #{e.message}")
138:      Loggregator.emit_error(@app.guid, "Encountered error: #{e.message}")
149:          Loggregator.emit_error(@app.guid, "Encountered error: #{e.message}")
183:      Loggregator.emit_error(@app.guid, "Exception checking staging status: #{e.message}")

./lib/cloud_controller/health_manager_respondent.rb
40:        Loggregator.emit_error(app_id, "Bad request from health manager: #{e.message}, payload: #{payload}")
67:        Loggregator.emit_error(app_id, "Bad request from health manager: #{e.message}, payload: #{payload}")
105:          Loggregator.emit_error(app.guid, "Bad request from health manager")

./lib/cloud_controller/hm9000_respondent.rb
35:        Loggregator.emit_error(app_id, "Bad request from health manager: #{e.message}, payload: #{message}")
57:        Loggregator.emit_error(app_id, "Bad request from health manager: #{e.message}, payload: #{message}")

./lib/cloud_controller/runner.rb
95:        Loggregator.emitter = LoggregatorEmitter::Emitter.new(@config[:loggregator][:router], "API", @config[:index], @config[:loggregator][:shared_secret])

./spec/loggregator_spec.rb
5:    before { Loggregator.emitter = nil }
9:      Loggregator.emit_error("app_id", "error message")
14:      Loggregator.emit("app_id", "log message")
22:      Loggregator.emitter = emitter
23:      Loggregator.emit_error("app_id", "error message")
29:      Loggregator.emitter = emitter
30:      Loggregator.emit("app_id", "log message")

Cheers,
Tammer Saleh

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Doug McClure]

Thanks David!  This is all great base information.

I think we have a few opportunities:

- Create a resource (webpage/wiki/gist/...) that catalogs all of the logging techniques, log sources and log messages examples available from the CF environment and associated components, services and apps. (by CF build #)  (your work and Dr Nic's work can provide a great start here)
- Request formal support for encrypted loggregator syslog drains
- Assess whether there is enough meta data associated with a given log message that it can be efficiently processed externally
- Assuming the use of LS, create grok patterns, codecs, etc. as needed to consume, parse and process CF log data
- Think about what's needed for scaling this out and providing resiliency, elasticity, performance, etc. (what's the loggregator and LS architecture look like, etc.)

Thoughts?

Doug

[James Bayer]

inline

On Tue, Dec 3, 2013 at 8:25 PM, Doug McClure <dmcc...@gmail.com> wrote:

Thanks David!  This is all great base information.

I think we have a few opportunities:

- Create a resource (webpage/wiki/gist/...) that catalogs all of the logging techniques, log sources and log messages examples available from the CF environment and associated components, services and apps. (by CF build #)  (your work and Dr Nic's work can provide a great start here)

this doc is the place where this logging info should go is here. it was mistakenly deleted awhile back. it still needs more info about the various log sources which can be .

https://github.com/cloudfoundry/cf-docs/blob/master/source/docs/using/deploying-apps/logging.html.md

 

- Request formal support for encrypted loggregator syslog drains

TLS is supported with and without cert validation depending on loggregator config.

 

- Assess whether there is enough meta data associated with a given log message that it can be efficiently processed externally

looking forward to you feedback, not sure of the criteria to assess

 

- Assuming the use of LS, create grok patterns, codecs, etc. as needed to consume, parse and process CF log data

currently it's syslog or "gcf logs APPNAME"

 

- Think about what's needed for scaling this out and providing resiliency, elasticity, performance, etc. (what's the loggregator and LS architecture look like, etc.)

loggregator architecture is shown here:

https://github.com/cloudfoundry/loggregator#architecture

there's a multi-az image here:

https://github.com/cloudfoundry/loggregator/tree/master/docs

the team did a bunch of scalability and resilency work as part of building loggregator. we can ask them to describe the tests they worked on and characterize what they know it can handle.

 

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

[Troy Astle]

Hi All,

I have been working to achieve exactly the same thing over the last couple of days. 

I understand the ability to bind a syslog_drain_url to an individual application as a user provided service is there an options to bind a drain globally or to all applications within an organization?

I have not found anything in the existing spec files to indicate this is a possibility and I am curious to know if it has been considered.

Thanks,

Troy.

[James Bayer]

troy, the original design was to have a "space log" whereby all app log streams in a space were put into a single loggregator stream and there could be one configuration in a space to send that whole stream to a syslog drain.

some users did not want logs from multiple apps mixed together and it also made the implementation more difficult. we adjusted this approach for loggregator to become a per-app focused log stream.

by use the syslog drain user provided service, you can bind all apps in a space to that single drain fairly easily by binding each to the same user-provided-service. there has been talk of adding a convenience method to "bind all apps in a space to a service" either in the cli (client orchestration) or in cloud controller (where all API clients could leverage it). so in the meantime you can just create a user provided service with the log drain and bind each app in the space to that.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Doug McClure]

Wouldn't flexible options to have drains at an app (and service level), a space level and an org level make sense?  I'd also want to stream and combine across multiple zones as well so I can get all logs headed to one centralized spot.

Doug

[James Bayer]

at a space level setting the log drain for all things (apps/services) in the space certainly could make sense and i totally understand the use case. i don't think it makes sense at an org level since apps and services are managed with space permissions.

for now i think we should stick with the app/service binding and learn a bit more and let loggregator get more mileage and use.

[Doug McClure]

I just want to make sure I can route and re-aggregate externally what a given org (company, LoB, dev team, etc.) might have across a given CF environment in various spaces. If I can identify that a logstream from a given space belongs to a certain org then I'm good, if not I'll need some other meta data to help with that.

Doug