Use ELK drain logs from Loggragetor,[STG] logs lost, Other logs not complete

[wlj...@gmail.com]

These days I am engaged to show real time log on a page when user deploy their apps.but meet some problem.

here is my solution:

1.Using Logstash to drain log from loggregator. my logstash config file is in the end of the email.

2.Invoke Elasticsearch RestAPI to read the most recent log(in 3 seconds) every 3 seconds and show it on a webpage.

The follow is the problem I meet:

1.the logs of [STG] kind never been drained. (I query it bothing using REST API and Kibana).

2.Other kind of logs sometimes not the same with the log I got from CLI(less than CLI,example is in eht end of email).

It seems there are someting wrong with my solution, because the CLI got the right logs.but after searching for days I still don't know where the problem is.

Any kind of advice will be helpful. Because I am a novice to CF.

ps. Please forgive my poor English...

[Alexander Jackson]

Hi,

   You are correct that the [STG] (staging) logs are never sent to the syslog drain.   This a known issue due to current system limitations.    The CF staging process doesn't have knowledge of the syslog drain and thus can't forward the information to loggregator.   I believe that if you have a version of the application running that has it's log forwarded to a syslog drain, you could get staging logs when you push a new version or re-stage that application.    We're working on fixing this but it will most likely require the new diego component before we can properly address this limitation.  

If you really need these logs, you can change your integration with loggregator from a syslog drain to the new streaming websocket endpoint.   This is the same endpoint that the cli uses and would allow you to get all the logs and even some metric data for your application.   See the loggregator readme[1] for more info on this endpoint and the data format.

         - Alex.

[1] https://github.com/cloudfoundry/loggregator#consuming-log-and-metric-data

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/2463b51b-3629-4b1c-9afa-14bffba946de%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[吴立军]

Thank you very much Alex! The information you provide me is very useful, I am trying to use the streaming websocket to get current log :-D

在 2014年10月17日星期五UTC+8下午10时42分22秒，ajackson写道：

[David Laing]

Hi,

My team is looking at consuming the firehost into logstash / ELK.

Would you be interested in collaborarting?

--

David
logsearch.io - build your own open source cloud logging cluster
http://davidlaing.com

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/a52c683a-bb4d-4758-b8f6-0cfa9c701f18%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--
David Laing
Trading API @ City Index
da...@davidlaing.com
http://davidlaing.com
Twitter: @davidlaing