Anyone doing logging via a REST service?

[nickdu]

I'm curious whether anyone has attempted to implement a central logging REST service?  My initial thinking is that the overhead of the call will be too much for a logging service.  I'm guessing most applications want the overhead of logging to be extremely low.  Also, if each log is a separate call then it would seem that you can't take as much advantage of compression as you could if you sent an entire log file.

Has anyone implemented centralized logging in a PaaS/cloud environment?  One thought is to make use of the common logging solutions like log4j and then periodically ftp log files to a central server where they can be processed.  Processing might consist of sending the logs to HDFS or splunk which would allow for analytics.

Thanks,
Nick

[Matt Wright]

In my opinion the only situation in which HTTP would be an appropriate protocol for a logging service is for logging events in user facing applications in the browser or mobile devices. Otherwise there is already a plethora of tools for centralized logging in the context of server side applications. Additionally, if you were to implement an HTTP logging service you then might also consider developing a client library which can encapsulate optimizations such as as queuing messages and sending in bulk.

[Chris Mullins]

I think it depends on what you mean by a central logging service. 

For example, I've built rich client applications that send telemetry about usage to a service. The transport on this service is HTTPS, and somewhat (but not really) restful. That's certainly logging, at least for some definition. 

For something like at-scale request logging, I don't think a REST service would work. For that, logging to a file then bulk sending that file is really the only answer I see on a regular basis. 

There are all sorts of logging solutions that exist in the PAAS / IAAS world. Amazon and Microsoft both have an entire ecosystem around this, and I suspect Google does as well.

Cheers,

Chris

[Joel Pitt]

On 16 May 2015 at 14:40, nickdu <nick.p...@gmail.com> wrote:
> Has anyone implemented centralized logging in a PaaS/cloud environment? One
> thought is to make use of the common logging solutions like log4j and then
> periodically ftp log files to a central server where they can be processed.
> Processing might consist of sending the logs to HDFS or splunk which would
> allow for analytics.

One option: Write your logs to syslog, configure the local syslog
daemon to forward to the central logging server (which can be splunk
or anything else that is designed to work with syslog). I think you
can choose UDP or TCP. No need for ftp or any custom log transports.

syslog daemons also tend to have flood protection in case a process
gets stuck spamming out the same log message indefinitely.

If you have an ELK stack somewhere (elasticsearch/logstash/kibana),
there is https://github.com/elastic/logstash-forwarder

Joel

[James Higginbotham]

Hi Nick,

I would echo the previous recommendations: create a logging API if you need to capture client-based business events that you would otherwise not be able to capture on the server-side easily but need to for whatever reason. I did this in a past project to tie client and server-side user events together for analytics. We then used the ELK stack (with LogStash being replaced by Fluentd for this particular project) to centrally aggregate and archive to S3. We wrote to a local file, then had the local LogStash/Fluentd instance (on each EC2 instance) tail the log files and push to a central Fluentd that stored into ElasticSearch and archived to S3.

Over the years, I have seen developers try and recreate network services and APIs around fundamental stack needs that can be solved in other ways (including something as building scheduling when something as simple as Cron would work fine). So, first spend some time to validate the need is actually there and that you can't solve it outside of writing code (i.e. using tools like centralized logging with something like ELK or an equiv SaaS service). Then proceed as appropriate for the needs at hand. 

James