Consolidated logging

[Hassan Schroeder]

I'm looking for recommendations on consolidating Rails logs from
multiple servers[*] into one.

[*] AWS EC2 instances running Ubuntu 16.04 + Ruby 2.3.x/Rails 5

Bonus points for workable integration with Docker instances.

Extra bonus points if the end point is a third-party service AND it
offers a HIPAA-compliant BA environment :-)

TIA!
--
Hassan Schroeder ------------------------ hassan.s...@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote

[Joe Guerra]

try this? 

https://newrelic.com/ruby

[Hassan Schroeder]

On Wed, Nov 23, 2016 at 7:51 AM, Joe Guerra <jgu...@jginfosys.com> wrote:
> try this?
> https://newrelic.com/ruby

Thanks, we already use NewRelic - even the free tier is super useful -
but that has nothing to do with exporting/consolidating logs.

>> I'm looking for recommendations on consolidating Rails logs from
>> multiple servers[*] into one.

--
Hassan Schroeder ------------------------ hassan.s...@gmail.com

http://about.me/hassanschroeder

[Stefan Frank]

probably not getting any Bonus points for pointing out that there are existing ami-images for setting up an Elastic-Stack(https://www.elastic.co), consisting of logstash (for sending out logfiles), elasticsearch (for collecting them centrally and making them searchable), and kibana (for visualising log-events). Integrating this into a ruby/rails-app is pretty straightforward, for rails there already exist log-adapters to properly json-format log-entries. We're also using this setup for sending out application-events we want to monitor (user logged in, purchase completed, item viewed etc.) - and then let logstash sort out where to put them. So apart from consolidating, this also comes in handy for splitting up logs into different buckets. Logstash had had its issues in the beginning, but has come a long way since Elastic has put it under its wings and now basically "just works"(tm). 

There are also existing docker-images for the whole elastic-stack, right now we just throw logstash in with our application-container and have the elasticsearch-server run in another container.

Works quite straightforward for us, reasonably easy to set-up, covers a wide range of scenarios, works for us, 

ok, now for half a bonus-point: There is also a hosted version called ElasticCloud (https://www.elastic.co/cloud/as-a-service), but it isn't exactly cheap and we haven't tested it...

[Frederick Cheung]

On Tuesday, November 22, 2016 at 5:35:32 PM UTC, Hassan Schroeder wrote:

I'm looking for recommendations on consolidating Rails logs from
multiple servers[*] into one.

[*] AWS EC2 instances running Ubuntu 16.04 + Ruby 2.3.x/Rails 5

Bonus points for workable integration with Docker instances.

Extra bonus points if the end point is a third-party service AND it
offers a HIPAA-compliant BA environment :-)

If you're on aws, then aws cloudwatch has a logs feature these days. It can do some things like creating metrics / alarms from logs, but it's not anything as full featured as logstash / elasticsearch/ kibana. If you just want to store logs for possible future analysis, it's fine though.

Fred

[Hassan Schroeder]

On Thu, Nov 24, 2016 at 2:17 AM, 'Stefan Frank' via Ruby on Rails:
Talk <rubyonra...@googlegroups.com> wrote:

> setting up an Elastic-Stack(https://www.elastic.co),

ELK is an option since we're already using Elasticsearch for text
search but I'd rather have log files stay in plain-text format.

TBD, might have to give on that... :-/

> application-events we want to monitor

That sounds useful, but in this case it's already handled by another
3rd-party service because events are also generated in iOS/Android
mobile clients.

Thanks for the perspective!

[Hassan Schroeder]

On Thu, Nov 24, 2016 at 2:20 AM, Frederick Cheung
<frederic...@gmail.com> wrote:

> If you're on aws, then aws cloudwatch has a logs feature these days. It can
> do some things like creating metrics / alarms from logs, but it's not
> anything as full featured as logstash / elasticsearch/ kibana. If you just
> want to store logs for possible future analysis, it's fine though.

This is an option, but the AWS web UI is painful at best, and there
doesn't appear to be (or I haven't found) any CloudWatch CLI that
provides access to the saved logs.

The main use case is troubleshooting errors where Honeybadger
doesn't provide enough context, if that helps clarify :-)

Thanks for the thoughts!

[Frederick Cheung]

On Thursday, November 24, 2016 at 8:25:05 PM UTC, Hassan Schroeder wrote:

On Thu, Nov 24, 2016 at 2:20 AM, Frederick Cheung
<frederic...@gmail.com> wrote:

> If you're on aws, then aws cloudwatch has a logs feature these days. It can
> do some things like creating metrics / alarms from logs, but it's not
> anything as full featured as logstash / elasticsearch/ kibana. If you just
> want to store logs for possible future analysis, it's fine though.

This is an option, but the AWS web UI is painful at best, and there
doesn't appear to be (or I haven't found) any CloudWatch CLI that
provides access to the saved logs.

You can export the logs to s3 ( http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html )

They've updated the cloud watch logs UI recently (within the past month or two I think). Still a little clunky, but better than it was

Fred