Setup lograge, logstash-logger

[belgoros]

Can anybody provide some links to how to set up correctly Lograge, logstash-logger with a Rails API app ? I followed their READMEs but still nothing sent to Kibana :(.

Here is how these gems are declared in Gemfile:

group :development do

  gem 'lograge',         '~> 0.10.0'
  gem 'logstash-event',  '~> 1.2'
  gem 'logstash-logger', '~> 0.26.1'
end

I added the following lines in  config/environments/development.rb:

config.lograge.enabled = true
  config.lograge.base_controller_class = 'ActionController::API'
  config.lograge.formatter = Lograge::Formatters::Logstash.new
  #config.lograge.logger = LogStashLogger.new(uri: ENV['logstash_uri'], verify_hostname: false)
  config.lograge.logger = LogStashLogger.new(host: ENV['logstash_host'], port: ENV['logstash_port'], verify_hostname: false)
  config.lograge.custom_options = lambda do |event|
    { name: "decastore-development" }
  end

As you see, I tried 2 different ways to initialize LogStashLogger, but still without success.

What's wrong with that ? 

I'm using Rails 5.2.0 (API only), ruby 2.5.0.

Thank you

[belgoros]

Even moving the above declaration to config/application.rb:

module MyAppApi

  class Application < Rails::Application


...
config.api_only = true

   
    config.lograge.enabled = true
    config.lograge.base_controller_class = 'ActionController::API'
    config.lograge.formatter = Lograge::Formatters::Logstash.new

    config.lograge.logger = LogStashLogger.new(type: :tcp, host: ENV['logstash_host'], port: ENV['logstash_port'])

    config.lograge.custom_options = lambda do |event|

      { name: "myapp-development" }
    end

didn't send anything to Kibana.

[Hassan Schroeder]

On Tue, Aug 14, 2018 at 6:21 AM, belgoros <s.ca...@gmail.com> wrote:

>> Can anybody provide some links to how to set up correctly Lograge,
>> logstash-logger with a Rails API app ? I followed their READMEs but still
>> nothing sent to Kibana :(.

>> config.lograge.logger = LogStashLogger.new(host: ENV['logstash_host'],
>> port: ENV['logstash_port'], verify_hostname: false)

Never used either of these but had a little play.

My impression is that LogStashLogger defaults to a UDP connection,
so you might try with e.g.

LogStashLogger.new(type: :tcp, host: ENV['logstash_host'], ...)

HTH,
--
Hassan Schroeder ------------------------ hassan.s...@gmail.com
twitter: @hassan
Consulting Availability : Silicon Valley or remote

[belgoros]

On Tuesday, 14 August 2018 22:53:44 UTC+2, Hassan Schroeder wrote:

On Tue, Aug 14, 2018 at 6:21 AM, belgoros <s.ca...@gmail.com> wrote:

>> Can anybody provide some links to how to set up correctly Lograge,
>> logstash-logger with a Rails API app ? I followed their READMEs but still
>> nothing sent to Kibana :(.

>>   config.lograge.logger = LogStashLogger.new(host: ENV['logstash_host'],
>> port: ENV['logstash_port'], verify_hostname: false)

Never used either of these but had a little play.

My impression is that LogStashLogger defaults to a UDP connection,
so you might try with e.g.

LogStashLogger.new(type: :tcp, host: ENV['logstash_host'], ...)

Yeas, I knew that. I tried both tcp and udp, none of them worked. The only one that works is when I specify the type as file:

config.lograge.logger = LogStashLogger.new(type: :file, path: 'log/development.log', sync: true)

This way the logs shall be written in Logstash format in development.log. 

The question I have not found the answer to is what host and port values should correspond to ? Are they the values of Logstash server ?

[Hassan Schroeder]

On Thu, Aug 16, 2018 at 1:04 AM, belgoros <s.ca...@gmail.com> wrote:

> Yeas, I knew that. I tried both tcp and udp, none of them worked. The only
> one that works is when I specify the type as file:
>
> config.lograge.logger = LogStashLogger.new(type: :file, path:
> 'log/development.log', sync: true)

Taking "lograge" out of the picture for the moment, using this config:

config.logger = LogStashLogger.new(type: :udp, host: '127.0.0.1',
port: 5228, verify_hostname: false)

> This way the logs shall be written in Logstash format in development.log.
> The question I have not found the answer to is what host and port values
> should correspond to ? Are they the values of Logstash server ?

I do this below in a Rails console:

Loading development environment (Rails 5.2.1)
2.5.1 (main):0 > Rails.logger.info("test message from a rails app")
=> true
2.5.1 (main):0 > Rails.logger.flush()
=> true
2.5.1 (main):0 >

with this test endpoint (Erlang, but that's not important)

Eshell V9.2 (abort with ^G)
1> {ok, Logstash} = gen_udp:open(5228, [binary, {active,false}]).
{ok,#Port<0.490>}
2> gen_udp:recv(Logstash, 0).
{ok,{{127,0,0,1},
49369,
<<"{\"message\":\"test message from a rails
app\",\"@timestamp\":\"2018-08-16T13:25:48.113-07:00\",\"@versio"...>>}}
3>

Note that nothing showed up on receiving end until I flushed the logger
on the Rails side, so that might have something to do with your testing
not apparently doing anything. But for sure, the `host` and `port` values
are for the remote logstash endpoint.

HTH!

[belgoros]

Thank you for your time. Yeah, I also tested Lograge/Logstash from inside the rails console in the deployed docker container:

rails c

logger = LogStashLogger.new(type: :tcp, host: 'logstash_host', port: logstash_port, verify_hostname: false)

logger.info 'logstash-draft => test port XXX'

and I could see the produced log message in Kibana dashboard.

The problem is now on Logstash server configuration side :) (devops will take a look at that).