Bunyan to loggly, and best practices for a custom stream

[Scott Mebberson]

Hey there,

I've created a custom bunyan stream, that will send the logs through to loggly. It uses node-loggly and the loggly HTTP/S API. You can view the code here, https://github.com/smebberson/bunyan-loggly.

It's early days, but it works fine (as long as your configuration object is accurate).

I have a couple of questions:

• what is the best practice to report an error when creating a new bunyan logger (in this case, the node-loggly configuration object might not be accurate)
• what is the best practice to report an error on log.info (or other level), for example (a bad node-loggly request), at present it's dying silently which majorly sucks

I'd like to add support for these scenarios but unsure of how best to proceed. I notice the RingBuffer stream throws an error when it can't write to stream, but I also noted the following from the read me:

The reason for the above leniency is because IMO logging a message should never break your app. This leads to this rule of logging: a thrown exception from log.info(...) or equivalent (other than for calling with the incorrect signature) is always a bug in Bunyan.

I also intend on adding a basic buffer support.

Thanks for your help.

cheers,

Scott.

[Trent Mick]

I've created a custom bunyan stream, that will send the logs through to loggly. It uses node-loggly and the loggly HTTP/S API. You can view the code here, https://github.com/smebberson/bunyan-loggly.

Scott, that's great to hear.

 

I have a couple of questions:

• what is the best practice to report an error when creating a new bunyan logger (in this case, the node-loggly configuration object might not be accurate)

Is this a type of error that can always be determined synchronously? E.g. missing a required field in the config object? If so, then I'd throw an exception. For comparison, here is what happens if you try to `bunyan.createLogger` without the required 'name' field:

```
> bunyan.createLogger({})
TypeError: options.name (string) is required
    at new Logger (/Users/trentm/tm/node-bunyan-0.x/lib/bunyan.js:254:19)
    at Function.createLogger (/Users/trentm/tm/node-bunyan-0.x/lib/bunyan.js:1258:12)
    at repl:1:9
    at REPLServer.self.eval (repl.js:110:21)
    at Interface.<anonymous> (repl.js:239:12)
    at Interface.EventEmitter.emit (events.js:95:17)
    at Interface._onLine (readline.js:202:10)
    at Interface._line (readline.js:531:8)
    at Interface._ttyWrite (readline.js:760:14)
    at ReadStream.onkeypress (readline.js:99:10)
```

 

• what is the best practice to report an error on log.info (or other level), for example (a bad node-loggly request), at present it's dying silently which majorly sucks

Ideally I think I'd want node-loggly to buffer and retry (e.g. using node-backoff or whatever). You say that is planned below, so that's cool. When that ultimately fails, perhaps emiting an 'error' event on the logger? Here is an example of the Bunyan Logger re-emitting error events on the 'file' stream type: https://github.com/trentm/node-bunyan/blob/master/lib/bunyan.js#L375-L377

I'm not sure if non-core stream types have a way to get a handle on the Logger object though to set that up. If not, perhaps we'd want the Bunyan Logger to always pass a ref to itself to a stream when adding the stream. If this is something you'd need, let's discuss that.

The RingBuffer `throw` example is an exception I think. In that case the `throw` is when one attempts to log to that stream after having already explicitly `.destroy()`d it. IOW, usage error instead of runtime error.

Cheers,
Trent

[Scott Mebberson]

Hey Trent,

Thanks for getting back to me. Much appreciated.

When creating a node-loggly instance, it already throws if it doesn't have the required information. So that's good as it is directly inline with your suggestion and I didn't even have to write any code!

I like the idea of an ultimate fail emitting an error event. I'm going to implement testing as is, then get to a simple buffer, then get to write-fail-retrys using node-backoff. Once I'm at that point, I'll let you know if I have the necessary access to the Bunyan Logger. From I can see, that isn't the case. Maybe it's just better to make my class inherit from EventEmitter? That solves the problem and keeps things less coupled?

cheers,

Scott.

[Scott Mebberson]

Hi Trent,

FYI, I've published version v0.0.3 of bunyan-loggly which now has buffering support (I'm yet to add failure support through node-backoff however).

Further to my previous question (which I've just sent through) I had another thought...

The bunyan-loggly stream requires a raw stream type. The reason being, I need to change the name of the time property to timestamp. This is so loggly will recognise it as the timestamp of the action being logged.

Is there a means for me to determine that type: 'raw' has been declared when someone is setting up the stream?

Or am I forced to use regexp so that it works either with an object or string. I've got it working with an object so far, but not a string. I can do it with a regexp no problems, but I just wondered if there was a better bunyan style of checking for a raw stream.

Thanks!

cheers,

Scott.

[Trent Mick]

On Sun, Mar 16, 2014 at 8:34 PM, Scott Mebberson <sc...@scottmebberson.com> wrote:

The bunyan-loggly stream requires a raw stream type. The reason being, I need to change the name of the time property to timestamp. This is so loggly will recognise it as the timestamp of the action being logged.

Definitely. I would expect it to work with "raw" streams only.

 

Is there a means for me to determine that type: 'raw' has been declared when someone is setting up the stream?

Not really currently. Honestly I'd have your stream just blow up (throw) if, from the first record, it isn't 'raw'. That would be a usage error. I wouldn't support non-raw (i.e. a JSON.stringify'd string) record coming through).

The bunyan "streams" interfaces here could really be cleaner. As it is there are two-levels of things call "stream"... and for examples like yours (a similarly for https://github.com/trentm/node-bunyan-winston/blob/master/restify-winston.js#L66-L74) this is a bit awkward.

I don't know when I'll have a chance to get to such a clean up tho. A long while back I'd started a bunyan 1.x branch to make Bunyan more node.js-streamy, but I haven't been able to work on that for quite a while.

Cheers,
Trent

--
Trent Mick