node.js cluster multiprocess logging solution

[gen chen]

I am now working on a node.js project based on cluster.  I got stuck on the logging.  After doing some research, I worked out a solution. here is it. i don't know if it is a good idea.  The idea is like this.  only master process can wirte to the log file, if the current process is a worker, then it send a log message to the master and then write to the log file while the master can directly write to the log file. this can avoid multiple process open and write to a same file. 

var util = require('util');

var fs = require('fs');

var cluster = require('cluster');

var logger = module.exports;

var levels  = ['debug', 'info', 'warn', 'error', 'fatal'];

var logLevel = 'debug';

var logfile = null;

var errorLogfile  = null;

if(cluster.isMaster){

    logfile = fs.createWriteStream('debug.log', {flags:'a'});

    errorLogfile = fs.createWriteStream('error.log', {flags:'a'});

    cluster.on('online', function(worker){

        //collect log message from child and write to logfile.

        worker.on('message', function(msg){

            if(msg.type == 'logging') {

                var level = msg.data.level;

                var logStr = msg.data.msg;

                if(levels.indexOf(level) >= levels.indexOf('error')){

                    errorLogfile.write(logStr + '\n');

                }else{

                    logfile.write(logStr + '\n');

                }

            }

        });

    });

}

function log(level, args){

    if(levels.indexOf(level) < levels.indexOf(logLevel)) return;

    var args = Array.prototype.slice.call(args);

    args = args.map(function(a){

        if(typeof a !== 'string') 

            return JSON.stringify(a);

        else return a;

    });

    var msg = util.format.apply(null, args);

    var out = [];

    out.push(new Date());

    out.push('[' + level.toUpperCase() + ']');

    out.push(msg);

        

    if(cluster.isMaster){

        //write directly to the log file

        if(levels.indexOf(level) >= levels.indexOf('error')){

            errorLogfile.write(out.join(' ') + '\n');

        }else{

            logfile.write(out.join(' ') + '\n');

        }

    }else{

        //send to master

        cluster.worker.process.send({

            type : 'logging', 

            data : {

                level : level,

                msg : out.join(' ')

            }

        });

    }

}

logger.debug = function(){log('debug', arguments);}

logger.info = function(){log('info', arguments);}

logger.warn = function(){log('warn', arguments);}

logger.error = function(){log('error', arguments);}

logger.fatal = function(){log('fatal', arguments);}

[Jose Luis Rivas]

I would suggest you to not write directly but redirect output to a file.

node app.js > file.log

keep it simple.

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to nodejs+un...@googlegroups.com
> <mailto:nodejs+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

--
Jose Luis Rivas - http://joseluisrivas.net
Venezuela - GPG: 0xB9AC8C43

[Alexey Petrushin]

Adam Wiggins, one of creator of Heroky wrote interested post about logging, you may find it interesting

http://adam.heroku.com/past/2011/4/1/logs_are_streams_not_files/

[hij1nx]

Yeah, but don't log to disk if you care about performance,

which I'm guessing you do since you're using cluster :)

Here's why -- https://medium.com/node-js-javascript/37a93d4e0013.

You may also want to look at this -- https://github.com/hij1nx/net-log

--

https://twitter.com/hij1nx

https://github.com/hij1nx

[Dan Peddle]

Interesting comment, surely asynchronously writing to a file isn't going to have much of an impact...? Are there any examples around of what kind of an effect it could have...?

D

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines

---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/01359a4a-ef2d-46b4-ae0d-f3505ac32f98%40googlegroups.com.

[Paolo Fragomeni]

I should also add, if you know you're running on good hardware, its probably not that big of a deal.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/t0SyYRkRR-A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACTUpAccoCMAAK-Zj7FE7c8h8mivWqjcuVKGNhSnrDLdHp-BMw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Paolo Fragomeni

Founder, Here is How

http://hereishow.to

github.com/hij1nx

twitter.com/hij1nx

[Paolo Fragomeni]

It really depends on the machine you're running on, here is some more info -- http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf

On Mon, May 12, 2014 at 1:03 PM, Dan Peddle <dan.p...@gmail.com> wrote:

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/t0SyYRkRR-A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACTUpAccoCMAAK-Zj7FE7c8h8mivWqjcuVKGNhSnrDLdHp-BMw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Paolo Fragomeni]

An updated benchmark can be found here -- https://github.com/hij1nx/net-log#performance

[Tim Dickinson]

Take a look at https://github.com/MangoRaft/Logger

Its not ready for use yet but could be something you would use.

In the next few days I'll be pushing a new version that is independent to raft and could be used by any program.

[Ryan Graham]

You got a lot of feedback and suggestions. What did you end up doing to solve this?

~Ryan

On 28 April 2014 03:45, gen chen <chenge...@gmail.com> wrote:

I am now working on a node.js project based on cluster.  I got stuck on the logging.  After doing some research, I worked out a solution. here is it. i don't know if it is a good idea.  The idea is like this.  only master process can wirte to the log file, if the current process is a worker, then it send a log message to the master and then write to the log file while the master can directly write to the log file. this can avoid multiple process open and write to a same file. 

<code removed for brevity>

~Ryan

--
http://twitter.com/rmgraham

[WooD Fung]

you can have a try my library. "ln" is the fastest logging lib. under cluster environment, it supports date rotation on a single file and it is not using master logging mechanism.

https://github.com/wood1986/ln

[Ryan Graham]

That is a very interesting little module.

It's a shame that winston is so much slower that the benchmark script is actually annoying to run without omitting it.

To make that pain worse, I found a way of significantly speeding up ln. Pull requests welcome?

~Ryan

On 13 June 2014 07:26, WooD Fung <cw.woo...@gmail.com> wrote:

you can have a try my library. "ln" is the fastest logging lib. under cluster environment, it supports date rotation on a single file and it is not using master logging mechanism.

https://github.com/wood1986/ln

 

--
http://twitter.com/rmgraham

[WooD Fung]

(Please don't delete! i want to reply Ryan)

really? I thought ln had already reach the limit. Pull request please!! Thanks