Spawn with tail and grep

[Emerson Macedo]

Hi,

I'm trying to spawn with tail -f somefile | grep sometext but with -f it doesn't work. It I remove -f it works but of course do not wait for new log entries. I'm writing a web log viewr.

Here is the gist: https://gist.github.com/938632

Thanks,

Emerson Macedo

[Isaac Schlueter]

Yeah, that is very interesting.

It looks like the "grep" process is buffering for a pretty long time
before processing data. For instance, if you save that code to t.js,
and then do this:

touch nohup.out
node t.js > test-out.log &
while true ; do echo HEAD >> nohup.out ; done

then let that go for a few seconds, you'll probably find that
"test-out.log" gets one "grep" output for about every 3000 lines of
"HEAD" output. The only way that I can see to force grep to output
what's in its buffer is to kill the process, but of course, you want
the grep to keep running. I'm not sure why it seems to work just fine
by doing `tail -f nohup.out | grep HEAD`. Maybe something to do with
nonblocking stdio?

Unless you're using features very specific to grep, you might have an
easier time just filtering the log output in javascript.

> --
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en.
>

[mscdex]

On Apr 23, 4:00 pm, Isaac Schlueter <i...@izs.me> wrote:
> It looks like the "grep" process is buffering for a pretty long time
> before processing data.  For instance, if you save that code to t.js,
> and then do this:

Hmm, I'm not able to reproduce this on ubuntu. However it does seem to
be a known problem with some utilities (including grep). Try using the
grep option --line-buffered to have it flush output on every line.

[mscdex]

Handy link fwiw: http://mywiki.wooledge.org/BashFAQ/009

[Floby]

If it is of any interest to you, Gnagno and I have been working on a
similar app (also using tail -f to watch log files). The grepping
thing however is done on the client side with javascript.
https://github.com/Floby/log-file-watcher
It is still work in progress but the pages for individual files work
fine. Use `node app.js -h` to see how it's used. Also I haven't
written a package.json file yet so the dependencies are express, jade,
socket.io and lines. All of them are available on npm. Of course you
need tail, but you already have this one =)

I'd be happy to have your feedback on this, if you think we're going
in the wrong direction for real world use.

[Franz Enzenhofer]

hi, you might wanna check out

https://github.com/franzenzenhofer/nolog

npm install nolog

it is basically a logfile watcher / parser. (actually it can parse any readable stream (stdout, https, http, files))

it maps patterns (i.e. regular expressions) to events. 

oh yeah, it uses a spawn for tail, but grepping is done via javascript as i ran into a lot off issues / limitations with the tail | grep pipe.

lg

franz enzenhofer

[Eugene Ventimiglia]

How are you guys dealing with logfile rotation? Typically logrotate will rename a file and HUP the service that writes the log so that a new log will be created - If you're using a spawned tail -f, it will be attached to the old file (after the rename) 

I did this a year or so in perl, and I had to restart the log watching daemon in the logrotate script, but I'm wondering how you guys are handing it - I have need for a similar thing, and I'd like to do it in node

[Marcello Bastéa-Forte]

For what it's worth...

     -F      The -F option implies the -f option, but tail will also check to see if the file being followed

             has been renamed or rotated.  The file is closed and reopened when tail detects that the file-

             name being read from has a new inode number.  The -F option is ignored if reading from standard

             input rather than a file.

Marcello

[Eugene Ventimiglia]

I guess there's always the RTFM method :) - I was using File::Tail in perl, so it didn't apply to my last project but -F makes sense here.

[Emerson Macedo]

Thanks guys. With --line-buffered it worked. I'm taking a look at your projects and surely I'll extend one of them.

Cheers,

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

--
Emerson Macedo