Tailing a file in Clojure

[viksit]

Hi all,

What would you recommend as the best method to tail a file using
Clojure? Are there any built in functions in contrib or core that
allow a program to read the last line of a file as it is appended to?
If not - how do people solve a problem like this?

My aim is simple - I've got a log file and I'd like to parse it as it
gets appended to.

Thanks
Viksit

[Ken Wesson]

Well, there's always (last (line-seq foo)) ...

[Alex Osborne]

viksit <vik...@gmail.com> writes:

Tail just polls the file every 1 second or whatever (it's configurable
-s option) and check's if the file's length has changed. You can do
this yourself easily enough.

Some discussion here (it's Java, but you'd do exactly the same in Clojure):

http://stackoverflow.com/questions/557844/java-io-implementation-of-unix-linux-tail-f

When I've had this problem myself (on unix), I've just been lazy and
implemented it by doing:

tail -f somefile.log | java ...

Or shelling out to `tail' from within the program. Ugly perhaps, but I
know GNU tail behaves the way I want when the file is truncated and
such. :-P

[kkw]

Hi Viksit,

For production support, I also wanted to "tail -f" a file on a
Windows server with no "tail" facility (no one had installed the
Windows Resource Kit yet, and executing the install file myself
would've been prohibitive (it's a long story)). However, this Windows
server did have Java. I thought to write a "tail -f" with clojure
until I discovered the JDK installed was 1.4.2.

I found an old version of Jython that was compatible (at the time,
I used jython 2.1 and 2.2) and wrote a Jython script that implemented
"tail -f" for me. It's not clojure, but it did solve my initial
problem, which was doing a "tail -f" on a log file. I didn't build any
parsing facility because I was content to have the file write out to a
Windows GUI dialog.

I realise it's not exactly what you want, but it's ready (with no
guarantees that it's safe, as with any code received from strangers)
and located in it's entirety at:

https://sites.google.com/site/kevinkwoo/work/guitail-fforjython21-1

It should be publicly accessible, but if it's not do let me know!

Regards,

Kevin

[patrickdlogan]

Java has a file watch API to avoid polling. Stuart Sierra uses it to
good effect in lazytest.

On Dec 2, 9:14 am, Alex Osborne <a...@meshy.org> wrote:
> viksit <vik...@gmail.com> writes:
> > What would you recommend as the best method to tail a file using
> > Clojure? Are there any built in functions in contrib or core that
> > allow a program to read the last line of a file as it is appended to?
> > If not - how do people solve a problem like this?
>
> > My aim is simple - I've got a log file and I'd like to parse it as it
> > gets appended to.
>
> Tail just polls the file every 1 second or whatever (it's configurable
> -s option) and check's if the file's length has changed.  You can do
> this yourself easily enough.
>
> Some discussion here (it's Java, but you'd do exactly the same in Clojure):
>

> http://stackoverflow.com/questions/557844/java-io-implementation-of-u...

[Saul Hazledine]

I have had to do this in the past and I use unix commands:

https://gist.github.com/726875

The code above runs a unix command in a separate thread and executes a
given function, f, on a sequence of output lines. It handles restarts
if the command gets killed by a log rotation.

For the actual tail command I'd recommend looking at a recent version
of GNU tail or install inotail - a filewatching API which makes these
commands much more efficient:

http://distanz.ch/inotail/

Saul

[Alex Osborne]

patrickdlogan <patric...@gmail.com> writes:

> Java has a file watch API to avoid polling.

I assume you're talking about the NIO 2 watch service? That's not
yet in a released version of Java, it's coming in Java 7.

> Stuart Sierra uses it to good effect in lazytest.

It looks to me like lazytest polls the last modified date every 500 ms.

http://github.com/stuartsierra/lazytest/blob/master/modules/lazytest/src/main/clojure/lazytest/watch.clj

[Stuart Sierra]

On Dec 2, 10:55 pm, patrickdlogan <patrickdlo...@gmail.com> wrote:
> Java has a file watch API to avoid polling. Stuart Sierra uses it to
> good effect in lazytest.

No, there is no such Java API that I am aware of. Lazytest watches
the filesystem by polling.

-S

[Alex Miller]

This is being added with JSR 203 in Java 7:
- http://java.dzone.com/articles/introducing-nio2-jsr-203-part-2

There was a backport for JDK 6 started: http://code.google.com/p/jsr203-backport/
but I don't think it's been touched in a couple years.

[patrickdlogan]

On Dec 3, 4:42 am, Alex Osborne <a...@meshy.org> wrote:

> patrickdlogan <patrickdlo...@gmail.com> writes:
> > Java has a file watch API to avoid polling.
>
> I assume you're talking about the NIO 2 watch service?  That's not
> yet in a released version of Java, it's coming in Java 7.

oh I see.

[Todd]

As an exercise in learning clojure, I implemented a basic binary search
tree in clojure:

https://gist.github.com/727982

Any feedback would be appreciated.

-Todd

[ax2groin]

Does any familiar with this NIO Watch service know if it handles NFS
issues? We have an in-house log monitoring tool at work (doesn't
everyone) which is written in Java, but the two big issues are message
framing (knowing when a multi-line message has ended) and getting null
bytes when tailing an NFS file. We have some fairly heavy code to
handle these problems. I've thought about trying to write something
similar in Clojure as a project, but haven't started to tackle it yet
precisely because I didn't see any way to make it more efficient.

[Miki]

http://jnotify.sourceforge.net/ ?