mtail only working in oneshot mode?

[Robert Collier]

Hi,

I've returned to trying to get mtail to work for me. I'm using a fresh debian install with mtail 0.0+git20161027.a7b3e3c-1

mtail only seems to be working in oneshot mode. I'm just using linecount in testing.

Oneshot:

/usr/bin/mtail -progs /etc/mtail -logtostderr -logs /var/log/daemon.log -port 3904 -one_shot -one_shot_metrics

I1125 13:43:17.955885    7750 loader.go:83] Skipping /etc/mtail/README due to file extension.

I1125 13:43:17.956435    7750 loader.go:120] Loaded program linecount.mtail

Oneshot results:

I1125 13:43:17.956554    7750 mtail.go:47] Oneshot "/var/log/daemon.log"

/var/log/daemon.log: 4 MAXPROCS, 4 CPUs, I1125 13:43:17.956587    7750 vm.go:623] Starting program linecount.mtail

1009 lines, 41.525588ms total time, 41.155 µs/line

Metrics store:[

  {

    "Name": "line_count",

    "Program": "linecount.mtail",

    "Kind": 1,

    "LabelValues": [

      {

        "Value": {

          "Value": 1010,

          "Time": 1480081397998262498

        }

      }

    ]

  }

]I1125 13:43:17.998680    7750 mtail.go:265] Shutdown requested.

I1125 13:43:17.998709    7750 mtail.go:269] No tailer, closing lines channel.

I1125 13:43:17.998804    7750 loader.go:239] Shutting down loader.

I1125 13:43:17.999082    7750 log_watcher.go:57] Shutting down log watcher.

I1125 13:43:17.999134    7750 vm.go:628] Stopping program linecount.mtail

I1125 13:43:17.999172    7750 mtail.go:275] All done.

As a daemon:

/usr/bin/mtail -progs /etc/mtail -logtostderr -logs /var/log/daemon.log -port 3904 &

I1125 13:45:01.367338    7812 loader.go:83] Skipping /etc/mtail/README due to file extension.

I1125 13:45:01.367839    7812 loader.go:120] Loaded program linecount.mtail

I1125 13:45:01.368139    7812 vm.go:623] Starting program linecount.mtail

I1125 13:45:01.368172    7812 tail.go:315] Tailing /var/log/daemon.log

I1125 13:45:01.368771    7812 mtail.go:230] Listening on port 3904

wget -qO- http://localhost:3904/json

[

  {

    "Name": "line_count",

    "Program": "linecount.mtail",

    "Kind": 1

  }

]

Why might this be happening? How can I fix it? Thanks.

--

Robert Collier
r...@ossifrage.net

[Jamie Wilkinson]

Have any new lines been written to daemon.log ? :)

You can increase the verbosity level of the logging with --v=2, which will spew out a lot more.

Once you get some interesting logs, you can refine it with --vmodule instead, e.g.:

--vmodule=tail=2

would increase verbosity just for tail.go's logging.

The data flow should be that the log_watcher gets notified of a file system event when daemon.log changes, which triggers a read to EOF of that file by the tailer.  Each line is fed through a channel to the program loader, which fans them out into each VM.  The VM handles the log lines, including mutaing the exported variables.

Potential bugs:

 1. The fsnotify code doesn't work.

 2. The tailer isn't getting notified of a change to the log file

 3. A bug in the EOF read in tail.go

 4. A bug in the loader not passing log lines to VMs

 5. A bug in the exporter causing metrics to not be exported properly.

You mention this is the debian package.  I don't know how it's packaged, but assuming it's barely changed from the git repo, then I presume you should be able to check out that git commit from the final part of the version string on the master branch and be able to repro this, and if you can, I'd like to know what go compiler version you have, and the output of 'make check'.

Then the loader will 

The two places i'd be looking at first are the tailer, to ensure logs are being read from

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users+unsubscribe@googlegroups.com.
To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/CAO7542JG9Yku7bDT-%2B9Xgwg_QFJSfH4jo7W_DQM_Pw0KbT_Yjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jamie Wilkinson]

We're you able to figure this out?

To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.

To post to this group, send email to mtail...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/CAO7542JG9Yku7bDT-%2B9Xgwg_QFJSfH4jo7W_DQM_Pw0KbT_Yjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mtail-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mtail-users...@googlegroups.com.

To post to this group, send email to mtail...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mtail-users/CAHxKJaaMAhOSutErC9129YvjMAJ723t%2BnZZPEueXT-3Or2FOAw%40mail.gmail.com.