make_format.pl - and a bug in lnav?

[Paul Wayper]

Hi all,

Because computers are much better at filling out templates than I am, I wrote make_format.pl, which is in my lnav_formats git repository.  An example use is:

tail -4 example.log | perl make_format.pl -n example -o example_log.json

It takes lines from stdin, escapes their " and \ characters, and turns them into sample lines in a basic template for a log format.  You can then edit this file to improve it.

It does not try to guess a regex pattern from what it's given, though that is something I'm working on.

The bug: I thought of using it by marking lines in lnav and issuing:

:pipe-to perl make_format.pl

(which just prints the format to stdout.)  But this didn't actually seem to produce any output to the pipe.  Testing with 'echo' showed that no lines were output from this.

Thanks in advance,

Paul

[Timothy Stack]

On Wed, Jul 8, 2015 at 11:03 PM, Paul Wayper <pau...@mabula.net> wrote:

Hi all,

Because computers are much better at filling out templates than I am, I wrote make_format.pl, which is in my lnav_formats git repository.  An example use is:

tail -4 example.log | perl make_format.pl -n example -o example_log.json

It takes lines from stdin, escapes their " and \ characters, and turns them into sample lines in a basic template for a log format.  You can then edit this file to improve it.

It does not try to guess a regex pattern from what it's given, though that is something I'm working on.

​This is neat, thanks for sharing!

​

The bug: I thought of using it by marking lines in lnav and issuing:

:pipe-to perl make_format.pl

(which just prints the format to stdout.)  But this didn't actually seem to produce any output to the pipe.  Testing with 'echo' showed that no lines were output from this.

Hmm, it seems to be working ​for me on the top-of-tree, I don't remember

if I fixed any issues around this recently, though.  Here's what I see:

  

Thanks in advance,

Pau

​l

​tim​

[Paul Wayper]

Hmmm, well something terrible has gone wrong for me - I've tried to compile head and it failed for some reason.  Git bisect told me:

$ git bisect visualize 

commit 5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff

Author: Timothy Stack <redacted>

Date:   Sun Jul 12 09:57:02 2015 -0700

    [pt] pass search query to papertrail

commit 7ef92dab79877f184622f9d33b9dc8b41248b303

Author: Timothy Stack <redacted>

Date:   Sat Jul 11 16:32:48 2015 -0700

    [format] try to validate formats against a given file

$ git bisect log

git bisect start

# bad: [d4e7f83a77e95d230f4748540b37a6454b6b50b5] try to use travis containers

git bisect bad d4e7f83a77e95d230f4748540b37a6454b6b50b5

# good: [87ec101f9c379ecd76dfa19e20a44c64d5ba6fb7] [fmts] log format contributions

git bisect good 87ec101f9c379ecd76dfa19e20a44c64d5ba6fb7

# good: [7679599afc1d3e71abac950098b8d185208941b0] fix a refactor error

git bisect good 7679599afc1d3e71abac950098b8d185208941b0

# good: [f392822aa8f7b4b5b4810211ba190c87f99f9e9e] [parse] show the path of the pattern used to parse the top line

git bisect good f392822aa8f7b4b5b4810211ba190c87f99f9e9e

# bad: [5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff] [pt] pass search query to papertrail

git bisect bad 5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff

Does this help you at all?

I'm running Red Hat Enterprise Linux 7 Workstation, so I know that it's not exactly cutting edge :-)

However, the good news is that the latest version I can get to compile does do as you say - :pipe-to works.  Dunno what was going on there.

Have fun,

Paul

[Timothy Stack]

On Mon, Jul 13, 2015 at 10:03 PM, Paul Wayper <pau...@mabula.net> wrote:

Hmmm, well something terrible has gone wrong for me - I've tried to compile head and it failed for some reason.  Git bisect told me:

$ git bisect visualize 

commit 5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff

Author: Timothy Stack <redacted>

Date:   Sun Jul 12 09:57:02 2015 -0700

    [pt] pass search query to papertrail

commit 7ef92dab79877f184622f9d33b9dc8b41248b303

Author: Timothy Stack <redacted>

Date:   Sat Jul 11 16:32:48 2015 -0700

    [format] try to validate formats against a given file

$ git bisect log

git bisect start

# bad: [d4e7f83a77e95d230f4748540b37a6454b6b50b5] try to use travis containers

git bisect bad d4e7f83a77e95d230f4748540b37a6454b6b50b5

# good: [87ec101f9c379ecd76dfa19e20a44c64d5ba6fb7] [fmts] log format contributions

git bisect good 87ec101f9c379ecd76dfa19e20a44c64d5ba6fb7

# good: [7679599afc1d3e71abac950098b8d185208941b0] fix a refactor error

git bisect good 7679599afc1d3e71abac950098b8d185208941b0

# good: [f392822aa8f7b4b5b4810211ba190c87f99f9e9e] [parse] show the path of the pattern used to parse the top line

git bisect good f392822aa8f7b4b5b4810211ba190c87f99f9e9e

# bad: [5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff] [pt] pass search query to papertrail

git bisect bad 5ef60412db146a7bdb47e0fb8b9f69bd9652b7ff

Does this help you at all?

​Looks like I messed the case where libcurl was not available.  I pushed a change that I

think should fix it.​

I'm running Red Hat Enterprise Linux 7 Workstation, so I know that it's not exactly cutting edge :-)

However, the good news is that the latest version I can get to compile does do as you say - :pipe-to works.  Dunno what was going on there.

​That's ​good to hear.

Have fun,

Pau

​l

​thanks,

tim​