mawk 2.0 beta

[mawk...@gmail.com]

https://drive.google.com/open?id=0B_Q-mbHiy9g-Ry1WY1BjVUtJX3M

is a google drive folder holding
about-mawk.pdf
mawk-1.9.9.2.tar.gz

This a beta release, which I believe is stable, and will soon be mawk 2.0

The pdf is two pages describing reasons for the release and what is new.

Feedback to Mike Brennan <mawk...@gmail.com>

[Kenny McCormack]

In article <dd0e322f-642f-469e...@googlegroups.com>,

1) I cannot read the second page of the PDF file. Error message is
"Whoops. Problem loading next page". Would it be possible to make the
file available as a simple download somewhere (instead of goofy Google) ?

2) While I understand your not wanting to use any of the code produced by
the other maintainers/developers, I am curious if your version includes the
bugs-fixed and features-added of those versions. In particular, I think
there was something discussed here recently that was fixed in the 1.3.4
version. Unfortunately, I can't locate it at the moment by searching this
group, but it may have been multi-character as either FS or RS or both.

I think somebody made the claim that allowing one or both of these to be
more than a single character was "gawk specific" and I objected to this
characterization on the grounds that other versions besides Gawk had it. I
believe that subsequent investigation showed that stock Mawk does not have
it, but that the 1.3.4 versions does.

--
A liberal, a moderate, and a conservative walk into a bar...

Bartender says, "Hi, Mitt!"

[Ed Morton]

On 7/30/2016 1:37 PM, mawk...@gmail.com wrote:
> https://drive.google.com/open?id=0B_Q-mbHiy9g-Ry1WY1BjVUtJX3M
>
> is a google drive folder holding
> about-mawk.pdf
> mawk-1.9.9.2.tar.gz
>
> This a beta release, which I believe is stable, and will soon be mawk 2.0

Why? If you have some free time to do awk tool development, why not contribute
that time to gawk development instead of some other variant?

Ed.

[lucas...@gmail.com]

Dear Mike,
Thank you very much for this. Love your work!

Kind Regards
Lucas

[Kaz Kylheku]

On 2016-07-31, Ed Morton <morto...@gmail.com> wrote:
> On 7/30/2016 1:37 PM, mawk...@gmail.com wrote:
>> https://drive.google.com/open?id=0B_Q-mbHiy9g-Ry1WY1BjVUtJX3M
>>
>> is a google drive folder holding
>> about-mawk.pdf
>> mawk-1.9.9.2.tar.gz
>>
>> This a beta release, which I believe is stable, and will soon be mawk 2.0
>
> Why? If you have some free time to do awk tool development, why not contribute
> that time to gawk development instead of some other variant?

You missed this:

>> Feedback to Mike Brennan <mawk...@gmail.com>

Mawk was originally written by Mike Brennan twenty years ago. So, it
follows that Mike Brennan is most familiar with the mawk code, and finds
it easier to hack on mawk than some other awk.

[Ed Morton]

No, I saw it. From a user perspective, obviously I'd find it far more useful to
have 1 awk that does w, x, y, and z than one that does w and x, one that does w
and y and none that do z so I'm just interested in why today we have multiple
awks with overlapping non-POSIX enhancements being developed simultaneously.

Ed.

[Kenny McCormack]

In article <nnjm9q$g1f$1...@dont-email.me>,

Ed Morton <morto...@gmail.com> wrote:
>On 7/30/2016 1:37 PM, mawk...@gmail.com wrote:
>> https://drive.google.com/open?id=0B_Q-mbHiy9g-Ry1WY1BjVUtJX3M
>>
>> is a google drive folder holding
>> about-mawk.pdf
>> mawk-1.9.9.2.tar.gz
>>
>> This a beta release, which I believe is stable, and will soon be mawk 2.0
>
>Why? If you have some free time to do awk tool development, why not contribute
>that time to gawk development instead of some other variant?

You really are nuts, aren't you?

P.S. Dear Linux developers: Why do you waste your time (and ours) working
on this silly "Linux" thing, when you could be spending your time working
on and contributing to the one true OS, Windows?

P.P.S. The Gawk code is pretty "opaque". I *have* worked on it some, and
I've found it daunting. Mawk's source code is much clearer by comparison.
The fact that Gawk has had a lot of cooks stirring the broth over the
years, shows.

--
I am not a troll.
Rick C. Hodgin
I am not a crook.
Rick M. Nixon

[Marc de Bourget]

The "why" question for MAWK can be simply answered:
It is ligthning-fast.
It is the fastest AWK implementation, I think it is
even as fast as a C program.

If I can use simple scripts I always use MAWK.
However, it lacks a lot of functions GAWK and TAWK
have (but maybe that is a reason why it is so fast).

I use this Windows built:
http://www.klabaster.com/freeware.htm

@Kenny: If you download the file, you can read the
second page afterwards.

Great to see there will be a new MAWK version soon!

[Kaz Kylheku]

These simply complement the multiple shell implementation, vi clones and
forks of Emacs.

Which telnet do you want, the one from inet-utils or netkit?

[Ed Morton]

On 7/31/2016 3:27 AM, Marc de Bourget wrote:
> The "why" question for MAWK can be simply answered:
> It is ligthning-fast.

Couldn't the performance of gawk be improved if someone dedicated and
experienced in writing high performance awk execution code, e.g. whoever wrote
that code for mawk, worked on that rather than introducing gensub() etc. to mawk?

> It is the fastest AWK implementation, I think it is
> even as fast as a C program.

Many awk programs execute as fast as C programs since the code awk provides for
us to do read/split/search/replace/etc. is non-trivial to write efficiently in C.

Ed.

[Janis Papanagnou]

On 31.07.2016 19:49, Ed Morton wrote:
>
> Many awk programs execute as fast as C programs since the code awk provides
> for us to do read/split/search/replace/etc. is non-trivial to write
> efficiently in C.

If you want _exactly_ what happens to be available in awk then this might be
true. Or not. It depends what you want, and how that function is implemented
in both cases. Generally you will have better performance with hard-tailored
implementations. The more relevant point is that you spare time and effort
if you just use what's there instead of re-implementing some function. But
beyond that don't expect to have better performance. The point in using awk
is not performance, it's the abstraction it provides to formulate efficient
solutions for certain (restricted) classes of problems with minimum effort.

Janis

>
> Ed.
>

[Marc de Bourget]

For me, speed is most important. I have to deal with very big datasets (most times at least 1 million records). Belong my experiences, MAWK is the fasted scripting language.

[Janis Papanagnou]

On 31.07.2016 22:39, Marc de Bourget wrote:
>> On 31.07.2016 19:49, Ed Morton wrote:
>>>

>>> Many awk programs execute as fast as C programs [...]

>
> For me, speed is most important. I have to deal with very big datasets
> (most times at least 1 million records). Belong my experiences, MAWK is the
> fasted scripting language.

Sure. (My response was only addressing the above quoted statement.)

BTW; I am using gawk in similar conditions (1+ GB data, 3+ million records),
and it's pretty fast. (I have no comparison with MAWK, though.)

Janis

[Kenny McCormack]

In article <nnlp45$8rn$1...@news-1.m-online.net>,
Janis Papanagnou <janis_pa...@hotmail.com> wrote:
...

>BTW; I am using gawk in similar conditions (1+ GB data, 3+ million records),
>and it's pretty fast. (I have no comparison with MAWK, though.)

It would be good for someone to work up a good benchmark, and then let us
test it on the various implementations.

Mawk certainly has the reputation of being the fastest free implementation.
But it should be noted that TAWK is very fast (in addition to being very
full-featured - demonstrating that you shouldn't have to choose between
these two desirable goals)). I'd actually be a little surprised to find
that MAWK is in any way significantly faster.

--
Marshall: 10/22/51
Jessica: 4/4/79

[Marc de Bourget]

Le lundi 1 août 2016 02:05:48 UTC+2, Kenny McCormack a écrit :
> In article <>,

> Janis Papanagnou <> wrote:
> ...
> >BTW; I am using gawk in similar conditions (1+ GB data, 3+ million records),
> >and it's pretty fast. (I have no comparison with MAWK, though.)
>
> It would be good for someone to work up a good benchmark, and then let us
> test it on the various implementations.
>
> Mawk certainly has the reputation of being the fastest free implementation.
> But it should be noted that TAWK is very fast (in addition to being very
> full-featured - demonstrating that you shouldn't have to choose between
> these two desirable goals)). I'd actually be a little surprised to find
> that MAWK is in any way significantly faster.
>
> --
> Marshall: 10/22/51
> Jessica: 4/4/79

There is a comparison which howevewer does not include TAWK:
https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/

[Janis Papanagnou]

On 01.08.2016 09:42, Marc de Bourget wrote:
>
> There is a comparison which howevewer does not include TAWK:
> https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/

Given that the 'Type' column says 'Interpreted' I'm not quite sure whether
the tests already consider newer gawk versions with intermediate byte code;
the test - assuming it's from 2009 - seems a bit outdated.

I'm also quite astonished that nawk is faster than gawk; I racall an (even
older, from ~1999) benchmark of sereral awk's where that had not been the
case; nawk had been the slowest candidate.

A speed factor of ~8 between mawk and gawk is immense. It would be more
comforting, though, if this comment would not be there:
"I have since found large datasets where mawk is buggy and gives the
wrong result. nawk seems safe."


Janis

[Kenny McCormack]

In article <nnnkgi$qsj$1...@news-1.m-online.net>,

Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>On 01.08.2016 09:42, Marc de Bourget wrote:
>>
>> There is a comparison which howevewer does not include TAWK:

Which, in principle, makes it useless for my purposes, since the whole
point was to compare TAWK to the others.

>I'm also quite astonished that nawk is faster than gawk; I racall an (even
>older, from ~1999) benchmark of sereral awk's where that had not been the
>case; nawk had been the slowest candidate.

Gawk has matured quite a bit between versions 3 & 4. I think it used to
have a (deserved) rap for being slow.

>A speed factor of ~8 between mawk and gawk is immense. It would be more
>comforting, though, if this comment would not be there:
> "I have since found large datasets where mawk is buggy and gives the
> wrong result. nawk seems safe."

Yeah, that's a problem...

Anyway, I wasn't really looking for a canned, off-the-shelf comparison done
by other people (under unknown conditions, with unknown versions).

I was looking for somethng we develop here (in this newsgroup) and that we
can run the tests ourselves on the various versions and discuss our results
here.

--
Amazingly, it's beginning to look like W is the smart one.

But Jeb will still be the nominee (seriously, who else can/could it be?)

[Kenny McCormack]

In article <nnnq2q$htj$1...@news.xmission.com>,
Kenny McCormack <gaz...@shell.xmission.com> wrote:
...

>But Jeb will still be the nominee (seriously, who else can/could it be?)

Guess I was wrong about that. The GOP went and did something even stupider
than nominating Jeb.

--
I'm building a wall.

[mawk...@gmail.com]

Thank you.

[mawk...@gmail.com]

I've now posted this to https://github.com/mikebrennan000/mawk-2, which may be
easier for some to access than a google drive.

$ git clone https://github.com/mikebrennan000/mawk-2.git

will pull both files. The tarball is now mawk-1.9.9.4.tar.gz, but the changes
are trivial and produce the same executable.

[Marc de Bourget]

As for the speed testing:
I have found another comparison which seems quite well suited:
http://awka.sourceforge.net/compare.html
or even better, because it contains newer source files:
https://github.com/chadbrewbaker/awka/tree/master/benchmark

cast.awk (lots of string to number & back casts in a loop)
array.awk (tests arrays with string elements)
array2.awk (tests arrays with integer elements)
array3.awk (speed accessing multi-dimension arrays)
io.awk (writes out a large file, reads it in & splits $0)
split.awk (how fast is the split functionality?)
function.awk (many repeated function calls passing arguments)
expr.awk (speed evaluating a reasonably complex expression)
parsecsv.awk (a benchmarking example from comp.lang.awk)
loop.awk (heavily nested for loops and if statements)
recurse.awk (deep recursive function test)
regexp.awk (test regular expression speed)

I have used these Compiler versions for Microsoft Windows:
GAWK 4.1.3:
https://sourceforge.net/projects/ezwinports/files/gawk-4.1.3-w32-bin.zip/download

MAWK 1.3.3:
http://www.klabaster.com/freeware.htm#dl

TAWK 5.0c
http://www.tasoft.com/tawk.html

For the timer I used a Windows console program named timer.exe:
http://www.gammadyne.com/cmdline.htm#timer
e.g. 0:00:04.18 means 4 seconds 18 milliseconds

Here are my results:
*** array.awk ***
1st Winnner: MAWK time for array.awk:
0:00:04.18
GAWK time for array.awk:
0:00:06.40
2nd Winnner: TAWK time for array.awk:
0:00:02.34

*** array2.awk ***
1st Winnner: MAWK time for array2.awk:
0:00:00.68
2nd Winnner: GAWK time for array2.awk:
0:00:01.01
TAWK time for array2.awk:
0:00:01.64

*** array3.awk ***
2nd Winnner: MAWK time for array3.awk:
0:00:04.21
1st Winnner: GAWK time for array3.awk:
0:00:02.42
TAWK time for array3.awk:
0:00:05.76

*** cast.awk ***
1st Winnner: MAWK time for cast.awk:
0:00:05.37
2nd Winnner: GAWK time for cast.awk:
0:00:09.37
TAWK time for cast.awk:
0:00:10.40

*** expr.awk ***
1st Winnner: MAWK time for expr.awk:
0:00:02.07
GAWK time for expr.awk:
0:00:13.09
2nd Winnner: TAWK time for expr.awk:
0:00:06.75

*** function.awk ***
MAWK time for function.awk:
0:00:02.57
1st Winnner: GAWK time for function.awk:
0:00:01.93
2nd Winnner: TAWK time for function.awk:
0:00:02.51

*** io.awk ***
1st Winnner: MAWK time for io.awk:
0:00:02.56
2nd Winnner: GAWK time for io.awk:
0:00:03.78
TAWK time for io.awk:
0:00:08.84

*** loop.awk ***
1st Winnner: MAWK time for loop.awk:
0:00:02.82
GAWK time for loop.awk:
0:00:07.64
2nd Winnner: TAWK time for loop.awk:
0:00:03.21

*** parsecsv.awk ***
1st Winnner: MAWK time for parsecsv.awk:
0:00:01.73
2nd Winnner: GAWK time for parsecsv.awk:
0:00:04.73
TAWK time for parsecsv.awk:
0:00:05.14

*** recurse.awk ***
1st Winnner: MAWK time for recurse.awk:
0:00:00.15
GAWK time for recurse.awk:
0:00:00.35
2nd Winnner: TAWK time for recurse.awk:
0:00:00.25

*** regexp.awk ***
1st Winnner: MAWK time for regexp.awk:
0:00:00.48
GAWK time for regexp.awk:
0:00:27.59
2nd Winnner: TAWK time for regexp.awk:
0:00:01.26

*** split.awk ***
1st Winnner: MAWK time for split.awk:
0:00:02.00
2nd Winnner: GAWK time for split.awk:
0:00:04.84
TAWK time for split.awk:
0:00:05.23

The result as a summary:
1st Winnner: MAWK (array.awk)
1st Winnner: MAWK (array2.awk)
1st Winnner: GAWK (array3.awk)
1st Winnner: MAWK (cast.awk)
1st Winnner: MAWK (expr.awk)
1st Winnner: GAWK (function.awk)
1st Winnner: MAWK (io.awk)
1st Winnner: MAWK (loop.awk)
1st Winnner: MAWK (parsecsv.awk)
1st Winnner: MAWK (recurse.awk)
1st Winnner: MAWK (regexp.awk)
1st Winnner: MAWK (split.awk)

2nd Winnner: TAWK (array.awk)
2nd Winnner: GAWK (array2.awk)
2nd Winnner: MAWK (array3.awk)
2nd Winnner: GAWK (cast.awk)
2nd Winnner: TAWK (expr.awk)
2nd Winnner: TAWK (function.awk)
2nd Winnner: GAWK (io.awk)
2nd Winnner: TAWK (loop.awk)
2nd Winnner: GAWK (parsecsv.awk)
2nd Winnner: TAWK (recurse.awk)
2nd Winnner: TAWK (regexp.awk)
2nd Winnner: GAWK (split.awk)

So MAWK wins in 10/12 tests as "1st Winner".

Some more comments about the test:
1. cast.awk doesn't work for TAWK.
https://github.com/chadbrewbaker/awka/blob/master/benchmark/cast.awk
I've had to change:
sprintf(xx, "v1->ival = %d, v2->ptr = %s\n",v1,v2)
to:
sprintf("v1->ival = %d, v2->ptr = %s\n",v1,v2)
(not sure what 'xx' is meant to be).

2. Some of the AWK scripts contain Unix systems calls
like system("rm -f io.txt") which I haven't changed because
they don't cause errors although they don't work for Windows.

3. Although the results for GAWK generally look good,
there is a case where GAWK is extremely slow compared
to MAWK and TAWK:
*** regexp.awk ***
1st Winnner: MAWK time for regexp.awk:
0:00:00.48
GAWK time for regexp.awk:
0:00:27.59
2nd Winnner: TAWK time for regexp.awk:
0:00:01.26
https://github.com/chadbrewbaker/awka/blob/master/benchmark/regexp.awk
Either of these three cases seems to be very slow:
# Manually doing a gsub
while (match(s1, j))
s1 = substr(s1, 1, RSTART-1) Switch[j] substr(s1, RSTART+RLENGTH)

# Use gsub
gsub(j, Switch[j], s2)

# gsub, and prevent RE recompile
gsub(Switch_R[j], Switch[j], s3)

GAWK developpers might want to scrutinize this test file.

[Janis Papanagnou]

On 13.08.2016 12:28, Marc de Bourget wrote:
> As for the speed testing:
> I have found another comparison which seems quite well suited:
> http://awka.sourceforge.net/compare.html

I seem to recall that those tests are very old, from the late 1990's.
(At that time the author of awka wanted to show the speed advantages
of a compiled awk variant. Other awks, notably gawk, have developed
since.)

> or even better, because it contains newer source files:
> https://github.com/chadbrewbaker/awka/tree/master/benchmark

These are certainly only the check-in dates, not the date of the
creation of the files. And, in fact, if I inspect the README file
I see a date "17-8-2000".

Janis

> [...]

[Marc de Bourget]

Thank you for your input.
However, I don't think AWK have changed that much since the 90's and
if we want to compare GAWK, MAWK and TAWK we have to use code which
work for all of these AWK variants (we can't use gensub etc).
So, I do think this comparison is still interesting for new results.

[Ed Morton]

See this item:

The gawk internals were rewritten, bringing the dgawk debugger and possibly
improved performance (see Debugger).

in the gawk Feature History
(https://www.gnu.org/software/gawk/manual/gawk.html#Feature-History) under
Version 4.0. Note in particular "The gawk internals were rewritten" which imho
means gawk has changed much and so any performance testing before that point
shouldn't be assumed to apply afterwards.

Ed.

[Marc de Bourget]

Yes, therefore I used GAWK 4.1.3 which performs well except for regexp.awk:
https://github.com/chadbrewbaker/awka/blob/master/benchmark/regexp.awk
For my results see above. I don't know why this script is named regexp.awk.
I can't see any regular expression, but it uses gsub and match.
It seems there are speed problems with gsub and match with GAWK.

[Aharon Robbins]

In article <8533c133-a5b3-4625...@googlegroups.com>,

Marc de Bourget <marcde...@gmail.com> wrote:
>Yes, therefore I used GAWK 4.1.3 which performs well except for regexp.awk:
>https://github.com/chadbrewbaker/awka/blob/master/benchmark/regexp.awk
>For my results see above. I don't know why this script is named regexp.awk.
>I can't see any regular expression, but it uses gsub and match.
>It seems there are speed problems with gsub and match with GAWK.

It ends up exercising gawk's compilation of regular expressions more
than how fast they are executed.

I did some profiling. As a result I was able to make a simple change
that yielded a 14% improvement on this program and will push it shortly.

Thanks,

Arnold
--
Aharon (Arnold) Robbins arnold AT skeeve DOT com

[Marc de Bourget]

Le dimanche 14 août 2016 16:20:26 UTC+2, Aharon Robbins a écrit :
> It ends up exercising gawk's compilation of regular expressions more
> than how fast they are executed.
>
> I did some profiling. As a result I was able to make a simple change
> that yielded a 14% improvement on this program and will push it shortly.
>
> Thanks,
>
> Arnold
> --
> Aharon (Arnold) Robbins arnold AT skeeve DOT com

Hi Arnold, great, thank you!

[Marc de Bourget]

With my tests, GAWK 4.1.4 has a 34% improvement with regexp.awk. Good news!

BTW, I've noticed a new MAWK 2 beta version 1.9.6.6 has been released:
https://github.com/mikebrennan000/mawk-2
Has someone compiled and tested it? There is no MinGW support yet so I can't compile it for Windows.

[Marc de Bourget]

Is or will MAWK be part of Cygwin? That would be great. ->
If so, we could use Cygnal to create a native Windows port.
That would be great. If so, we could use Cygnal to create a "native" Windows port.

[Aharon Robbins]

In article <1e9e2991-54d3-4e43...@googlegroups.com>,

Marc de Bourget <marcde...@gmail.com> wrote:

>Is or will MAWK be part of Cygwin?

It compiles out of the box on Cygwin. The usual untar + configure + make
incantation works. (I speak from personal experience.)

Cygwin is a volunteer-run project. Someone who's interested enough
could volunteer to maintain mawk for that project. It should be
pretty trivial.

[Marc de Bourget]

Thank you Arnold for this information.
If I have understood correctly, it may be possible to build a Cygwin version
and thus a Cygnal version?

[Aharon Robbins]

> If I have understood correctly, it may be possible to build a Cygwin version

No "may be" about it. I build under Cygwin all the time. You can start from
the tarball or use Git to check out the repo.

wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.gz
tar -xpzvf gawk-4.1.4.tar.gz
cd gawk-4.1.4
./configure && make -j && make check

It's that easy.

[Aharon Robbins]

In article <nq76jf$3gl$1...@dont-email.me>,

Aharon Robbins <arn...@skeeve.com> wrote:
>> If I have understood correctly, it may be possible to build a Cygwin version
>
>No "may be" about it. I build under Cygwin all the time. You can start from
>the tarball or use Git to check out the repo.
>
> wget http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.gz
> tar -xpzvf gawk-4.1.4.tar.gz
> cd gawk-4.1.4
> ./configure && make -j && make check
>
>It's that easy.

Hmm... Were you talking about mawk? If so, the same recipe applies.
I did it just yesterday or today.

[Marc de Bourget]

Le mercredi 31 août 2016 20:15:07 UTC+2, Aharon Robbins a écrit :
>
> Hmm... Were you talking about mawk? If so, the same recipe applies.
> I did it just yesterday or today.
> --
> Aharon (Arnold) Robbins arnold AT skeeve DOT com

Hi Arnold, thank you for the information. Yes, this time I meant MAWK.
For a native Windows MAWK 2 version "Cygnal" sounds very interesting
because AFAIK there is no MinGW compatible source code available yet.

[Kaz Kylheku]

Try it. By the way, I noticed a strange issue with the "Cygnalized"
awk I rolled up. Namely, when I run it from a cmd.exe window, the
readline history recall and editing doesn't work properly. The display
gets messed up when I use up arrow, and the editing is garbled also.
Not quite sure what is wrong there. Maybe ncurses is picking up the
wrong emulation?

Cygwin (and thus Cygnal) provides complete VT100/ANSI emulation which
translates between the Win32 console API and escape sequences (in both
directions). And it provides the POSIX termios interface, mapping its
semantics to the console. I fixed a couple of bugs in the VT100
emulation in Cygnal. The "linenoise" library, which uses termios calls
and raw VT100 escapes, works quite perfectly. In theory, middleware like
readline and ncurses should Just Work.

[Marc de Bourget]

Hi Kaz, I have tested your gawk version with a bunch of script files
and haven't encountered any issues. Command history also works (WIN 10).