First public beta of upcoming gawk 4.0 release
Aharon Robbins
It is available from:
http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
This release represents over a year and half's very hard work by a
number of people, and introduces a significant number of important new
features, as well as some more minor improvements. The NEWS file is
appended below.
As far as I can tell, the documentation and code have both hit the
freeze point.
So, why do a beta release? So that you, yes you, the end user, can see
if anything I've done breaks gawk for you. Then you can TELL ME ABOUT
IT so that I can fix it for the final release.
Thanks,
Arnold Robbins
arn...@skeeve.com
---------------------------------------------
Copyright (C) 2010, 2011 Free Software Foundation, Inc.
Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.
Changes from 3.1.8 to 4.0.0
---------------------------
1. The special files /dev/pid, /dev/ppid, /dev/pgrpid and /dev/user are
now completely gone. Use PROCINFO instead.
2. The POSIX 2008 behavior for `sub' and `gsub' are now the default.
THIS CHANGES BEHAVIOR!!!!
3. The \s and \S escape sequences are now recognized in regular expressions.
4. The split() function accepts an optional fourth argument which is an array
to hold the values of the separators.
5. New -b / --characters-as-bytes option that means "hands off my data"; gawk
won't try to treat input as a multibyte string.
6. New --sandbox option; see the doc.
7. Indirect function calls are now available.
8. Interval expressions are now part of default regular expressions for
GNU Awk syntax.
9. --gen-po is now correctly named --gen-pot.
10. switch / case is now enabled by default. There's no longer a need
for a configure-time option.
11. Gawk now supports BEGINFILE and ENDFILE. See the doc for details.
12. Directories named on the command line now produce a warning, not
a fatal error, unless --posix or --traditional.
13. The new FPAT variable allows you to specify a regexp that matches
the fields, instead of matching the field separator. The new patsplit()
function gives the same capability for splitting.
14. All long options now have short options, for use in `#!' scripts.
15. Support for IPv6 added via /inet6/... special file. /inet4/... forces
IPv4 and /inet chooses the system default (probably IPv4).
16. Added a warning for /[:space:]/ that should be /[[:space:]]/.
17. Merged with John Haque's byte code internals. Adds dgawk debugger and
possibly improved performance.
18. `break' and `continue' are no longer valid outside a loop, even with
--traditional.
19. POSIX character classes work with --traditional (BWK awk supports them).
20. Nuked redundant --compat, --copyleft, and --usage long options.
21. Arrays of arrays added. See the doc.
22. Per the GNU Coding Standards, dynamic extensions must now define
a global symbol indicating that they are GPL-compatible. See
the documentation and example extensions.
THIS CHANGES BEHAVIOR!!!!
23. In POSIX mode, string comparisons use strcoll/wcscoll.
THIS CHANGES BEHAVIOR!!!!
24. The option for raw sockets was removed, since it was never implemented.
25. If not in POSIX mode, gawk turns ranges of the form [d-h] into
[defgh] before compiling a regexp. Maybe this will stop all the
questions about [a-z] matching uppercase letters.
THIS CHANGES BEHAVIOR!!!!
26. PROCINFO["strftime"] now holds the default format for strftime().
27. Updated to latest infrastructure: Autoconf 2.68, Automake 1.11.1,
Gettext 0.18.1, Bison 2.5.
28. Many code cleanups. Removed code for many old, unsupported systems:
- Atari
- Amiga
- BeOS
- Cray
- MIPS RiscOS
- MS-DOS with Microsoft Compiler
- MS-Windows with Microsoft Compiler
- NeXT
- SunOS 3.x, Sun 386 (Road Runner)
- Tandem (non-POSIX)
- Prestandard VAX C compiler for VAX/VMS
- Probably others that I've forgotten
29. If PROCINFO["sorted_in"] exists, for(iggy in foo) loops sort the
indices before looping over them. The value of this element
provides control over how the indices are sorted before the loop
traversal starts. See the manual.
30. A new isarray() function exists to distinguish if an item is an array
or not, to make it possible to traverse multidimensional arrays.
31. asort() and asorti() take a third argument specifying how to sort.
See the doc.
--
Aharon (Arnold) Robbins arnold AT skeeve DOT com
P.O. Box 354 Home Phone: +972 8 979-0381
Nof Ayalon Cell Phone: +972 50 729-7545
D.N. Shimshon 99785 ISRAEL
Manuel Collado
> This note is to announce the first BETA release of GNU Awk 4.0.
>
> It is available from:
>
> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
>
> This release represents over a year and half's very hard work by a
> number of people, and introduces a significant number of important new
> features, as well as some more minor improvements. The NEWS file is
> appended below.
>
> As far as I can tell, the documentation and code have both hit the
> freeze point.
>
> So, why do a beta release? So that you, yes you, the end user, can see
> if anything I've done breaks gawk for you. Then you can TELL ME ABOUT
> IT so that I can fix it for the final release.
Just to point out that the 'texinfo.tex' macros (part of the texinfo
utility) are included in the /doc subdir of the tarball. Is it really
appropriate to distribute a copy here? (I see they have been also
included in previous gawk distributions).
Regards.
--
Manuel Collado - http://lml.ls.fi.upm.es/~mcollado
Kenny McCormack
Aharon Robbins <arn...@skeeve.com> wrote:
>This note is to announce the first BETA release of GNU Awk 4.0.
>
>It is available from:
>
> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
Is that really the filename (for GAWK 4) ?
--
Windows 95 n. (Win-doze): A 32 bit extension to a 16 bit user interface for
an 8 bit operating system based on a 4 bit architecture from a 2 bit company
that can't stand 1 bit of competition.
Modern day upgrade --> Windows XP Professional x64: Windows is now a 64 bit
tweak of a 32 bit extension to a 16 bit user interface for an 8 bit
operating system based on a 4 bit architecture from a 2 bit company that
can't stand 1 bit of competition.
Hermann Peifer
>
> Is that really the filename (for GAWK 4) ?
>
No. It is the file name of the third beta test tar ball for 4.0.
From the ChangeLog:
* Release 3.1.84: Third beta test tar ball for 4.0.
From doc/gawk.texi:
(Note, however, that patch levels greater than or equal to 70 denote
``beta'' or nonproduction software;
Hermann
Janis Papanagnou
> This note is to announce the first BETA release of GNU Awk 4.0.
>
> It is available from:
>
> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
Thank you!
Janis
Kenny McCormack
OK - it looked more like a typo (or old cut-and-paste) to me, but if that's
the way they are doing things, OK by me.
--
They say compassion is a virtue, but I don't have the time!
- David Byrne -
Janis Papanagnou
> This note is to announce the first BETA release of GNU Awk 4.0.
>
> It is available from:
>
> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
Just inspected the man page and stumbled across these...
> HISTORICAL FEATURES
> There are two features of historical AWK implementations that gawk
> supports. First, [...]
And I was wondering what the second point is; it seems to be missing?
> GNU EXTENSIONS
I've just briefly read the chapter about the extension but noticed that
some extension (e.g. BEGINFILE, etc.) doesn't seem to be listed.
Thanks again.
Janis
Aharon Robbins
Hi. This is partly historical, but still practical. It dates from the time
when most people were usng gawk on Unix and may have had TeX but not
had the Texinfo macros.
It is still practical for people like that (using Unix, not a GNU/Linux
dist) but also for other operating systems such as MS-Windows or Mac OS X.
Thanks,
Arnold
Aharon Robbins
In article <irq864$vke$1...@news.m-online.net>,
Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>On 27.05.2011 09:57, Aharon Robbins wrote:
>> This note is to announce the first BETA release of GNU Awk 4.0.
>>
>> It is available from:
>>
>> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
>
>Just inspected the man page and stumbled across these...
>
>> HISTORICAL FEATURES
>> There are two features of historical AWK implementations that gawk
>> supports. First, [...]
>
>And I was wondering what the second point is; it seems to be missing?
An editing error. The 2nd point went away, since even BWK's awk
no longer supports the "feature", which was break and continue working
outside of a loop like "next".
Thanks for the catch!
>> GNU EXTENSIONS
>
>I've just briefly read the chapter about the extension but noticed that
>some extension (e.g. BEGINFILE, etc.) doesn't seem to be listed.
You're right!
Both things are fixed and will show up in the git repo shortly.
Thank you!
Arnold
Bill Wang
Regards,
Bill
Aharon Robbins
Bill Wang <bw5...@gmail.com> wrote:
>Good news. How about the speed, will the new version run faster?
I haven't done any major tests. In general it seems to run at least as
fast, and sometimes faster, than the old version. I've yet to encounter
a case where it runs slower.
Anyone who wants to work on testing this and let me know the results,
I'd greatly appreciate it; I don't have the cycles.
Thanks,
Arnold
John DuBois
Aharon Robbins <arn...@skeeve.com> wrote:
>In article <d260ac33-cf97-4d45...@j13g2000pro.googlegroups.com>,
>Bill Wang <bw5...@gmail.com> wrote:
>>Good news. How about the speed, will the new version run faster?
>
>I haven't done any major tests. In general it seems to run at least as
>fast, and sometimes faster, than the old version. I've yet to encounter
>a case where it runs slower.
>
>Anyone who wants to work on testing this and let me know the results,
>I'd greatly appreciate it; I don't have the cycles.
I have a gawk test suite that runs a wide variety of utilities, mainly to catch
differences in output between new and previous versions of gawk. It also
records CPU time for each run, and runs the program before the comparison run
in an attempt at caching data to reduce perturbing effects. However, this is
run on a moderately active system - that being where the data that these
utilities use resides - which will have some effect on the CPU time
measurement. In any case, here are some results.
3.1.84 3.1.7
3.70s 3.93s imagelog
0.03s 0.05s recipeToHtml
2.08s 2.14s websearches
2.19s 2.35s referrers
18.85s 18.51s flog
0.16s 0.18s imgbrowse.cgi
2.67s 2.77s httplog
0.05s 0.05s lastlogin
0.13s 0.15s esearch
0.40s 0.44s add
0.15s 0.16s from
0.53s 0.60s rainfall
0.04s 0.05s aj
0.05s 0.06s satvisib
0.11s 0.12s divisions
0.11s 0.11s mmdfq
0.29s 0.30s lowtide
0.11s 0.13s purity-test
0.03s 0.04s quota
0.02s 0.04s awkmail
0.03s 0.02s eventrem
0.03s 0.04s awg
0.09s 0.08s proctree
0.04s 0.05s mlast
0.04s 0.03s aw
0.11s 0.13s mkptestdb
0.11s 0.12s treset
0.11s 0.12s fsu
0.11s 0.12s vu
10.42s 10.26s jdu
0.04s 0.03s name
0.03s 0.04s mailalert
0.02s 0.02s activity
0.75s 0.73s htmluser
0.02s 0.02s uucon
0.02s 0.03s exp
0.02s 0.02s modemuse
0.02s 0.02s tbackFrameInfo
0.02s 0.02s canonlog
15.37s 15.71s shist
0.02s 0.02s libupdate
0.49s 0.51s ucomm
0.04s 0.04s udaemon
0.02s 0.03s last-syslog
150.44s 150.29s vtree
0.13s 0.12s pss
0.01s 0.03s e2rem
22.47s 40.98s peakprocs
0.05s 0.05s mbill
0.04s 0.04s checks
0.01s 0.02s nacllog
0.08s 0.09s bots
0.04s 0.04s fix_mailx
0.03s 0.03s altchars
0.10s 0.11s htmlweb
0.02s 0.02s modems
0.06s 0.07s ftpi2html
0.10s 0.10s nethog
0.02s 0.02s galfile
0.02s 0.03s upstat
0.02s 0.02s nvdtool
0.02s 0.02s fb
0.01s 0.02s tcpdumpconn
3.38s 3.27s numexec
0.01s 0.02s getdtv
0.01s 0.02s upsince
0.02s 0.02s hamtest
0.01s 0.02s fov
0.06s 0.06s exp_notify
0.02s 0.02s frac
22.70s 23.89s dparamlog
0.01s 0.02s ptyclean
0.02s 0.02s pto
0.01s 0.02s unixtime
0.01s 0.02s phonepad
0.01s 0.02s rotrate
0.02s 0.02s datediff
0.01s 0.02s extminor
0.01s 0.02s l-len
0.05s 0.05s mungeforward
0.02s 0.02s url2path
0.02s 0.02s hdminor
0.03s 0.03s phonegen
The interesting peakprocs result above is reproducible.
It uses a priority queue.
John
--
John DuBois spc...@armory.com KC6QKZ/AE http://www.armory.com/~spcecdt/
mb
> This note is to announce the first BETA release of GNU Awk 4.0.
>
> It is available from:
>
> http://www.skeeve.com/gawk/gawk-3.1.84.tar.gz
>
> This release represents over a year and half's very hard work by a
> number of people, and introduces a significant number of important new
> features, as well as some more minor improvements. The NEWS file is
> appended below.
>
> As far as I can tell, the documentation and code have both hit the
> freeze point.
>
> So, why do a beta release? So that you, yes you, the end user, can see
> if anything I've done breaks gawk for you. Then you can TELL ME ABOUT
> IT so that I can fix it for the final release.
>
> Thanks,
>
> Arnold Robbins
Thank you for the efforts and hard work.
mss
>> Arnold Robbins
>> [...]
>
> Thank you for the efforts and hard work.
Yes, very much so.
--
later on,
Mike
Aharon Robbins
John DuBois <spc...@armory.com> wrote:
>In article <irvg1v$shl$1...@tornado.tornevall.net>,
>Aharon Robbins <arn...@skeeve.com> wrote:
>>Anyone who wants to work on testing this and let me know the results,
>>I'd greatly appreciate it; I don't have the cycles.
>
>I have a gawk test suite that runs a wide variety of utilities, mainly to catch
>differences in output between new and previous versions of gawk. It also
>records CPU time for each run, and runs the program before the comparison run
>in an attempt at caching data to reduce perturbing effects. However, this is
>run on a moderately active system - that being where the data that these
>utilities use resides - which will have some effect on the CPU time
>measurement. In any case, here are some results.
The results are interesting; they show that the development gawk is
about the same or faster than 3.1.8, which is pretty much what I thought.
What's important is that performance hasn't gotten worse, while there
are significant new features.
It'd be interesting to understand why the peakprocs example was so much
faster; I'm guessing there's a lot of recursion in it, but that's just
a wild guess.
Thanks for sharing the info!
Unrelated - thanks also to everyone posting "thank you" notes; it's good
to be appreciated. :-)
John DuBois
Aharon Robbins <arn...@skeeve.com> wrote:
>It'd be interesting to understand why the peakprocs example was so much
>faster; I'm guessing there's a lot of recursion in it, but that's just
>a wild guess.
The program is ftp://ftp.armory.com/pub/admin/peakprocs
It processes OSR5 process accounting records, but pseudo-data can be generated
with e.g.:
gawk 'BEGIN {
for (i = 10000; i > 0; i--)
printf "x y z %s %s\n", strftime("%FT%T", i+100000),
strftime("%FT%T", i+100100) } ' |
gawk -f peakprocs -i
The profiler shows that it's the loop at lines 261-270 that is executed most
frequently, and indeed commenting out that loop entirely eliminates the
execution-time difference.
Manuel Collado
> So, why do a beta release? So that you, yes you, the end user, can see
> if anything I've done breaks gawk for you. Then you can TELL ME ABOUT
> IT so that I can fix it for the final release.
Just a marginal typo in the docs. A couple of references to XMLgawk have
a wrong URL:
http://xgawk.sourceforge.net
should be
http://xmlgawk.sourceforge.net
They are located at:
gawk.texi:11406:
...used only by the @uref{http://xgawk.sourceforge.net, XMLgawk ...
gawk.texi:28517:
@uref{http://xgawk.sourceforge.net, the XMLgawk project web site}.
Regards,
Aharon Robbins
Fixed, thanks.
Please cc me these kinds of things in email to make sure I see them.
Thanks,
Arnold
Radoulov, Dimitre
> This note is to announce the first BETA release of GNU Awk 4.0.
[...]
>
> Changes from 3.1.8 to 4.0.0
> ---------------------------
[...]
> 13. The new FPAT variable allows you to specify a regexp that matches
> the fields, instead of matching the field separator. The new patsplit()
> function gives the same capability for splitting.
[...]
GNU Awk 3.1.84 on Ubuntu 11.04 32bit.
Is this a bug:
% cat infile
Robbins,Arnold,"1234 A Pretty Street, NE",MyTown,MyState,12345-6789,USA
% awk '{ print $1 }' FPAT='([^,]+)|("[^"]+")' infile
Robbins
% awk '{ print $3 }' FPAT='([^,]+)|("[^"]+")' infile
"1234 A Pretty Street, NE"
% awk '{ print $1, $3 }' FPAT='([^,]+)|("[^"]+")' infile
Robbins
In the output of the last command the content of the third field is missing.
Consider the following also:
% awk 'BEGIN {
FPAT = "([^,]+)|(\"[^\"]+\")"
}
{
for (i = 1; i <= NF; i++)
print i, $i
}' infile
1 Robbins
2 Arnold
3 "1234 A Pretty Street, NE"
4 MyTown
5 MyState
6 12345-6789
7 USA
% awk 'BEGIN {
FPAT = "([^,]+)|(\"[^\"]+\")"
}
{
print $1
for (i = 1; i <= NF; i++)
print i, $i
}' infile
Robbins
1 Robbins
Regards
Dimitre
Aharon Robbins
I'll see what I can do.
Arnold
In article <isg4ki$mcp$1...@dont-email.me>,
Radoulov, Dimitre
> Looks like a bug. Good catch. FPAT is more painful than I thought.
>
> I'll see what I can do.
Thank you!
Best regards
Dimitre
Aharon Robbins
Here's the fix. Turned out to not be too terrible.
Thanks,
Arnold
-------------------------
diff --git a/field.c b/field.c
index 151fea6..0b9c100 100644
--- a/field.c
+++ b/field.c
@@ -892,7 +892,7 @@ get_field(long requested, Func_ptr *assign)
char *rec_end = fields_arr[0]->stptr + fields_arr[0]->stlen;
if ( parse_extent > rec_end
- || (parse_extent > rec_start && parse_extent < rec_end))
+ || (parse_extent > rec_start && parse_extent < rec_end && requested == UNLIMITED-1))
NF = parse_high_water;
else if (parse_extent == rec_start) /* could be no match for FPAT */
NF = 0;
@@ -1601,7 +1601,6 @@ fpat_parse_field(long up_to, /* parse only up to this field number */
start = scan;
while (research(rp, scan, 0, (end - scan), regex_flags) != -1
&& nf < up_to) {
- regex_flags |= RE_NO_BOL;
if (REEND(rp, scan) > RESTART(rp, scan)) { /* if (RLENGTH > 0) */
non_empty = TRUE;
Radoulov, Dimitre
> In article<isghlv$d3k$1...@dont-email.me>,
> Here's the fix. Turned out to not be too terrible.
[...]
It's working fine,
thank you again!
Best regards
Dimitre
Tim Menzies
t
Aharon Robbins
Tim Menzies <comp.l...@googlegroups.com> wrote:
>thanks++ for "isarray()". i was going to nag you that but i see u beat me to it.
>
>t
There was a lot of internal discussion about what kind of type-related
function and/or how many to provide - we finally went for the minimalist
solution.
Arnold
Edward Rosten
NEXTFILE, multidimensional arrays, useful character classes.
I think I've found a bug in the sandbox mode:
./gawk --sandbox 'BEGIN{ARGV[1]="/etc/passwd"}1' /dev/null
shows the contents of /etc/passwd.
Also, does anyone think it is worth making WHINY_USERS change the
default setting of PROCINFO["sorted_in"] and print a warning to
prevent silent behavioral changes in old shell scripts? Also, the
removal of the feature seems to be undocumented.
-Ed
Aharon Robbins
In article <563f66f4-fd04-43c1...@y7g2000prk.googlegroups.com>,
Edward Rosten <edward...@gmail.com> wrote:
>Thanks for the excellent work. I'm very much looking forward to using
>NEXTFILE,
You mean BEGINFILE and ENDFILE, I assume.
>I think I've found a bug in the sandbox mode:
>
>./gawk --sandbox 'BEGIN{ARGV[1]="/etc/passwd"}1' /dev/null
>
>shows the contents of /etc/passwd.
Why is this a bug? I bet that cat /etc/passwd shows the contents
of the file as well. Sandbox mode prevents *writing* files, not reading them.
>Also, does anyone think it is worth making WHINY_USERS change the
>default setting of PROCINFO["sorted_in"] and print a warning to
>prevent silent behavioral changes in old shell scripts?
I certainly don't, and as I have a strong opinion in this matter, I doubt
that discussing it further will change my mind. :-)
>Also, the removal of the feature seems to be undocumented.
The feature was never documented to start with. If someone relied on an
undocumented feature, that's their problem. Sorry.
Thanks,
Aharon Robbins
Aharon Robbins <arn...@skeeve.com> wrote:
>Hi.
>
>In article <563f66f4-fd04-43c1...@y7g2000prk.googlegroups.com>,
>Edward Rosten <edward...@gmail.com> wrote:
>>Thanks for the excellent work. I'm very much looking forward to using
>>NEXTFILE,
>
>You mean BEGINFILE and ENDFILE, I assume.
Oops. I also meant to say "you're welcome". :-) It's nice to be appreciated.
Manuel Collado
> Hi.
>
> In article<563f66f4-fd04-43c1...@y7g2000prk.googlegroups.com>,
> Edward Rosten<edward...@gmail.com> wrote:
>> I think I've found a bug in the sandbox mode:
>>
>> ./gawk --sandbox 'BEGIN{ARGV[1]="/etc/passwd"}1' /dev/null
>>
>> shows the contents of /etc/passwd.
>
> Why is this a bug? I bet that cat /etc/passwd shows the contents
> of the file as well. Sandbox mode prevents *writing* files, not reading them.
Well, the info manual says:
`--sandbox'
Disable the `system()' function, input redirections with `getline',
output redirections with `print' and `printf', and dynamic
extensions. This is particularly useful when you want to run
`awk' scripts from questionable sources and need to make sure the
scripts can't access your system (other than the specified input
data file).
IMHO, disabling getline redirection only makes sense if you want to
prevent *reading* system files from dubious scripts, right?
So perhaps the reported behavior can be seen as a bug, because it is a
workaround for some of the intended --sandbox restrictions.
Kind regards,