Ed.

These ones are hard to read and save:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

These are not quite as hard to read or save, but still cause some problems:

Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

These ones are easy to read and save:

Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

The ones that are hardest to read and save appear to be produced by Windows
Live Mail.

Perhaps one could tell awk to delete such messages. Would it also be able to
convert "quoted printable" into something more readable?

--
Ineptocracy

I'm glad that not all awk users are as rude and unhelpful as you.

[follow ups set]

I've previously got some very usefull scripts from USEnet
collaborations, like:
  list all files in dir-tree $1
  which are less than are N-days old,
  and which contain String1,
  ...
  and which contain StringN

I use those scripts every day, and THIS one will be valuable too.
So, I've made a test-script, to help beta-test the versions.

The task is to delete repeated/redundant blocks of lines
[although as Ed Morton pointed out, awk is not limited to lines]
from text files.

Input files have the format of:  Ha|Hb|Hc|Hd|...Hn|
where H is the repeated/redundant block of text,
and a,b...n is the valuable text, to be kept,
and | represents a one-line-section-separator:
typically "<><><><>"

My test script which assembles a,d,..H blocks into the Infile,
by using the human-edited H block, gave the following results,
which should probably be ignored, for difficulty of understanding.

The test conclusions, so far, are that:
   if chars "(", "]", "[" are in the DeleteFile: H,
   this gives problems.

Are these special-chars for `bash` ?

Thanks,

== Chris Glur.

Copy existing files for: a, b, c, d, e
  use simple 1-char: H

-> ./BuildI == construct FileIn from parts:
 len-I = 713

-> cp H R

-> TstDG ==
len Infile = 713
len DeleteFile = 1
len ouTfile = 687
==> 713 - 687 == 26 <-- expect 713 - 4 == 699
===> perhaps extra 'H' files were found.
====> test with unusual type of 'H' file

-> echo qzxv >> H
-> ./BuildI == construct FileIn from parts:
len-I = 718
-> cp H R
-> TstDG ==
len Infile = 718
len DeleteFile = 2
len ouTfile = 710
==> 718 - 710 == 8 == 2*4 == OK

====> now use a big random: H
-> ./BuildI == construct FileIn from parts:
len-I = 1538
-> cp H R
-> ./TstDG ==
len Infile = 1538
len DeleteFile = 166
len ouTfile = 1538
==> suspect <special chars> in R

====> combine
-> ./BuildNtest ==
construct FileIn from parts:
len-I = 1538
len Infile = 1538
len DeleteFile = 166
len ouTfile = 1538
==> as expected/confirmed

==> keep problematic 'H' as Horg & edit out suspected line/s
==> Let 'H'contain NO-square-brackets.
-> ./BuildNtest ==
construct FileIn from parts:
len-I = 763
len Infile = 763
len DeleteFile = 11
len ouTfile = 708
==> 763 - 708 == 55 == 5*11
===> suspect that now-reduced: H appears 6-times in FileIn
==> Yes, with difficulty, an editor confirms: H appears 6-times

-> cp H Horg2
=> edit/select 'H' to contain line:"   [44]Gravatar"
-> ./BuildNtest ==
construct FileIn from parts:
len-I = 838
len Infile = 838
len DeleteFile = 26
len ouTfile = 838
=> confirm problem with chars: "[","]"
=?=> which one or both: == deleting BOTH still FAILS
=> iteratively delete-1st-line of H until notFAIL
--------------- one line deleted between tests ---------------
construct FileIn from parts:
len-I = 793
len Infile = 793
len DeleteFile = 17
len ouTfile = 793
bash-3.1# ./BuildNtest
construct FileIn from parts:
len-I = 783
len Infile = 783
len DeleteFile = 15
len ouTfile = 723
---------------------------------------------------------
=?=!=> the line that caused the FAIL:-
 Name (required)

==> adding an un-matched "(" 10 lines before end of 'H' causes:
-> ./BuildNtest ==
construct FileIn from parts:
len-I = 783
awk: DGscript:15: (FILENAME=I FNR=7) fatal: Unmatched ( or \(: /     * (/
len Infile = 783
len DeleteFile = 15
len ouTfile = 0

==> remove "(" & test "]"
-> ./BuildNtest ==
construct FileIn from parts:
len-I = 783
len Infile = 783
len DeleteFile = 15
len ouTfile = 783

=> replace with "[]"
->  DGscript ==
:15: (FILENAME=I FNR=7) fatal: Unmatched [ or [^: /     * []/
len Infile = 783
len DeleteFile = 15
len ouTfile = 0

  <http://tools.ietf.org/html/rfc2045>

Your mail user agent should be able to convert them to local format
while saving them.

Have a look at these:

  <http://www.convertstring.com/EncodeDecode/Base64Decode>
  <http://www.convertstring.com/EncodeDecode/QuotedPrintableDecode>

  <http://www.fourmilab.ch/webtools/base64/>

Anyway, assuming messages are in a digest, separated by lines containing
the string "-- End --" and none of them are multipart messages.

#v+
----script begins on next line
/^-- End --/ {
  if (!b64)
    print
  body = 0
  b64 = 0
  next

!body && /^$/ {
  for (n = 1; n <= hlines; n++)
    print header[n]
  hlines = 0
  body = 1  

/^[Cc][Oo][Nn][Tt][Ee][Nn][Tt]-[Tt][Rr][Aa][Nn][Ss][Ff][Ee][Rr]-[Ee][Nn][Cc ][Oo][Dd][Ii][Nn][Gg]: [Bb][Aa][Ss][Ee]64/ {
  b64 = 1
  hlines = 0
  next

#v+
----script begins on next line
BEGIN {
  hex["0"] = 0  ; hex["1"] = 1  ; hex["2"] = 2  ; hex["3"] = 3
  hex["4"] = 4  ; hex["5"] = 5  ; hex["6"] = 6  ; hex["7"] = 7
  hex["8"] = 8  ; hex["9"] = 9  ; hex["A"] = 10 ; hex["B"] = 11
  hex["C"] = 12 ; hex["D"] = 13 ; hex["E"] = 14 ; hex["F"] = 15
  for (n = 0 ; n <= 255; n++)
    ch[n] = sprintf("%c", n)

/^$/ { body = 1 }

!body && /^[Cc][Oo][Nn][Tt][Ee][Nn][Tt]-[Tt][Rr][Aa][Nn][Ss][Ff][Ee][Rr]-[Ee][Nn][Cc ][Oo][Dd][Ii][Nn][Gg]: [Qq][Uu][Oo][Tt][Ee][Dd]-[Pp][Rr][Ii][Nn][Tt][Aa][Bb][Ll][Ee]/ {
  qp = 1
  $NF = "8bit"

###     not yet complete nor compliant ;-)
###
###     this is a simple base64 enc/dec in awk
###     mostly made for fun but actually used in a few awk scripts I use
###     in some other tools I wrote for fine grain analysis of
###     texts, mainly emails, mainly spams to try and generate some
###     synthetic regexps (or ideas of) regarding false positives or reinforcements.
###     (and yes I know some tools exist in perl and I even use some of
###     them which is another reason why I also do it in awk ;-)
###     the wrapping is set at 72 like the 'mimencode' usage.
###     (so, to avoid wrapping do set ORS to nil).
###
Aargh(){
        r=$1
        shift
        printf "\n%s\nThats all, folks...\n\n" "${@}"
        exit $r

                ###     transform to BASE64 values  
                _ba1 = rshift(_by1, 2)
                _ba2 = lshift(and(_by1, 3), 4) + rshift(and(_by2, 240), 4)
                _ba3 = lshift(and(_by2, 15), 2) + rshift(and(_by3, 192), 6)
                _ba4 = and(_by3, 63)

                ###     transmute values to BASE64 string  
                _res = _res substr(_BASE64, _ba1 + 1, 1)
                _res = _res substr(_BASE64, _ba2 + 1, 1)
                if (length(_b64str) == 1){
                        _res = _res "=="
                        _b64str = ""
                }
                if (length(_b64str) == 2){
                        _res = _res substr(_BASE64, _ba3 + 1, 1)
                        _res = _res "="
                        _b64str = ""
                }
                if (length(_b64str) >= 3){
                        _res = _res substr(_BASE64, _ba3 + 1, 1)
                        _res = _res substr(_BASE64, _ba4 + 1, 1)
                        _b64str = substr(_b64str, 4)
                }
                if( 0==(++_wrap) %18) _res=_res ORS
        }
        return _res

The short answer is that indeed awk can do what you want.  The trick in
processing mail headers is that continuation lines are marked by a leading
space or tab *after* the header they are part of. You should also take into
account that header names are case insensitive - "To:", "to:" and "TO:" are
all the same.

You can see a fairly fancy program at http://www.skeeve.com/sendout3.ps.gz
which I wrote a long time ago - part of it processes Unix mailbox files,
including the headers.  (Note that it is quite old, and tailored just for
a personal situation. Also note that all of the example email addresses
in it are invalid.)

Converting quoted printable in awk is also not hard. Basically, the "="
sign either precedes a newline that was added, or is followed by two
hexadecimal digits indicating an encoded character.

You do not need to buy any awk book. The gawk documentation is available
on line, free of charge, in a variety of formats at

        http://www.gnu.org/software/gawk/manual/

Although I'm biased, I think this a great way to learn awk.

As a general plan of action, I recommend reading the gawk doc first,
in order to come up to speed on the language (hopefully in a gentle fashion :-)
and then attempting to write some code to do what you want.  Once you have
that, if it doesn't work, come back here to ask questions.

I also recommend using the latest released version of gawk.

Good luck,

Arnold

In article <f97k58hdtpqnct0ihqvhmobn15qb54s...@4ax.com>,
Steve Hayes  <hayes...@yahoo.com> wrote:

But when I've learnt a bit more of the basics I might try it.

If you mean honest-to-goodness actual MS-DOS, then getting a version for
it will be harder. I believe that current sources can be compiled with
DJGPP, but I don't know if that will get you what you want.

You can probably find something on the Internet, but it's likely to be
an older version, and often such versions have bugs...  So, Caveat Emptor. :-)

Good luck,

Maybe think of it as:

  IF RegexCompare(CurrentLine, delete_list[bufpos + 1]) THEN

Here's the fixed version of the script:

----script begins on next line
#! /usr/bin/awk -f

function flush_buffer(discard,     n) {
  if (!discard)
    for (n = 1; n <= bufpos; n++)
      print buffer[n]
  bufpos = 0

bufpos {
  flush_buffer(0)
  if (try_seq($0))
    next

END {
  flush_buffer(0)

The script works by loading the first file's contents into an
array.  The array is a sequence of regular expressions.

The second file is scanned for sequences of lines in which each
line matches the corresponding entry in the array of regular
expressions.

When a complete sequence of matches is made the matched lines are
discarded if they are not the first occurrence of a match-sequence.

Input file 1 (FileDelete2) contains three lines:
a
b
[cz]

Input file 2 (FileIn2) contains 17 lines:
1 : nothing
2 : a           MATCHES THE FIRST PATTERN IN THE SEQUENCE
3 : b           MATCHES THE SECOND PATTERN IN THE SEQUENCE
4 : k
5 : a 2         SEQUENCE BEGINS
6 : b 2
7 : c 1         FULL SEQUENCE MATCHES (FIRST TIME: 5,6,7 PRINTED)
8 : a 3         SEQUENCE BEGINS
9 : a 4         SEQUENCE FAILS, LINE 8 PRINTED, NEW SEQUENCE BEGINS
10: b 3
11: c 2         FULL SEQUENCE MATCHES (NOT FIRST TIME: 9,10,11 OMITTED)
12: NO MATCH
13: c 3         OUT OF SEQUENCE, NO MATCH
14: a 5         FIRST IN SEQUENCE
15: b 4         SECOND IN SEQUENCE
16: z 1         THIRD IN SEQUENCE (14, 15, 16 DROPPED)
17: example     SEQUENCE BEGINS, FAILS DUE TO END-OF-INPUT, 17 PRINTED

The command

  awk -f the_above_script FileDelete2 FileIn2

Someone contaminated my thread.

lets move this to:
Newsgroups: comp.lang.awk,comp.os.linux.misc
Subject: awk: DeleteRepeatingTextBlocks

Let's try to add-value for the *nix community by revealing
methods which others can modify and use for their problems.

===========

It's called top-down-design.
The implementation details, which must be bottom-up,
are best not explained until the top-down-design is
known.  Here's my STRUCTURED view:

SpeedUp knowledge absorbtion from http-fetched text
   Delete noise/distracting repeated garbage
     Identify garbage
       Must be done by human intelligence
        use a standard editor -- while reading/studying the text
     Automate the removal of further garbage-repeats
       Search the InFile for further matches of human-identified-trash

==> PS. I'm writing this WHILE I'm trying to decode your explanation.
The decomposition-chain is: Delete needs Match needs Regex.

You are going to compare the DeleteFile with the InFile.
To handle the regex requirement, you are building
"an array of regular expressions".
Apparently to <match the array with InFile parts> ?

My test results for your new script are dumb, since I have no
intermediate output traces.
See Subject: awk: DeleteRepeatingTextBlocks

Thanks,

On 2012-09-28, f...@informatik.uni-bremen.de <f...@informatik.uni-bremen.de> wrote:

--keith