Excessive blank lines in Dialog

[PeterC]

Is there a way for Dialog to remove the blank, quoted lines in received
items? They make scrolling down a chore - sometimes several pages for a few
words. e.g:

> >> the backs of cabinets, which will never actually be seen - i.e all
>
> >>
>
> >> but the end panels - made from solid timber is something only the
>
> >>
>
> >> suppliers could possibly answer.
>
> >>
>
> >>
>
> >>
>
> >>
>
> >>
>
> >> michael adams
>
> >>
>
> >>
>
> >>
>
> >> ...
>
> >>
>
> >>
>
> >>
>
> >>
>
> >>
>
> >> Examples of suppliers:
>
> >>
>
> >> http://www.discountkitchenfactory.co.uk/
>
> >>
>
> >> http://www.solidwoodkitchencabinets.co.uk/
>
> >>
>
> >> http://www.solidwoodkitchen.co.uk/
>
> >>
>
> >>
>
> >>
>
> >> thanks for any comments,
>
> >>
>
> >>
>
> >>
>
> >> Robert
>
> >>
>
> >> laws at bcs.org.uk
>
> >
>
> >
>
> > Chip doesn't last well.
>
>
>
> It usually lasts long enough to fulfil its intended purpose.

I plan to use my kitchen for the rest of my life, then pass it on.

> Up to 40 years and more, in some cases

That is very optimistic for chip. I do however know a real wood kitchen
still in use that looks over a century old.


> > Timber does.
>
>
>
> Often too long for a kitchen which will often be replaced within
>
> 30 years if not sooner.
--
Peter.
The gods will stay away
whilst religions hold sway

[Ted S.]

On Sat, 29 Sep 2012 09:35:43 +0100, PeterC wrote:

> Is there a way for Dialog to remove the blank, quoted lines in received
> items?

Have you tried pressing Q?

--
Ted S.
fedya at hughes dot net
Now blogging at http://justacineast.blogspot.com

[VanguardLH]

"PeterC" wrote:

> Is there a way for Dialog to remove the blank, quoted lines in received
> items? They make scrolling down a chore - sometimes several pages for a few
> words. e.g:
>

>> [83 quoted lines suppressed]

>
> I plan to use my kitchen for the rest of my life, then pass it on.
>

>> [1 quoted line suppressed]

>
> That is very optimistic for chip. I do however know a real wood kitchen
> still in use that looks over a century old.
>

>> [7 quoted lines suppressed]

See above on how the "Q" (Suppress Quoted Lines) option works. As
further suggestion to Ted's response, notice the down-arrow at the left
of the Subject line in the preview pane's header area? Press it.
Notice the options you can select, like Message View Options. That will
show the "Q" hotkey for the "Suppress quoted lines" option. It's the
same menu as the one for the Message -> Message View Options menu.

Some users add lots of blank lines to their posts as an anti-spoiler
maneuver. To see the answer means having to scroll down to see it.
Another cause is for someone using Outlook Express which quoted the
empty lines at the bottom instead of trimming them (I don't know if this
defect migrated into Windows Live Mail). Probably the biggest cause for
sloppy quoted content are due to the LAZY posters that never bother to
review, trim, and compress their reply. They just slap in their new
content in a reply and immediately click Send without trimming what they
quote. While such laziness is exhibited by many users of all NNTP
clients, OE, WLM, and G2 (Google Groups) users are the worst offenders.

You chose not to provide the news article via MID (Message-ID header
value so I had to go searching based only on some of the content (which
you say contains blanks lines but does not). I found the thread
starting with MID ca649461-8db0-4244...@googlegroups.com
(news:ca649461-8db0-4244...@googlegroups.com):

https://groups.google.com/group/uk.d-i-y/browse_frm/thread/bf880d94cac5397b

michael adams reply did not have the blank lines. Meow's reply added
the blank lines (which were NOT part of the original post). That poster
added NEW content by adding blank lines BETWEEN the lines that he quoted
in his reply. The new content constitutes inline replies to each line.
Meow used Google Groups to submit his post but the G2 client (that I
don't use) so I cannot comment if the problem is due to using G2, how
meow has his web browser configured, or if meow deliberately and
willfully added the inline comments that were blank. Then michael
adams, a LAZY user of Outlook Express, didn't bother to trim his reply,
especially the quoted content. meow followed with equal laziness and
didn't bother trimming his reply along with adding yet more blank lines
to the quoted content. The more these two posters reply to each other
the more they fuck up their articles because they are a lazy posters.
Neither one trims their reply and meow keeps inserting blank inline
comments into the quoted content.

Perhaps you should tell meow aka NT to stop inserting blank inline
comments into the quoted portion of his replies. He may not be aware of
the problem since most users that don't bother to trim their replies
also never bother to review their own submitted posts. As far as
suggesting to meow and adams that their review their posts before
submitting them and to trim them, well, I've found lazy poster won't
bother to change so they continue to be lazy.

If sloppiness in replies by lazy posters really upsets you, it is
possible to write a script for use within Dialog that will strip out
blank lines. However, it is unlikly that blank lines used as whitespace
for legibility, like between paragraphs, could be differentiated from
superfluous blank lines added as inline comments (which is what meow did
in his replies). You would end up removing all blank lines. Also,
while it may be possible to have this script get exercised on an article
when it gets retrieved (as part of the OnBeforeSavingMessage event), a
better use would be to employ it when you reply to an article (as part
of the OnBeforeSending event). I'm not good at writing new scripts for
Dialog particularly as the documentation is missing on what objects are
available within the scope in which the script is executed. You end up
writing scripts for an unknown environ. However, you can see example
scripts at http://dialog.datalist.org/scripts/script_library.html. I've
used those either as-is, modified them, or to write up new scripts.
Then you'll have to figure within which event to add the script
(Settings -> Scripting -> Scripting). The examples may help but not if
you want to define a script within an event not used by those examples
and there's no documentation of what events are available to you.

The "Q" keyboard shortcut smashes all quoted content into a summary
message of "[x quoted lines suppressed]". While that gets rid of
showing all of the quoted content, it really doesn't do what you want
which is to *remove* the blank lines. Dialog supports user-defined
scripts. You would modify the OnBeforeSavingMessage event to add script
that would scan a retrieved article, remove blank lines, and then save
the modified version of that article (which you would then see inside of
Dialog). Besides removing blank lines, you'll probably only want to
remove partially blank lines where the first character of the line is
one, or more, contiguous quoting characters (with intervening spaces
between them ignored) and where the rest of the line is blank. By using
a loop that compresses the spaces after a quoting character until the
next non-blank character, you would end up with lines like ">>> text" or
">>>" where in the last case you removed spaces but never found a
following non-blank character, and that's the line you would then remove
from the article. I've got a script that does something similar. It
removes the worthless PGP sigs that some posters add to their posts
thinking that I'm going to waste my time going elsewhere to validate
their PGP key. It's crap in a post that the script will remove. It
came from http://dialog.datalist.org/scripts/ScriptNukePGP.html.
Another script that removes content from articles is shown at
http://dialog.datalist.org/scripts/ScriptNukeYahooAds.html. They could
be modified into a new script that removes blank lines and also any
blank quoted lines. They give you an idea of the looping needed and how
to delete a line but you would need to add the logic to test for a blank
line or a quoted blank line to decide that was what you wanted deleted.
Of course, in deleting all blank lines and all quoted blank lines, you
could render the article harder to read since blank lines are often
added as whitespace to improve legibility. I suppose the loop could
keep track of the first blank line and only delete the next line if it
was also blank (and the same when scanning for quoted blank lines) which
would eliminate just the multiply contiguous blank lines but leave one
blank line. Have fun with that. If you that route, and come up with a
robust script that removes [quoted] blank lines or compresses multiple
blank lines, please post it here (as I don't know if anyone at
dialog.datalist.org is administering that site to add new scripts).

[tlvp]

On Sat, 29 Sep 2012 09:35:43 +0100, PeterC wrote:

> Is there a way for Dialog to remove the blank, quoted lines in received

> items? They make scrolling down a chore - sometimes several pages ...

Ah, yes, the Google multi-page blank lines plague :-) .

Alas, there's not much you can do beyond (i) trying to convince those who
program for Google Groups to stop interspersing such blank lines in between
their lines of quoted material, (ii) displaying no quoted lines at all, and
(iii) living with the nuisance of those blank quoted lines.

Dialog lets you toggle on/off "Suppress [all] quoted lines" with Q, but i
it has no toggle I'm aware of for "Suppress [only] blank quoted lines".

Just about every news reader will suffer the same problem, it's not unique
to Dialog :-) .

HTH. Cheers, -- tlvp
--
Avant de repondre, jeter la poubelle, SVP.

[PeterC]

Many thanks for the info. and suggestions. Looks as if Q is my friend
(rather have Moneypenny) and to be used judiciously.
I did want to suppress just the blank lines (somehow I'd guessed that they
might be from Google) but, as in a recent post, suppressing all can be handy
:-)

[PeterC]

On Sat, 29 Sep 2012 11:21:25 -0500, VanguardLH wrote:

Apologies for top-posting, but...

Looks as if the problem has been addressed, then. Scripts and so-on are well
beyond my expertise so I'll stick with Q for now, but thanks for the info.

[Bernd Rose]

On Sun, 30 Sep 2012 10:04:39 +0100, PeterC wrote:

> On Sat, 29 Sep 2012 11:21:25 -0500, VanguardLH wrote:

[Remove consecutive blank quoted lines]

> Looks as if the problem has been addressed, then. Scripts and so-on are well
> beyond my expertise so I'll stick with Q for now, but thanks for the info.

Maybe you can take the following code as an example and starting point
for you own adjustments. It is just a quick hack, but hopefully will
not destroy valid parts of messages. - Although any bugs are mine, the
responsibility of using the script is left to you... ;-)

The script tries to detect and retain valid paragraphs and cleans up
quoting chars with spaces in between. - Part of the code is adopted
from Dietmar Vollmeiers QuoteCorrection scripts:

http://4d.vollmeier.at/scripte/ereignisscripte/onbeforesavingmessage/quotecorrection.html

If you don't already use an OnBeforeSavingMessage event script, just
replace the current code with the one below. Else, just copy the
TrimQLines procedure inside the current OnBeforeSavingMessage program
body and insert the

TrimQLines(Message);

line into the OnBeforeSavingMessage procedure body.

Afterwards, you need to save and compile the event script.

HTH.
Bernd

-----------------------------<Snip>------------------------------------
program OnBeforeSavingMessage;

Procedure TrimQLines(var Message : TStringlist);
Var
iPtr, iCnt, iMax : Integer;
sLine, sTmp, sBuffer : String;
bBuff : Boolean;
Begin
With Message Do
Begin
If IndexOf('-- ') > -1 Then iMax := IndexOf('-- ') - 1
If IndexOf('--') > -1 Then iMax := IndexOf('--') - 1
Else iMax := Count - 1;
iPtr := IndexOf('') + 1;
sBuffer := '';
bBuff := True;
For iCnt := IndexOf('') + 1 To iMax Do
Begin
sLine := Strings[iCnt];
sTmp := StringReplace(sLine, '>', '', [rfReplaceAll]);
sTmp := StringReplace(sTmp, ':', '', [rfReplaceAll]);
sTmp := Trim(StringReplace(sTmp, '|', '', [rfReplaceAll]));
If (Length(sTmp) > 0) Or (Trim(sLine) = '') Then
Begin
If (Length(sBuffer) > 0) And bBuff Then
Begin
Strings[iPtr] := sBuffer;
iPtr := iPtr + 1;
End;
sTmp := TrimRight(sLine);
If (Length(sTmp) > 2) Then
If AnsiPos(sTmp[1], '>:|') > 0 Then
Begin
sTmp := StringReplace(sTmp, '> >', '>>', [rfReplaceAll]);
sTmp := StringReplace(sTmp, ': :', '::', [rfReplaceAll]);
sTmp := StringReplace(sTmp, '| |', '||', [rfReplaceAll]);
End;
Strings[iPtr] := sTmp;
iPtr := iPtr + 1;
sBuffer := '';
If (Length(sTmp) < 55) Then bBuff := True
Else bBuff := (AnsiPos(sTmp[Length(sTmp)], '.?!-') > 0);
End
Else
Begin
sTmp := StringReplace(sLine, ' ', '', [rfReplaceAll]);
If Length(sTmp) > Length(sBuffer) Then sBuffer := sTmp;
End;
End;
For iCnt := iPtr To iMax Do Delete(iPtr);
End;
End;

procedure OnBeforeSavingMessage(var Message: TStringlist; Servername:
string; IsEmail: boolean);
begin
If Message.Count = 0 Then Exit;
TrimQLines(Message);
end;

begin
end.

[VanguardLH]

I even noticed Google Groups renders a different version of those
articles. When you view their normal web page showing the article, you
see blank lines which are at the level of the current article; that is,
they look like inline comments (which are blank lines). Yet if you look
at the original/raw source of the article (which means having to use
their old format), those inline comments of blank lines actually have a
quoting character at their start. Besides fucking up their UI with
their "new" layout which remove functionality that was available in
their old layout, apparently Google can't figure out how to display
multiple blank quoted lines.

Thankfully I filter out all the spammers, malcontents, trolls, and
uber-boobs that post via Google Groups.

[Bernd Rose]

On Sun, 30 Sep 2012 19:36:43 -0500, Sqwertz wrote:

> What is the OnBeforeSavingMessage trigger? Is this the point after it
> is read VIA NNTP and before it is written to the database?

Yes.

Bernd

[PeterC]

On Sun, 30 Sep 2012 21:17:18 +0200, Bernd Rose wrote:

> On Sun, 30 Sep 2012 10:04:39 +0100, PeterC wrote:
>
>> On Sat, 29 Sep 2012 11:21:25 -0500, VanguardLH wrote:
> [Remove consecutive blank quoted lines]
>> Looks as if the problem has been addressed, then. Scripts and so-on are well
>> beyond my expertise so I'll stick with Q for now, but thanks for the info.
>
> Maybe you can take the following code as an example and starting point
> for you own adjustments. It is just a quick hack, but hopefully will
> not destroy valid parts of messages. - Although any bugs are mine, the
> responsibility of using the script is left to you... ;-)
>
> The script tries to detect and retain valid paragraphs and cleans up
> quoting chars with spaces in between. - Part of the code is adopted
> from Dietmar Vollmeiers QuoteCorrection scripts:
>
> http://4d.vollmeier.at/scripte/ereignisscripte/onbeforesavingmessage/quotecorrection.html
>
> If you don't already use an OnBeforeSavingMessage event script, just
> replace the current code with the one below. Else, just copy the
> TrimQLines procedure inside the current OnBeforeSavingMessage program
> body and insert the
>
> TrimQLines(Message);
>
> line into the OnBeforeSavingMessage procedure body.
>
> Afterwards, you need to save and compile the event script.
>
> HTH.
> Bernd

Bernd.

Thank you for this. Eventually I dropped a script in to the Custom folder
and found the Compile part in Dialog. Not sure if it's working as, due to
Sod's Law, I had to wait several days to get enough posts coming in to see
what was happening.
So far it looks as if > might have gone but >> or more are still there.

I don't actually understand any of this, so is the following OK?

I have OnBeforeSavingMessage.cds:

(*******************************************************************************
* QuoteCorrection 2.0.2
*
* OnBeforeSavingMessage
*
* OnBeforeSendingMessage
*
*******************************************************************************)

[Bernd Rose]

On Wed, 3 Oct 2012 16:37:52 +0100, PeterC wrote:

[Script to remove consecutive blank quoted lines]

> Thank you for this. Eventually I dropped a script in to the Custom folder
> and found the Compile part in Dialog. Not sure if it's working

No. This way it will not work. First, it needs to be an *Event* script.
And second, *.cds should be the compiled version (the uncompiled is *.ds).

You could move <Dialog Folder>\Scripts\Custom\OnBeforeSavingMessage.cds to
<Dialog Folder>\Scripts\Custom\OnBeforeSavingMessage.ds (note the change
in the extension) while Dialog is not running and - after starting Dialog -
got to the event scripts section to compile it. At least in case of an
event script I suggest doing it the "proper" way though:

Got to <Settings>-><Scripting>-><Scripting...>, switch to [Event scripts]
(a tab on the left side of the "Scripting" dialog) and select the entry
"OnBeforeSavingMessage". It should be there, even if it currently has no
content. (There is only a finite number of event scripts with exactly
defined names. Therefore, Marcus created empty stubs for all of them.)

If you already have code inside OnBeforeSavingMessage, then it will show
black (with few other colors) in the right window pane. In this case you
need to insert the 50 lines of "Procedure TrimQLines" into the upper
part of "program OnBeforeSavingMessage". (Best scroll down till you find
the first "Procedure" or "Function" already there and insert the 50 lines
immediately before it. - Maybe with one empty line in between for better
readability of the code.)

Afterwards, scroll down to the "procedure OnBeforeSavingMessage" and
insert the "TrimQLines(Message);" line. If it isn't entirely clear to
you, where to insert this code, post "procedure OnBeforeSavingMessage"
here for advise.

It is likely, though, that you *don't* already have code for the event.
In this case, you'll see just two empty gray lines in the right window
pane stating:

// There currently exists no script for the selected
// event. Please select File>New to create it now.

In this case, follow the explanation of these two lines and execute
<File>-><New Script...>. (Be sure, that OnBeforeSavingMessage still
is selected while doing so.) *Replace* the whole code created in the
right window pane with the complete code I provided in my last
posting. Please note, that there is /no/ line wrap after "Servername:"!
Maybe it doesn't matter with PascalScript (it shouldn't), but better
make sure.

Whatever way you had to choose (inserting into existing code or
creating a new script), now you need to save (menu <File>-><Save>)
and compile (menu <Script>-><Compile>) the script. If no errors
show up, you can close the "Scripting" Dialog and see, if the
script works as intended when loading new messages.

As a test, I suggest you to browse to a message with offending
blank lines. (Maybe the one you provided as an example.) Delete
the /body/ (not the whole message!) of that message (right mouse
menu or menu <Message>-><Delete Body>) and retrieve the body
anew by double clicking on the header. If all went well, the
script should provide fewer "pseudo-blank" lines.

> So far it looks as if > might have gone but >> or more are still there.

Although it doesn't matter until you have the script up and
running: There will be lines with just quote characters left
(for better readability) after the script did its job. The
quote kept should show always the deepest quote level found
for consecutive pseudo-blank lines.

HTH.
Bernd