Ping Bernd Rose: How do you set and define a variable in 40Tude Dialog

[Stijn De Jong]

In another thread, much to everyone's surprise and dismay, we found it
impossible to figure out the syntax to simply set and define a variable in
40Tude Dialog?
http://i.imgsafe.org/94013af980.jpg

The thread is here but the question now is simpler than the original:
Request help with 40tude dialog program syntax
https://groups.google.com/forum/#!topic/news.software.readers/sr56Wrng4Ks

The new question is simpler in that all we ask is the syntax that allows us
to do two very simple things:
1. How do you set a variable in 40Tude dialog,
2. So that you can then assign that variable a value?

Very many syntactic attempts failed, so here is just one attempt:

program OnBeforeSendingMessage;
var
Remove_Headers : String;
Remove_Headers:='foo';
Remove_Headers:='bar';
const
RemoveFromEmails = true;
RemoveFromNews = true;
procedure RemoveHeaders ( Message : TStringlist;
IsEmail : boolean
);

Failed when compiling
[Error] OnBeforeSendingMessage.ds(4:1): Duplicate identifier
'Remove_Headers'

Is it syntactically possible to set & then define a variable in 40Tude?
How?

[Peter Köhlmann]

Stijn De Jong wrote:

>
> Is it syntactically possible to set & then define a variable in 40Tude?
> How?

You should perhaps ask in a windows group about your outdated newsreader.
A linux group will not yield that may answers to your question

[Stijn De Jong]

Op Sun, 01 Jan 2017 20:19:20 +0100, Peter K?hlmann schreef:

> You should perhaps ask in a windows group about your outdated newsreader.
> A linux group will not yield that may answers to your question

The code base is here:
http://dialog.datalist.org/scripts/script_library.html

I was hoping someone like Marek, who knows the language of "common syntax",
could instantly figure out the syntax for
a. Defining a string variable, and then,
b. Redefining the variable.

I'm not a programmer, but many of the Linux gurus are, so they can probably
figure out the syntax in a second from the two files here:
This is the code which sets the identity.
http://dialog.datalist.org/scripts/CheckingIdentity.html

This is the code which removes the headers:
http://dialog.datalist.org/scripts/RemoveHeaders.html

All I'm trying to do is meld the two, where the outgoing headers to be
removed depend on the identity of the sender.

To do that, I need to re-define the string variable based on the identity.

It seems tenable, to me, to argue that a Linux guru who is familiar with
various script languages may be able to point out the syntax issue. I could
easily be wrong though, as you imply, that Linux gurus don't necessarily
know scripting syntax any better than Windows users.

[Bernd Rose]

On Sun, 1 Jan 2017 19:02:25 +0100, Stijn De Jong wrote:

> 1. How do you set a variable in 40Tude dialog,
> 2. So that you can then assign that variable a value?

Program Test;
var sTest:string;
Begin
sTest:='XXX';
sTest:='YYY';
WriteLn(sTest);
End.

Will print 'YYY'. (As expected.)

Bernd

[Jernej Simončič]

On Sun, 1 Jan 2017 19:02:25 +0100, Stijn De Jong wrote:

> Is it syntactically possible to set & then define a variable in 40Tude?

Dialog uses (a fairly old version of) RemObjects Pascal Script
<http://www.remobjects.com/ps.aspx> - look up Delphi and Inno Setup script
examples for the syntax (just remember that you can't define your own
classes, and not all constructs from Delphi are supported).

--
begin .sig
< Jernej Simončič ><>◊<>< jernej|s-ng at eternallybored.org >
end

[Stijn De Jong]

Op Sun, 1 Jan 2017 21:01:12 +0100, Bernd Rose schreef:

> Program Test;
> var sTest:string;
> Begin
> sTest:='XXX';
> sTest:='YYY';
> WriteLn(sTest);
> End.
>
> Will print 'YYY'. (As expected.)

Thanks Bernd for the syntax help in setting variables in 40tude language.

That snipped works just fine standalone:
http://i.imgsafe.org/99dc6c0dbe.jpg

Program setRemoveHeadersString;
var Remove_Headers:string;
Begin
Remove_Headers := 'X-Scoring: ,X-Hamster-Info: ';
Remove_Headers := 'X-Scoring: ';
WriteLn(Remove_Headers);
End.

Integrating the snippet into the OnBeforeSendingMessage utility is the task
at hand, where an integration requires insertion into the if/then/else.

I spent a few hours more and finally gave up as my conclusion is that the
syntax is just too daunting for a novice to overcome without prior
knowledge of (probably quite a few) programming languages.

I paste below where I gave up so that others may pick up if they like from
where I left off. What I pasted is syntactically correct. There are no
errors. But it doesn't work either. So it ends up doing absolutely nothing.

To explain the confusingly long script below, it's simply these two scripts
put together:
1. Determine Identity (either work or personal):
http://dialog.datalist.org/scripts/CheckingIdentity.html
2. Remove Headers:
http://dialog.datalist.org/scripts/RemoveHeaders.html

The goal was simply to meld the two above so that the headers change
depending on the identity (either work or personal).
--------------- cut here -------------
// The goal is different headers for work use versus for personal use.
// This script has no syntax errors.
// Modify the following values to fit your setup
// 1. Remove_Headers (one for work and one for personal)
// 2. Identity (one for work and one for personal)
// 3. Newsgroups (work newsgroups can be different than personal ones)
// 4. Servers (work servers can be different than personal servers)
//
program OnBeforeSendingMessage;

var Remove_Headers:string;
Begin
// These are just default values.
// The real values are set in an if-then-else statement later.
Remove_Headers := 'X-Scoring: ,X-Hamster-Info: ';
Remove_Headers := 'X-Scoring: ';
WriteLn(Remove_Headers);
End.

RemoveFromEmails = true;
RemoveFromNews = true;
procedure RemoveHeaders ( Message : TStringlist;
IsEmail : boolean
);

var i : integer;
k : integer;
s : string;
CommaPos : integer;
DelHeader : TStringlist;
RemoveH : String;
begin
RemoveH := Remove_Headers;
i := 0;
if ((IsEmail=true) and (RemoveFromEmails=true)) or
((IsEmail=false) and (RemoveFromNews=true)) then begin
If ( RemoveH <> '' ) then begin
try
DelHeader := TStringlist.Create;
if ansipos ( ',', RemoveH) = 0 then begin
DelHeader.Add ( LowerCase ( TrimLeft( RemoveH )));
end // if
else begin
CommaPos := 0;
for k := 1 to length ( RemoveH ) do begin
If RemoveH[k] = ',' then begin
DelHeader.Add ( LowerCase ( TrimLeft (copy ( RemoveH,
CommaPos + 1, k - ( CommaPos + 1 )))));
CommaPos := k;
end; // if
if k = length ( RemoveH ) then
DelHeader.Add ( LowerCase ( TrimLeft (copy ( RemoveH,
CommaPos + 1, k - CommaPos ))));
end; // for
end; // else
s:=Message.text;
while (Message.Strings[i]<>'') do begin
k := 0;
while k <= ( DelHeader.Count - 1 ) do begin
if pos( DelHeader[k], LowerCase ( Message.Strings[i] )) = 1
then begin
delete ( s, pos(DelHeader[k], LowerCase (s) ), length (
Message.Strings[i] ) + 2 );
i := i - 1;
k := DelHeader.Count - 1;
message.text := s;
end; // if
k := k + 1;
end; // while
i := i + 1;
end; //while
message.text:=s;
finally
DelHeader.Free;
end; // try - finally
end; // if
end; // if
end; // RemoveHeaders

function StrMatch(str: String; pattern: String):Boolean;
var
patternSize : Integer;
subStr : String;
compareRes : Integer;
begin
patternSize := Length(pattern);
subStr := Copy(str, 1, patternSize);
compareRes := CompareStr(pattern, subStr);
if (compareRes = 0) then
result := true
else
result := false;
end;

function From2Identity(var Remove_Headers TStringlist; from: String;):
String;
begin
if (StrMatch(from, 'First1 Last1 <ema...@example.com>')) then
result := 'id1'
Remove_Headers='User-Agent: ,Message-ID: '
else if (StrMatch(from, 'First2 Last2 <ema...@example.com>')) then
result := 'id2'
Remove_Headers='Message-ID: ,Mime-Version: '
else
result := 'unknow';
WriteLn(Remove_Headers);
end;

function NewsGroup2Identity(newsgroup: String): String;
begin
if (StrMatch(newsgroup, 'news.software.readers') or
StrMatch(newsgroup, 'alt.os.linux')) then
result := 'id1'
else
result := 'id2';
end;
// The server identity doesn't seem to play any role in the decisions
though...
function Server2Identity(server: String): String;
begin
if (CompareStr(server, 'aioe_119') = 0) then
result := 'id1'
else
result := 'id2';
end;

function BadIdentity(): boolean;
begin
result := false;
end;

function CheckIdentity(var message: TStringlist; servername: string;
isEmail: boolean):boolean;
var
fromIdentity : String;
newsgroupIdentity : String;
serverIdentity : String;
i : Integer;
begin
if (not IsEmail) then
begin
for i := 0 to Message.Count - 1 do
begin
if (strMatch(Message[i], 'From:')) then
fromIdentity := Copy(Message[i], 7, Length(Message[i]) - 6);
if (strMatch(Message[i], 'Newsgroups:')) then
newsgroupIdentity := Copy(Message[i], 13, Length(Message[i]) - 12);
end;
fromIdentity := From2Identity(fromIdentity);
newsgroupIdentity := NewsGroup2Identity(newsgroupIdentity);
serverIdentity := Server2Identity(servername);
WriteToLog(' fromIdentity = ' + fromIdentity, 7);
WriteToLog(' newsgroupIdentity = ' + newsgroupIdentity, 7);
WriteToLog(' serverIdentity = ' + serverIdentity, 7);
WriteToLog(' Remove_Headers = ' + Remove_Headers,7);
if ((CompareStr(fromIdentity, newsgroupIdentity) = 0) and
(CompareStr(newsgroupIdentity, serverIdentity) = 0)) then
result := true
else
result := BadIdentity();
end
else
result := true;
end;

function OnBeforeSendingMessage(var Message : TStringlist;
Servername : string;
IsEmail : boolean
):boolean;
begin
// Added the result line below:
RemoveHeaders(Message,IsEmail);
WriteLn(Remove_Headers);
result := CheckIdentity(message, servername, isEmail);
result:=true;
end;
// ----------------------------------------------------------------------
begin
end.
--------------- cut here -------------

[Bernd Rose]

On Mon, 2nd Jan 2017 02:49:30 +0100, Stijn De Jong wrote:

> Thanks Bernd for the syntax help in setting variables in 40tude language.
>
> That snipped works just fine standalone:

[...]

> Integrating the snippet into the OnBeforeSendingMessage utility is the task
> at hand, where an integration requires insertion into the if/then/else.
>
> I spent a few hours more and finally gave up as my conclusion is that the
> syntax is just too daunting for a novice to overcome without prior
> knowledge of (probably quite a few) programming languages.

You should read the reference recommended by the author of 40tude Dialog,
Marcus Mönnig! If you follow the link provided in the help file:

http://www.marcocantu.com/epascal/default.htm

you get to the main page, which forkes to different versions, languages,
and places where a print-out book can be acquired. The current English
version starts here:

http://www.marcocantu.com/epascal/English/default.htm

If you take a look in chapter 11, you'll notice your main error: "end."
(with dot, /not/ with semicolon!) is the *last* instruction of a program.
The code of that *program* starts with the only same-level "begin" which
appears between "program" and "end." There may be lots of other "begin" /
"end;" (note the semicolon!) statements embedded. They need to /always/
come in pairs and /must not/ written entangled (like begin[1] - begin[2] -
end;[1] - end;[2]. Correct order is <last opened has to be closed first>,
e.g.: begin[1] - begin[2] - end;[2] - end;[1].

All declarations (var and const definitions, /as well as/ procedure and
function definitions[!!] appear between "program" and the program-level
"begin".

In your case, a special situation has to be regarded, though. All event
scripts in Dialog are /not/ run as a program. Therefore, the section
between program-level <begin - end.> is empty in both your basis scripts
you copied from datalist.org. (And is empty in every newly-created event
script stub in the Dialog scripting editor.) Event scripts have to carry
an "overload" function of the same name as the event. If you wish to
modify the normal OnBeforeSendingMessage behavior, you need to create
an event script, which has a program stub that carries a function named
OnBeforeSendingMessage inside the declaration area of the program.

HTH.
Bernd

[Stijn De Jong]

On Mon, 2 Jan 2017 19:38:13 +0100, you wrote:

> In your case, a special situation has to be regarded, though. All event
> scripts in Dialog are /not/ run as a program. Therefore, the section
> between program-level <begin - end.> is empty in both your basis scripts
> you copied from datalist.org. (And is empty in every newly-created event
> script stub in the Dialog scripting editor.) Event scripts have to carry
> an "overload" function of the same name as the event. If you wish to
> modify the normal OnBeforeSendingMessage behavior, you need to create
> an event script, which has a program stub that carries a function named
> OnBeforeSendingMessage inside the declaration area of the program.

Thanks to JJ <jj4p...@vfemail.net> in this newsgroup and in another
thread, here is a working script that has been modified to allow multiple
program conditions.

http://pastebin.com/LvPvMCPN

The overall goal is to keep personal and business news and/or email
separate simply by setting appropriate variables in the script below.

Here is that same script pasted into this post, but it might get modified
by newsreaders so the pastebin raw source is probably more reliable.

All the thanks go to the original authors, and JJ and those who helped in
between!

// JJAddAndRemoveHeadersWhileDoubleCheckingIdentityNgAndServers
// <http://pastebin.com/2tMYRKM4>
// From: JJ <jj4p...@vfemail.net>
// Newsgroups: news.software.readers,alt.windows7.general
// Subject: Re: Request help with 40tude dialog program syntax
// Date: Thu, 5 Jan 2017 23:06:19 +0700
// Message-ID: <7vpx5y09pu98.1sukl5d600sda$.d...@40tude.net>
// Original code for checking identity:
// http://dialog.datalist.org/scripts/CheckingIdentity.html
// Original code for modifying headers:
// http://dialog.datalist.org/scripts/RemoveHeaders.html
// CUSTOMIZATION:
// 1. Set the "ForEmail" Boolean as desired (search for "ForEmail :")
// 2. Set the "ForNewsgroup" Boolean as desired (search for "ForNewsgroup
:")
// 3. Change identity as desired (search for "from,")
// 4. If ForNewsgroup is true, change newsgroup as desired (search for
"newsgroup,")
// 5. Change server as desired (search for "server,")
// 6. Change remove header as desired (search for "Remove_Headers :")
// 7. Change add header as desired (search for "Add_Headers :")

program OnBeforeSendingMessage;

(*
Format for Remove_Headers: {} = required, [] = optional
{HeaderName: }[,HeaderName: ][,HeaderName: ][...]
Examples:
- Single header: 'User-Agent: '
- Multiple headers: 'User-Agent: ,X-Face: '
*)
procedure RemoveHeaders(Message : TStringlist;
const Remove_Headers: String

);
var i : integer;
k : integer;
s : string;
CommaPos : integer;
DelHeader : TStringlist;
RemoveH : String;
begin
RemoveH := Remove_Headers;
i := 0;

(Message.Strings[i] ) + 2 );

i := i - 1;
k := DelHeader.Count - 1;
message.text := s;
end; // if
k := k + 1;
end; // while
i := i + 1;
end; //while
message.text:=s;
finally
DelHeader.Free;
end; // try - finally
end; // if

end; // RemoveHeaders

(*
Format for Add_Headers: {} = required, [] = optional
{HeaderName: HeaderValue{#13#10}}[HeaderName: HeaderValue{#13#10}][...]
Examples: (each header must end with CR+LF)
- Single header: 'User-Agent: '#13#10
- Multiple headers: 'User-Agent: MyNewsClient'#13#10'X-Comment: To be, or
not to be'#13#10

WARNING: For the Add_Header, you MUST add the pound13-Pound10 as shown in
the examples!
If you forget to add the #13#10, then Dialog will error when you send news
saying something like:
Posting article failed: 437 Space before colon in "On Fri, 6 Jan 2017"
header;
*)
procedure AddHeaders(var Message : TStringlist;
const Add_Headers: String
);
var
SeparatorIndex: integer;
s: string;
begin
s:= Message.Text;
// writetolog('***before***'#13#10+s, 7);
SeparatorIndex:= pos(#13#10#13#10, s);
Insert(Add_Headers, s, SeparatorIndex+2);
Message.Text:= s;
// writetolog('***after***'#13#10+s, 7);
end;

function StrMatch(str: String; pattern: String):Boolean;
var
patternSize : Integer;
subStr : String;
compareRes : Integer;
begin
patternSize := Length(pattern);
subStr := Copy(str, 1, patternSize);
compareRes := CompareStr(pattern, subStr);
if (compareRes = 0) then
result := true
else
result := false;
end;

//the xxx2Identity() functions must return an empty string if specified
string is not identified

function From2Identity(from: String): String;

begin
if (StrMatch(from, 'First1 Last1 <ema...@example.com>')) then
result := 'id1'

else if (StrMatch(from, 'First2 Last2 <ema...@example.com>')) then
result := 'id2'

else if (StrMatch(from, 'First3 Last3 <ema...@example.com>')) then
result := 'id3'
else
result := '';

end;

function NewsGroup2Identity(newsgroup: String): String;

// news.software.readers,alt.windows7.general
//SYNTAX: if (StrMatch(newsgroup, 'ng1') or StrMatch(newsgroup, 'ng2'))
then
begin
if (StrMatch (newsgroup, 'news.software.readers') or StrMatch(newsgroup,
'alt.free.newssservers')) then
result := 'id1'
else if StrMatch(newsgroup, 'alt.free.newsservers') then
result := 'id2'
else if StrMatch(newsgroup, 'alt.test') then
result := 'id3'
else
result := '';
end;

function Server2Identity(server: String): String;
begin
if (CompareStr(server, 'aioe_119') = 0) then
result := 'id1'

else if (CompareStr(server, 'mixmin_563') = 0) then
result := 'id2'
else
result := '';
end;

procedure GetIdentities(var message: TStringlist; servername: string;
isEmail: boolean; var FromIdentity: String; var NewsgroupIdentity:
String;
var ServerIdentity: String);
var i : Integer;
begin
FromIdentity := '';
NewsgroupIdentity := '';
ServerIdentity := '';

if (not IsEmail) then
begin
for i := 0 to Message.Count - 1 do
begin
if (strMatch(Message[i], 'From:')) then
fromIdentity := Copy(Message[i], 7, Length(Message[i]) - 6);
if (strMatch(Message[i], 'Newsgroups:')) then
newsgroupIdentity := Copy(Message[i], 13, Length(Message[i]) - 12);
end;
fromIdentity := From2Identity(fromIdentity);
newsgroupIdentity := NewsGroup2Identity(newsgroupIdentity);
serverIdentity := Server2Identity(servername);

// The lines below write to the log file of the format
./40tude/logs/20161231.log

WriteToLog(' fromIdentity = ' + fromIdentity, 7);
WriteToLog(' newsgroupIdentity = ' + newsgroupIdentity, 7);
WriteToLog(' serverIdentity = ' + serverIdentity, 7);

end;
end;

procedure LogHeaders(var Message: TStringlist);
var
i: integer;
s: string;
begin
s:= '';
for i:= 0 to message.count-1 do
begin
if message[i] <> '' then s:= s+message[i]+#13#10
else break;
end;
writetolog(s, 7);

end;

function OnBeforeSendingMessage(var Message : TStringlist;
Servername : string;
IsEmail : boolean
):boolean;

var
ForEmail: boolean;
ForNewsgroup : boolean;
FromIdentity: String;
NewsgroupIdentity: String;
ServerIdentity: String;
Remove_Headers: String;
Add_Headers: String;
begin
//get the identities of the message
GetIdentities(message, servername, isEmail, FromIdentity,
NewsgroupIdentity, ServerIdentity);

// Four program decisions to be made if you wish to change the program
defaults:
// 1. This Boolean sets whether email headers are modified:
// ForEmail := false; //false means don't do email message by
default
// ForEmail := true; //true means do email headers
// 2. This Boolean sets whether newsgroup headers are modified:
// ForNewsgroup := false; //false means don't do newsgroup message by
default
// ForNewsgroup := true; //true means do newsgroup messages
// 3. This string syntax sets the default headers to remove (if not
redefined):
// Remove_Headers := ''; //null means don't remove any header by
default
// Remove_Headers := 'User-Agent: ,Message-ID: '; //string means remove
these headers
// 4. This string syntax sets the default headers to add (if not
redefined):
// Add_Headers := ''; //null means don't add any header by default
// Add_Headers := 'X-Comment: John Doe was here'; //string means add
these header strings
ForEmail := false; //false means don't do email message by default
ForNewsgroup := true; //true means do newsgroup messages
Remove_Headers := ''; //null means don't remove any header by default
Add_Headers := ''; //null means don't add any header by default

{The main decision.

For FromIdentity, comparison must match against string returned by
From2Identity() function.
Same applies to NewsgroupIdentity and ServerIdentity.
Note that identities may be an empty string.

Set Remove_Header to remove header(s).
Set Add_Header to add header(s).
Set ForEmail and/or ForNewsgroup to `true` to add/remove header for
email/newsgroup messages.
}
// Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version:
,Content-Type: ,Content-Transfer-Encoding: ';
if FromIdentity = 'id1' then
begin
ForNewsgroup := true;
Remove_Headers := 'User-Agent: ,Message-ID: ';
Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version:
,Content-Type: ,Content-Transfer-Encoding: ';
Add_Headers := 'X-Comment: JJ is a veritable genius!'#13#10;
end
else if (FromIdentity = 'id2') and (NewsgroupIdentity = 'id1') then
begin
ForNewsgroup := true;
Remove_Headers := 'User-Agent: ,Content-Transfer-Encoding: '
Add_Headers := 'X-Comment: John Doe was here'#13#10;
end
else if (FromIdentity = 'id3') and (ServerIdentity = 'id2') then
begin
ForEmail := true;
ForNewsgroup := true;
Remove_Headers := 'User-Agent: ,Mime-Version: ';
Add_Headers := 'X-Comment: Jane Doe was here'#13#10 +
'X-Greeting: Hello there!'#13#10;
end;

if (IsEmail and ForEmail) or ((not IsEmail) and ForNewsgroup) then
begin
if Remove_Headers <> '' then RemoveHeaders(Message, Remove_Headers);
if Add_Headers <> '' then AddHeaders(Message, Add_Headers);
end;

result := true;
// result := false; //uncomment this line for testing purposes (doesn't
send the message)

[Bernd Rose]

On Fri, 6th Jan 2017 00:57:19 +0000 (UTC), Stijn De Jong wrote:

[OnBeforeSendingMessage script]

> Thanks to JJ <jj4p...@vfemail.net> in this newsgroup and in another
> thread, here is a working script that has been modified to allow multiple
> program conditions.

I don't know, whether this was good for you (and for JJ, FWIW)... Changing
just a few settings will not teach you the necessary skills to adjust the
script (or write others), if need be. Therefore, you'll continue to depend
on JJ or others for explanations, adjustments, and so on. And other people
with /seemingly/ comparable needs may try to use the script, as well. (With
maybe even less knowledge about the inner workings, suitable settings, and
so on.)

Because OnBeforeSendingMessage is probably the most dangerous place for
scripted alterations inside Dialog, publishing ready-to-use script code
carries a considerable burden: It has to be written as failsafe as
possible and should take not only simple usage cases, but also "obscure"
languages, special characters, encoded attachments, header-like text in
the body, use of fixed field substrings inside search/replace terms,...
into account.

But okay, that's all up to you and JJ... ;-)

Bernd

[Stijn De Jong]

On Sat, 7 Jan 2017 00:34:06 +0100, you wrote:

> I don't know, whether this was good for you (and for JJ, FWIW)... Changing
> just a few settings will not teach you the necessary skills to adjust the
> script (or write others), if need be. Therefore, you'll continue to depend
> on JJ or others for explanations, adjustments, and so on. And other people
> with /seemingly/ comparable needs may try to use the script, as well. (With
> maybe even less knowledge about the inner workings, suitable settings, and
> so on.)

JJAddAndRemoveHeadersWhileDoubleCheckingIdentityNgAndServers
http://pastebin.com/sgVDqcqu

I can't possibly NOT understand what you're trying to say, which is that I
would be better served (from a programming standpoint) if I programmed the
code myself, instead of relying on JJ's rewrite of the code.

I think you're thinking exactly like expert programmers would think.
So, from an expert programmer's perspective, your thinking is apropos.

However, from a novice's perspective, the tradeoff of learning Dialog from
scratch to just the job done is a completely different equation.

In fact, I had already given up on Dialog scripting (and said so), BEFORE
JJ posted his modified script on pastebin.

Even after JJ posted his first version, when I tested it, it didn't work.
So JJ fixed it, and when I tested that, it worked but had a problem with
multiple newsgroups.
Then when JJ fixed that, we had a really good set of working code that many
others could use out of the box.

We got something done with Dialog.
And, we improved the code base for Dialog, so others can use it themselves.

So the real tradeoff was between:
a. Giving up on getting Dialog to do anything, or,
b. Using JJ's script (testing it, tweaking it, & re-publishing it).

Having said that, I understand your point of view which is that I was a
cripple before I opened this thread, and I'm still a cripple after JJ
provided the solution.

That's true.
But, I was never going to learn Dialog, so it was a choice really between
getting nothing done versus getting the job done.

From THAT perspective, JJ and I and you worked together.
The bonus is that millions of others (OK, maybe tens? or maybe hundreds?)
will benefit from the latest revised code, which I posted here earlier
today:

JJAddAndRemoveHeadersWhileDoubleCheckingIdentityNgAndServers
http://pastebin.com/sgVDqcqu

> Because OnBeforeSendingMessage is probably the most dangerous place for
> scripted alterations inside Dialog, publishing ready-to-use script code
> carries a considerable burden: It has to be written as failsafe as
> possible and should take not only simple usage cases, but also "obscure"
> languages, special characters, encoded attachments, header-like text in
> the body, use of fixed field substrings inside search/replace terms,...
> into account.

I can't possibly disagree with anything you wrote, but I will point out
that again you're speaking from an expert programmer's perspective, and not
from the perspective of someone just wanting to get Dialog to do something.

In fact, while everything you said is appropriate, very few people actually
do what you said for the Dialog scripts. I know this because I have tried
to get Dialog sripts to work where I ran into EXACTLY the issues you speak
of. In fact, MOST of the Dialog scripts I tried from the official site
failed in my tests.

For example, there were multiple Remove Header scripts, where the one I
chose was the ONLY one that worked for me in the first pass, and, even
then, it didn't have a decent example list of syntax for more than one
header (nor for the list of the 6 allowable headers that could be removed).

So I've personally experienced exactly that which you say shouldn't happen,
and, while I can't disagree with you that it shouldn't happen, the
alternative is to get nothing done on Dialog.

I think posting the script (as I do below), has more value to others than
to hold it tightly to my chest simply because it's not well tested in a
variety of circumstances. If it works for just half the people who use it,
then it would have served its purpose well.

At the very worst, we added dozens of comments that the original code never
had, so we left much less up for others to have to figure out on their own.

Here is the latest code (but the text may get messed up by newsreaders
JJAddAndRemoveHeadersWhileDoubleCheckingIdentityNgAndServers
http://pastebin.com/sgVDqcqu
====================== cut here =================
// JJAddAndRemoveHeadersWhileDoubleCheckingIdentityNgAndServers
// <http://pastebin.com/fcYjL9Vb> newer version
// <http://pastebin.com/2tMYRKM4> older version

// From: JJ <jj4p...@vfemail.net>
// Newsgroups: news.software.readers,alt.windows7.general
// Subject: Re: Request help with 40tude dialog program syntax
// Date: Thu, 5 Jan 2017 23:06:19 +0700
// Message-ID: <7vpx5y09pu98.1sukl5d600sda$.d...@40tude.net>
// Original code for checking identity:
// http://dialog.datalist.org/scripts/CheckingIdentity.html
// Original code for modifying headers:
// http://dialog.datalist.org/scripts/RemoveHeaders.html

// NOTES:
// The StrMatch fct is case sensitive and only matches the first item
found!
// Define a StrContains fct to match multiple either/or newsgroups.
// If you don't set Remove_Header, then none will be removed.
// If you don't set Add_Header, then none will be added.
// For some headers you have to remove them first, then add them back.
// Dialog will error when sending if you add headers sans (CR+LF) syntax!
// WriteLn doesn't seem to do anything.
// CUSTOMIZATION ORDER OF OPERATIONS:
// 1. Set the "ForNewsgroup" Boolean as desired (search for "ForNewsgroup
:")
// 2. Set the "ForEmail" Boolean as desired (search for "ForEmail :")
// 3. Change identity(ies) as desired (search for "from,")
// 4. Change newsgroup(s) as desired (search for "newsgroup,")
// 5. Change server(s) as desired (search for "server,")
// 6. Change remove header(s) as desired (search for "Remove_Headers :")
// 7. Change add header(s) as desired (search for "Add_Headers :")
// End of initial comments.

// Customize your identity below for personal or business use as needed.
// The string match is probably case sensitive!

function From2Identity(from: String): String;
begin

if (StrMatch(from, 'First1 Last1 <ema...@domain.net>')) then
result := 'id1'
else if (StrMatch(from, 'First2 Last2 <ema...@domain.net>')) then
result := 'id2'
else if (StrMatch(from, 'First3 Last3 <ema...@domain.net>')) then

result := 'id3'
else
result := '';
end;

// WARNING: The StrMatch function can only check the first listed
newsgroup!
// Define a StrContains function if you want to match either/or newsgroups!
// function StrContains(const Str: string; const Pattern: string): boolean;
// begin
// result:= pos(Pattern, Str) > 0;
// end;
// EXAMPLE: This StrMatch will only match the first newsgroup in the
outgoing Newsgroup header!
// if (StrMatch (newsgroup, 'news.software.readers') or StrMatch(newsgroup,
'alt.free.newsservers')) then result := 'id1'
// If defined, this StrContains will match either newsgroup or both in the
outgoing Newsgroup header:
// if StrContains(newsgroup, 'news.software.readers') or
StrContains(newsgroup, 'alt.free.newssservers') then result := 'id1'
// End of comments

function StrContains(const Str: string; const Pattern: string): boolean;
begin
result:= pos(Pattern, Str) > 0;

end;

function NewsGroup2Identity(newsgroup: String): String;

begin
if StrContains(newsgroup, 'ng1') or StrContains(newsgroup, 'ng2') then
result := 'id1'
else if StrMatch(newsgroup, 'ng3') then
result := 'id2'
else if StrMatch(newsgroup, 'ng4') then

result := 'id3'
else
result := '';
end;

// The name of the server is what Dialog lists in the "Available Servers"
UI.

function Server2Identity(server: String): String;
begin
if (CompareStr(server, 'aioe_119') = 0) then
result := 'id1'

else if (CompareStr(server, 'albasani_563') = 0) then

// The default log file is C:/Program Files/40tude/logs/YYYYMMDD.log

ForNewsgroup := true; //true means do newsgroup messages
Remove_Headers := ''; //null means don't remove any header by default

Remove_Headers := 'User-Agent: ,Message-ID: ,Date: '; //remove these
outgoing headers

Add_Headers := ''; //null means don't add any header by default

{The main decision.

For FromIdentity, comparison must match against string returned by
From2Identity() function.
Same applies to NewsgroupIdentity and ServerIdentity.
Note that identities may be an empty string.

Set Remove_Header to remove header(s).
Set Add_Header to add header(s).
Set ForEmail and/or ForNewsgroup to `true` to add/remove header for
email/newsgroup messages.
}

// EXAMPLES:

// Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version:
,Content-Type: ,Content-Transfer-Encoding: ';

// Add_Headers := 'Organization: none whatsoever'#13#10;
// Add_Headers := 'X-Comment: Jane Doe was here'#13#10 + 'X-Greeting: Hello
there!'#13#10;
// Note that some headers cannot be set (for example, NNTP Posting Host or
Newsgroups)
//

if FromIdentity = 'id1' then
begin
ForNewsgroup := true;
Remove_Headers := 'User-Agent: ,Message-ID: ';

// Add_Headers := 'X-Comment: Jane Doe was here'#13#10 + 'X-Greeting:
Hello there!'#13#10;

end
else if (FromIdentity = 'id2') and (NewsgroupIdentity = 'id2') then
begin
ForNewsgroup := true;
// Remove_Headers := 'User-Agent: ,Content-Transfer-Encoding: '
Add_Headers := 'Organization: none whatsoever'#13#10;
end

else if (FromIdentity = 'id3') and (ServerIdentity = 'id1') then
begin // id3

ForEmail := true;
ForNewsgroup := true;
Remove_Headers := 'User-Agent: ,Mime-Version: ';

Add_Headers := 'User-Agent: My user agent is 40Tude Dialog on Windows
10'#13#10;
// WriteLn(Remove_Headers); // This WriteLn command does not seem to do
anything.
// WriteLn(Add_Headers); // This WriteLn command does not seem to do
anything.
end; // id3

if (IsEmail and ForEmail) or ((not IsEmail) and ForNewsgroup) then
begin
if Remove_Headers <> '' then RemoveHeaders(Message, Remove_Headers);
if Add_Headers <> '' then AddHeaders(Message, Add_Headers);
end;

result := true;
// result := false; //uncomment this line for testing purposes (doesn't
send the message)
end;
// ----------------------------------------------------------------------
begin
end.

====================== cut here =================

[Bernd Rose]

On Sat, 7th Jan 2017 03:29:15 +0000 (UTC), Stijn De Jong wrote:

[OnBeforeSendingMessage script]

>> I don't know, whether this was good for you (and for JJ, FWIW)... Changing
>> just a few settings will not teach you the necessary skills to adjust the
>> script (or write others), if need be. Therefore, you'll continue to depend
>> on JJ or others for explanations, adjustments, and so on. And other people
>> with /seemingly/ comparable needs may try to use the script, as well. (With
>> maybe even less knowledge about the inner workings, suitable settings, and
>> so on.)

[...]

> I can't possibly NOT understand what you're trying to say, which is that I
> would be better served (from a programming standpoint) if I programmed the
> code myself, instead of relying on JJ's rewrite of the code.

[...]

> In fact, I had already given up on Dialog scripting (and said so), BEFORE
> JJ posted his modified script on pastebin.

If that's the case, then getting the script written is of course the best
outcome from *your* POV. Whether JJ will have the time and strength for
continuous error correction and support for you /and/ others (who might
want to use such a script, as well), has to be seen.

> We got something done with Dialog.
> And, we improved the code base for Dialog, so others can use it themselves.

When Dialog script development was at his heights, at least 5 people who
were firm with that subject tested and fine-tuned the scripts, until they
"released" them in script libraries. (Please note, there hasn't been an
/official/ script repository for nearly 10 years. All available download
sites are personal-preference sites with varying degrees of code-check.)
The script development mainly occurred inside the German Dialog newsgroup.
Nowadays, there's rarely a post on that subject. Most script developers
moved on, at least one of the most productive ones is dead by now. :-(

While releasing new scripts in test-state here in this group may seem to
be a step forward, chances of significant code-check and testing are
marginal. (I, for instance, currently just don't have the time for such
matters.) With most scripts, publishing unpolished versions isn't too
problematic. They may lead to problems on local installations. And that's
about it. With OnBeforeSendingMessage scripts, OTOH, the results may -
at worst - crash some receiving servers, if the message gets malformatted.
And are you /sure/ that the last version will work with any kind of mail
message, including headers in need of encoding, duplicated "headers" in
the body, encoded attachments, encrypted mails, and so on?

And some parts of the code /need/ to be polished for usability. Published
scripts shouldn't carry user-specific settings deep down in the code, like
your aioe/albasani-test. If you browse through well-written scripts, you'll
notice a "settings const section" at the top or (sometimes) even the usage
of an external settings file. Coming to this: Why only two possible mail
servers in a script published for common use? Why not start with a list
(or better: an array/dictionary of server/identity pairs), to begin with?
The same goes for newsgroups...

> So the real tradeoff was between:
> a. Giving up on getting Dialog to do anything, or,
> b. Using JJ's script (testing it, tweaking it, & re-publishing it).

*Your* tradeoff.

Bernd

[Stijn De Jong]

On Sat, 7 Jan 2017 09:47:01 +0100, you wrote:

>> In fact, I had already given up on Dialog scripting (and said so), BEFORE
>> JJ posted his modified script on pastebin.
>
> If that's the case, then getting the script written is of course the best
> outcome from *your* POV. Whether JJ will have the time and strength for
> continuous error correction and support for you /and/ others (who might
> want to use such a script, as well), has to be seen.

Yet again, there is absolutely nothing you say that I don't already agree
with, in that
a. I first tried to combine scripts using algorithmic intuition
b. But I was soundly defeated by the diabolical Dialog syntax
c. Luckily, where I failed, both you and JJ came to the rescue of all

The result is that we all (including you) now have a working script that is
far more powerful than any script like it on the Dialog code library.

I based that assessment of this script being worlds more powerful than any
like it on the Dialog code library after looking at EVERY similar script in
the code library (none of which do what this script does and only some of
which do some of what this script does).

Since this is easily and provably a far more powerful and versatile script
of any like it on the Dialog page, it has value over and above that of the
similar scripts from which it was originally composed of.

This is all as it should be, adding value at every turn.

>> We got something done with Dialog.
>> And, we improved the code base for Dialog, so others can use it themselves.
>
> When Dialog script development was at his heights, at least 5 people who
> were firm with that subject tested and fine-tuned the scripts, until they
> "released" them in script libraries. (Please note, there hasn't been an
> /official/ script repository for nearly 10 years. All available download
> sites are personal-preference sites with varying degrees of code-check.)
> The script development mainly occurred inside the German Dialog newsgroup.
> Nowadays, there's rarely a post on that subject. Most script developers
> moved on, at least one of the most productive ones is dead by now. :-(

I understand what you're saying, even as I didn't know the details.
It was clear that Dialog was not being actively developed, simply by a
search of this newsgroup prior to my posting (in Google's deja vu
interface).

The script repository where the two original components came from is:
http://dialog.datalist.org/scripts/script_library.html

Another script library (although tiny in comparison) is apparently:
http://www.elbiah.de/tools/dialog/

This seems to have a wider geographic scope:
http://www.wolfgang-bauer.at/4td_faq/links.html

Here is another seemingly incomprehensible 40tude dialog library:
http://larrainelaaril.ga/40tude-HTML-573.php
NOTE: There are many others like that one which are so incomprehensible as
to consist of some kind of joke or spam or something:
http://bonitapruettaj8oh22.ga/40tude-HTML-4945.php
http://virgenfeatherm7d2s7.gq/40tude-HTML-4975.php
http://johnsontotmanexj5yf.ga/40tude-HTML-4761.php

> While releasing new scripts in test-state here in this group may seem to
> be a step forward, chances of significant code-check and testing are
> marginal. (I, for instance, currently just don't have the time for such
> matters.) With most scripts, publishing unpolished versions isn't too
> problematic. They may lead to problems on local installations. And that's
> about it. With OnBeforeSendingMessage scripts, OTOH, the results may -
> at worst - crash some receiving servers, if the message gets malformatted.
> And are you /sure/ that the last version will work with any kind of mail
> message, including headers in need of encoding, duplicated "headers" in
> the body, encoded attachments, encrypted mails, and so on?

You bring up perfectly valid points, which can be summarized by the ancient
Justinian law code of "Caveat Emptor".

There is one and only one defense when receiving free stuff and that is
that you take a calculated risk whenever you receive free stuff.

Luckily, this free stuff is relatively innocuous, but caveat emptor still
holds (e.g., we could have inserted a clever virus into the code).

BTW, this ng is "news.software.readers", so, it is just about the best
place on the planet that we should strive to discuss such code, is it not?

> And some parts of the code /need/ to be polished for usability. Published
> scripts shouldn't carry user-specific settings deep down in the code, like
> your aioe/albasani-test.

Once again, I completely agree with you that user-specific settings should
be in the header and not buried deeply down in the code.

The funny thing is that we didn't change the code. We merely included the
"as official as it gets" official Dialog scripts into our code. So, what
you're really complaining about is that the official Dialog library of code
contains scripts which have user-specific settings buried deep down in the
code.

I can't disagree with you in the least.
a. The original code in the Dialog library had buried settings, hence,
b. Our use of that original code didn't change the depth of those settings.

We completely agree on the problem, but the "blame" (if we are to assign
blame) must go to the original developer, and to the people who published
that original code in the Dialog library.

I'm not going to blame them; I'm going to THANK them, so, I'm just
responding to your perfectly valid code-construct comment that user
settings shouldn't be buried deep down in the code.

Hence, I agree with you.
But I don't have the technical ability to fix that problem.
And that problem existed in the original code anyway.

> If you browse through well-written scripts, you'll
> notice a "settings const section" at the top or (sometimes) even the usage
> of an external settings file.

You are completely correct that 'well written' scripts would generally put
user-defined variables in the header section.

What that means is that the scripts we pulled off the Dialog Library
weren't well written (and I knew that because they didn't work anyway).

They worked even less when I put two of them together.
And that was my *starting* point!

> Coming to this: Why only two possible mail
> servers in a script published for common use? Why not start with a list
> (or better: an array/dictionary of server/identity pairs), to begin with?

Actually, what you're complaining about here is a classic case of "no good
deed ever goes unpunished". :)

The original code is here:
http://dialog.datalist.org/scripts/CheckingIdentity.html

What do they use for identity, server and newsgroups?

The problem with *that* script is that nobody knows what to put for the
"server1" name. I guessed "nntp.aioe.org" but that didn't work. I then
guessed "aioe" but that didn't work. I finally realized that the server
name is whatever you have listed in
"Dialog: Settings > Servers,Identities,Signatures > Newsservers >
Available Servers" column.

So what did I do?
I put a comment in the code that explains that.

So, in effect, the code that I published is *better* than the code that was
in the script library, in that a real-life real-world example was given.

It was, however, just an example.

So in this instance, I can only say that you wouldn't have complained about
this if I had brought it back to 'server1', but that you're only
complaining because I called it 'aioe_119' to provide an example for the
user.

No good deed ever goes unpunished, so I take your complaint in stride. :)

> The same goes for newsgroups...

In the case of newsgroups, I agree with you more than for the case of
servers, only because newsgroups don't require an example, whereas servers
do.

Remember the original code simply said "ng1" and "ng2", whereas the code
here at least provides syntax-correct examples (bearing in mind that Dialog
syntax is utterly diabolical).

>> So the real tradeoff was between:
>> a. Giving up on getting Dialog to do anything, or,
>> b. Using JJ's script (testing it, tweaking it, & re-publishing it).
>
> *Your* tradeoff.

If you like, I can further genericize the script, by removing all the
examples.

Would that be better?

[Bernd Rose]

On Sat, 7th Jan 2017 21:14:57 +0000 (UTC), Stijn De Jong wrote:

[OnBeforeSendingMessage script]

> If you like, I can further genericize the script

You can do, whatever pleases /you/ the most. If I ever needed such a script,
I'd write it, myself. (Adjusted to my needs.)

Btw., it seems to me, that you request help for hiding, that you wish to
use a non-approved Mail&News client at work. That's no good idea, out of
several reasons. Removing/altering headers like Content-Transfer-Encoding
or Mime-Version is no good idea, either, because you end up with messages,
that violate standards. Better set charset and attachment settings in the
Dialog options appropriately; and maybe take a pass on sending attachments
altogether.

If you just wish (and are permitted) to use one Dialog installation for
private and business matters, but want to see them separately and want
to be able to distinguish sent messages by header, you could do this
without altering outgoing messages by script.

You can define different identities and adjust them (as posting-identities)
to newsgroups. They can have the same or different internal settings, only
mail-in and/or mail-out, or only news settings. (Or no settings, at all.)
You can define X-headers and adjust them to identities. You can pre-define
some "normal" headers specific to identities, as well. You can use different
Message-Id creation methods and X-Faces for each identity, if you like.

You can create different categories for different usage scenarios (like
business or private use) and sort only the newsgroups and mailboxes into
them, that are needed. This way, you even could create a different
identity for each subscribed group from each server. Although this would
clutter the "Subscribed" category with Inboxes (which can be sorted to the
bottom, though), the specific categories could be organized in a clean
way with only one Inbox (or none at all).

You can retrieve news for one group from different servers. Although those
groups would be handled internally by Dialog, as if they were different
groups, you can set up, that each message shall be copied on reception
from one group to another (and vice versa) via Scoring&Actions (= "virtual
groups").

Bernd

[Stijn De Jong]

On Sat, 7 Jan 2017 22:57:01 +0100, you wrote:

> [OnBeforeSendingMessage script]
>> If you like, I can further genericize the script

I'm all about giving back to the newsgroup so here is the result with both
your comments and those of JJ taken into account as best I could do so.
JJAddAndRemoveHeadersWhileCheckingNewsgroupsIdentitiesAndServers
version 1.00 January 8th 2017 "http://pastebin.com/6Md5m67X

> You can do, whatever pleases /you/ the most. If I ever needed such a script,
> I'd write it, myself. (Adjusted to my needs.)

Again, almost nothing you say can I disagree with, in that expert
programmers like you are come to the problem armed with different tools
than people like I come to the problem with.

It's like you come to the battle armed with a Panzer tank, while all I have
is a molotov cocktail. We both work with what we have as our current tools.

I could NEVER write a 40Tude-Dialog script (and I know it), so, people like
me scavenge what we can from existing scripts.

That's why I'm adamant that I give back to the team, so that other
scavengers can make use of what I've compiled from the efforts of the
original coders, you, and JJ.

> Btw., it seems to me, that you request help for hiding, that you wish to
> use a non-approved Mail&News client at work. That's no good idea, out of
> several reasons. Removing/altering headers like Content-Transfer-Encoding
> or Mime-Version is no good idea, either, because you end up with messages,
> that violate standards. Better set charset and attachment settings in the
> Dialog options appropriately; and maybe take a pass on sending attachments
> altogether.

I have never seen any negative repercussions from getting rid of the
seemingly extraneous and wasteful and not needed additional headers that
Dialog adds to the header.

I only use a English and the 26 letters of the US alphabet though, but for
what I type, NEVER have those wasteful header lines ever been of any import
whatsoever.

Why even have them if they do absolutely nothing useful?

> If you just wish (and are permitted) to use one Dialog installation for
> private and business matters, but want to see them separately and want
> to be able to distinguish sent messages by header, you could do this
> without altering outgoing messages by script.

Dialog is not an approved newsreader.
I'm told that's because it's unsupported.
The approved newsreader is a Microsoft product.

In addition, the only sanctioned newsgroups are Microsoft newsgroups, and
specific product newsgroups such as Adobe, Corel, etc.

If I could obfuscate the Newsgroup line, I would, but I can't so I'm only
trying to minimize the existent transgressions.

> You can define different identities and adjust them (as posting-identities)
> to newsgroups. They can have the same or different internal settings, only
> mail-in and/or mail-out, or only news settings. (Or no settings, at all.)

I don't know if I understand your suggestion as I already set, for each
newsgroup, the default posting identity, but as you are well aware, you
can't set the default server UNLESS that's the server you use to read the
newsgroup. Since I use a different server to read the newsgroup than to
post to the newsgroup (for good reasons), then Dialog is ALWATYS defaulting
to the wrong newsgroup.

Hence, the outgoing server has to be set on every single post.
This fact that I must set the outgoing news server on every single post is
a usability deficiency in Dialog where I have concluded there is nothing I
can do about it because it is a basic design flaw.

The makers of Dialog never thought anyone would read from one server and
post to another, I guess. If Dialog simply had a "default outgoing server"
setting for each newsgroup, that problem would be solved instantly.

> You can define X-headers and adjust them to identities. You can pre-define
> some "normal" headers specific to identities, as well. You can use different
> Message-Id creation methods and X-Faces for each identity, if you like.

Yes. But you can't set the default outgoing server. Sigh.

> You can create different categories for different usage scenarios (like
> business or private use) and sort only the newsgroups and mailboxes into
> them, that are needed. This way, you even could create a different
> identity for each subscribed group from each server.

Yes. But you can't set a default outgoing server.
It's just impossible.
But it should have been as simple as "Default outgoing server = server1".

> Although this would
> clutter the "Subscribed" category with Inboxes (which can be sorted to the
> bottom, though), the specific categories could be organized in a clean
> way with only one Inbox (or none at all).

That clutter of inboxes drives me nuts.
You can't delete any of them without deleting the identity.
It's just a mess but I simply ignore that "Folders" tab because of all the
unnecessary Dialog-generated clutter.

Instead, I use the "Categories" tabs, one for business, and the other for
personal. I use different incoming servers for business than for personal,
and then again, a different set of outgoing servers for business and
personal (for a total of four servers).

There is a reason for this, but those reasons are specific to the company
that I work for and for the servers that I choose to use.

Suffice to say that a flaw in Dialog's implementation is that Dialog
doesn't have any concept of a "default outgoing server".

Dialog simply naively assumes that the outgoing server is the same as the
incoming server.

That's one setting that I'd love to add to dialog:
Default outgoing server = server1

> You can retrieve news for one group from different servers. Although those
> groups would be handled internally by Dialog, as if they were different
> groups, you can set up, that each message shall be copied on reception
> from one group to another (and vice versa) via Scoring&Actions (= "virtual
> groups").

Hmm. Maybe that method you just described can get around the fundamental
design flaw in Dialog that the outgoing server always defaults to the
incoming server?

[Bernd Rose]

On Sun, 8th Jan 2017 00:51:25 +0000 (UTC), Stijn De Jong wrote:

>> Removing/altering headers like Content-Transfer-Encoding
>> or Mime-Version is no good idea, either, because you end up with messages,
>> that violate standards. Better set charset and attachment settings in the
>> Dialog options appropriately; and maybe take a pass on sending attachments
>> altogether.
>
> I have never seen any negative repercussions from getting rid of the
> seemingly extraneous and wasteful and not needed additional headers that
> Dialog adds to the header.
>
> I only use a English and the 26 letters of the US alphabet though, but for
> what I type, NEVER have those wasteful header lines ever been of any import
> whatsoever.
>
> Why even have them if they do absolutely nothing useful?

Content-Transfer-Encoding and Mime-Version are /not/ "extraneous and
wasteful and not needed additional headers" (as you put it). Even if
/you/ never used a character outside 7-bit US-ASCII, you will likely
have sometimes extended characters somewhere in the text you keep as
a citation from others. If you unconditionally remove the encoding
information, you endanger any transmission and any receiving server
and client, if it isn't programmed to react robust on mal-formatted
messages. With robust transmission it may happen, OTOH, that the
message will be re-encoded (again) during transmission, what will
lead to wrong character interpretation on the receiving clients.

Mime-Version is a /required/ header in MIME-conform messages. So, as
long as you want your messages to not be mal-formatted, you /need/ to
leave that header untouched. Sending messages, that are /not/ conform
to MIME is *not* recommended and may need additional adjustments for
header and or body section. (And still may lead to transmission and
display problems.)

> The makers of Dialog

Only one: Marcus Mönnig. Some people supported with testing, translation
and the like. But only one person wrote all the code.

> never thought anyone would read from one server and
> post to another, I guess. If Dialog simply had a "default outgoing server"
> setting for each newsgroup, that problem would be solved instantly.

Dialog works just fine in this regard. The same newsgroup from different
servers is internally treated as different newsgroups. And the posting
server can be changed manually, if need really be (temporary server fault
and the like). The program was never meant to be used with setups like
the ones you have in mind.

> That clutter of inboxes drives me nuts.

In your case I'd probably use 3 identities:
* Private with complete private settings
* Business with complete business settings
* Empty with *no* settings at all

"Settings" refers to Message headers, Attribution, POP, and SMTP.

I'd set identity "Empty" as Posting Identity for any group that needs the
server to be changed on posting. This way, it is impossible to send a
posting without first selecting an identity with correct posting settings
(like FROM address). While selecting another identity before posting it
is easy to remember to change the outgoing server as well. Although this
means 2x2 clicks more for every message, it should be manageable, as long
as you don't write hundreds of postings a day. (And this method does not
carry the danger, which an OnBeforeSendingMessage script creates: The
creation of a mal-formatted/corrupted message.)

In such a setup, it isn't necessary to "subscribe" to the newsgroups
from the server(s) only used for posting. It is enough to have set up
all necessary connection parameters and have retrieved the group list.

Bernd

[Michael Bäuerle]

Bernd Rose wrote:
> On Sun, 8th Jan 2017 00:51:25 +0000 (UTC), Stijn De Jong wrote:

> > Bernd Rose wrote:
> > >
> > > Removing/altering headers like Content-Transfer-Encoding
> > > or Mime-Version is no good idea, either, because you end up with messages,
> > > that violate standards. Better set charset and attachment settings in the
> > > Dialog options appropriately; and maybe take a pass on sending attachments
> > > altogether.
> >
> > I have never seen any negative repercussions from getting rid of the
> > seemingly extraneous and wasteful and not needed additional headers that
> > Dialog adds to the header.
> >
> > I only use a English and the 26 letters of the US alphabet though, but for
> > what I type, NEVER have those wasteful header lines ever been of any import
> > whatsoever.
> >
> > Why even have them if they do absolutely nothing useful?
>
> Content-Transfer-Encoding and Mime-Version are /not/ "extraneous and
> wasteful and not needed additional headers" (as you put it). Even if
> /you/ never used a character outside 7-bit US-ASCII, you will likely
> have sometimes extended characters somewhere in the text you keep as
> a citation from others. If you unconditionally remove the encoding
> information, you endanger any transmission and any receiving server
> and client, if it isn't programmed to react robust on mal-formatted
> messages. With robust transmission it may happen, OTOH, that the
> message will be re-encoded (again) during transmission, what will
> lead to wrong character interpretation on the receiving clients.

The only exceptions are MIME Headerfields containing the default values:

Content-Type: text/plain; charset=US-ASCII"
Content-Transfer-Encoding: 7bit

"text/plain" "US-ASCII" and "7bit" are defined to be used by a MIME
compliant receiver if nothing is specified.

[Stijn De Jong]

On Sun, 8 Jan 2017 12:22:25 -0000, you wrote:

> Content-Type: text/plain; charset=US-ASCII"
> Content-Transfer-Encoding: 7bit
>
> "text/plain" "US-ASCII" and "7bit" are defined to be used by a MIME
> compliant receiver if nothing is specified.

I've seen these in headers with and without the doublequotes.
And I've seen the us-ascii as "US-ASCII", us-ascii, and "us-ascii".

Does that variable syntax matter?

[Stijn De Jong]

On Sun, 8 Jan 2017 12:59:35 +0100, you wrote:

> Content-Transfer-Encoding and Mime-Version are /not/ "extraneous and
> wasteful and not needed additional headers" (as you put it).

I've never once needed them.
So, I guess they just don't matter for 99.99% of text-only posts, which all
mine are. So at the very least, I don't need them.

> Even if
> /you/ never used a character outside 7-bit US-ASCII, you will likely
> have sometimes extended characters somewhere in the text you keep as
> a citation from others.

I'm sure it happens but 99.999999% of my posts don't have extraneous funky
characters unless someone else introduces them, which never seems to be a
problem for me.

> If you unconditionally remove the encoding
> information, you endanger any transmission and any receiving server
> and client, if it isn't programmed to react robust on mal-formatted
> messages.

I've never had a problem removing almost all those extraneous headers.
Nobody has complained either.

> With robust transmission it may happen, OTOH, that the
> message will be re-encoded (again) during transmission, what will
> lead to wrong character interpretation on the receiving clients.

I understand, but the simple fact is that I remove all those headers as a
matter of habit, and nothing bad happens (despite all the warnings).

It's like running with scissors or going outside without a hat.

> Mime-Version is a /required/ header in MIME-conform messages. So, as
> long as you want your messages to not be mal-formatted, you /need/ to
> leave that header untouched.

I don't think that statement is anywhere in the least even close to
correct, at least for my posts. There's no MIME header in this post and I'm
sure you can see it just fine.

> Sending messages, that are /not/ conform
> to MIME is *not* recommended and may need additional adjustments for
> header and or body section. (And still may lead to transmission and
> display problems.)

I understand what you're saying but you have to realize that it's really
simple stuff we're dealing with here.

It's just English text.
On purpose, I refrain from using any special characters.
KISS.

Why would regular text need a MIME header?

It makes no difference in this message (and every single message I've ever
sent) whether or not there is a MIME header.

>> The makers of Dialog
>
> Only one: Marcus M?nnig. Some people supported with testing, translation

> and the like. But only one person wrote all the code.

IMHO, Marcus deserves a medal!

> Dialog works just fine in this regard. The same newsgroup from different
> servers is internally treated as different newsgroups. And the posting
> server can be changed manually, if need really be (temporary server fault
> and the like). The program was never meant to be used with setups like
> the ones you have in mind.

I understood the logic that says the incoming server must be the outgoing
server, but, the simple fact is that this is not always going to be the
case.

It seems strange that Dialog is so powerful, and yet, for something as
simple and common as setting the correct outgoing server by default, Dialog
is a cripple.

It's just odd.

For a tool that's essentially unlimited in its possibilities, why limit
something so common as setting the outgoing server by default to the
correct server?

[Stijn De Jong]

On Sat, 7 Jan 2017 22:57:01 +0100, you wrote:

> Although this would
> clutter the "Subscribed" category with Inboxes (which can be sorted to the
> bottom, though), the specific categories could be organized in a clean
> way with only one Inbox (or none at all).

I just figured out how to hide all those inboxes that clutter up dialog!

// ScriptMakeInboxInvisible
// From 40tude Dialog Wiki
// Thanks go out to Eggs Zachtly for this script.
// The purpose is to make Inboxes invisible for those who don't use them.
// 40tudeDialog: Settings > Scripting > Scripting > OnNewsgroupListPane
// http://dialog.datalist.org/faq/frequently_asked_questions.html
//
// Q: I've set up a few identities, and Dialog keeps adding an inbox
// for each of them, I don't need them, so how do I get rid of them?
//
// A: You can't, but you can make them invisible with this script.
// http://dialog.datalist.org/scripts/ScriptMakeInboxInvisible.html
// Another workaround would be to move them to the bottom of the list
// and then adding a seperator so they don't distract you.

program OnNewsgroupListPaint;
function
OnNewsgroupListPaint(PaintString:widestring;ColumnIndex:integer):widestring;
begin
paintstring := StringReplace(paintstring,'Inbox','',[rfReplaceAll])

result:=paintstring;

[Dave Sparks]

On Mon, 09 Jan 2017 02:01:35 +0000, Stijn De Jong wrote:

...

> It makes no difference in this message (and every single message I've
> ever sent) whether or not there is a MIME header.

...

>> Only one: Marcus M?nnig. Some people supported with testing,
>> translation and the like. But only one person wrote all the code.

Your failure to use a suitable encoding, and record it using the required
MIME headers, has resulted in mangling someone's name. Discourteous, to
say the least.

[Michael Bäuerle]

Stijn De Jong wrote:
> On Sun, 8 Jan 2017 12:22:25 -0000, you wrote:
> >
> > Content-Type: text/plain; charset=US-ASCII"

^ ^
Sorry, this was a typo. There must be double quotes at beginning and end
or no double quotes at all.

> > Content-Transfer-Encoding: 7bit
> >
> > "text/plain" "US-ASCII" and "7bit" are defined to be used by a MIME
> > compliant receiver if nothing is specified.
>
> I've seen these in headers with and without the doublequotes.
> And I've seen the us-ascii as "US-ASCII", us-ascii, and "us-ascii".
>
> Does that variable syntax matter?

In general the headerfield names are case insensitive (RFC 5322 and
RFC 5234).

For the MIME headerfields above the media types and transfer encodings
are case insensitive too (RFC 2045).

In general for parameters the parameter name is case insensitive and
the value is case sensitive.
For the "charset" parameter above RFC 2046 defines an exception so that
the value of this parameter is case insensitive [1]. This means that
"US-ASCII" and "us-ascii" have the same meaning.

For the "charset" parameter above the value is allowed to contain
"quoted-string" values. RFC 2045 [2] says:
|
| Note that the value of a quoted string parameter does not include the
| quotes. That is, the quotation marks in a quoted-string are not a
| part of the value of the parameter, but are merely used to delimit
| that parameter value. [...]

Therefore:

charset="US-ASCII"
charset=US-ASCII

are both allowed and have the same meaning. In general the quoted string
variant is required only for values that contain forbidden characters
as defined in [2].


_________________
[1] <https://tools.ietf.org/html/rfc2046#section-4.1.2>
[2] <https://tools.ietf.org/html/rfc2045#section-5.1>

[Stijn De Jong]

On Mon, 9 Jan 2017 11:15:01 +0000, you wrote:

> Your failure to use a suitable encoding, and record it using the required
> MIME headers, has resulted in mangling someone's name. Discourteous, to
> say the least.

Testing what I see by including the Marcus M"?"nnig below
(where the second letter of Marcus' last name above uses a non-standard
character.)

[Stijn De Jong]

On Mon, 9 Jan 2017 13:35:59 +0100, you wrote:

>> Does that variable syntax matter?
>
> In general the headerfield names are case insensitive (RFC 5322 and
> RFC 5234).

Thank you for clarifying that the quote delimiters are only needed if
forbidden characters are used, and that the headers are, in general,
case insensitive.

So that we can leverage that information, I have added the following
comments to the latest code set where others can benefit from your
information.

// CASE-SENSITIVITY:
// Michael B?uerle advised in news.software.readers on Jan 9, 2017 that
// headerfield names are case insensitive as per RFC 5322 & RFC 5234.
// MIME headerfields above the media types & transfer encodings are not
// case sensitive as per RFC 2045.
// In general, parameter names are case insensitive while the
// value of the parameter is case sensitive.
// The "charset" parameter defines an exception as per RFC 2046 so
// that the value of the charset parameter is case insensitive
// This means that both "US-ASCII" & "us-ascii" are the same meaning.
// The "charset" parameter is allowed to contain "quoted-string" values
// as per RFC 2045 where the value of a quoted string parameter does not
// include the quote delimiters.
// Therefore: charset="US-ASCII" is the same as charset=US-ASCII
// Quote delimiters are used for values containing forbidden characters.
// End of initial comments

[Stijn De Jong]

On Mon, 9 Jan 2017 16:43:49 +0000 (UTC), you wrote:

> Testing what I see by including the Marcus M"?"nnig below
> (where the second letter of Marcus' last name above uses a non-standard
> character.)

I see what you mean on these two surnames:

Marcus M?nnig
Michael B?uerle

Which server munged the name incorrectly?
The sending server or the receiving server?

Either way, to get around this accidental error, I need to add at least
one header to prevent the non-standard "o" in Marcus' last name and the
non-standard "a" in Michael's last name from being turned into a question
mark by that server.

There are a billion RFCs, but Googling, it seems I need to change the
headerline for the default character set to either:
a. ISO/IEC 8859-15, or
b. ISO-8859-1, or
b. UTF-8

That would mean I need to add one of these:
a. Add_Headers := 'Content-Type: text/plain; charset=iso-8859-1'#13#10;
b. Add_Headers := 'Content-Type: text/plain; charset="iso-8859-15"'#13#10;
c. Add_Headers := 'Content-Type: text/plain; charset=utf-8'#13#10;

Again, I ran into a billion RFCs googling this, but according to RFC3629,
the standard character set of US-ASCII is a subset of UTF-8, so I'll try
UTF-8 in the header of this message because that seems to be the
"universal" character set overall.

TESTCASE:
1. Marcus M?nnig <== the "o" in the surname is the test
2. Michael B?uerle <== the "a" in the surname is the test

Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version: ,Content-Type: ,Content-Transfer-Encoding: ';

Add_Headers := 'Content-Type: text/plain; charset=utf-8'#13#10 + 'X-Comment: No MIME, no encoding headerline; charset is set to utf-8'#13#10;

[Michael Bäuerle]

Stijn De Jong wrote:
>
> [...]

> TESTCASE:
> 1. Marcus M?nnig <== the "o" in the surname is the test
> 2. Michael B?uerle <== the "a" in the surname is the test

Still question marks. Should be:

Marcus Mönnig
Michael Bäuerle

Look at this article for the variant that use ISO 8859-1 encoding.

[Michael Bäuerle]

Stijn De Jong wrote:
>
> [...]

> // CASE-SENSITIVITY:
> // Michael B?uerle advised in news.software.readers on [...]
^
If you have no option to correctly encode my name, please use "ae" as
ASCII transcription for the umlaut "ä".

The complete list of ASCII transcriptions for non-ASCII special
characters used in german language looks like this:

Ä => Ae
ä => ae
Ö => Oe
ö => oe
Ü => Ue
ü => ue
ẞ => SS (sometimes "SZ" if ambigous otherwise, never "B"!)
ß => ss (sometimes "sz" if ambigous otherwise)

[Stijn De Jong]

On Mon, 9 Jan 2017 18:50:56 +0100, you wrote:

> Reply-To: Michael B?uerle <michael....@gmx.net>
> Mime-Version: 1.0
> Content-Type: text/plain; charset=UTF-8; format=fixed
> Content-Transfer-Encoding: 8bit

Looking at your headerlines, I see this:
Reply-To: Michael B?uerle <michael....@gmx.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit

So, while I don't understand why just the one content-type line won't work,
I'll try the first 2 of those 3 lines to see if the name works:
Marcus M?nnig
Michael B?uerle

Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version: ,Content-Type: ,Content-Transfer-Encoding: ';

Add_Headers := 'Mime-Version: 1.0'#13#10 + 'Content-Type: text/plain; charset=ISO-8859-1; format=fixed'#13#10 + 'X-Comment: mime + charset but no encoding'#13#10;

[Bernd Rose]

On Mon, 9th Jan 2017 17:18:19 +0000 (UTC), Stijn De Jong wrote:

[Wrong encoding/charset]

> I see what you mean on these two surnames:
>
> Marcus M?nnig
> Michael B?uerle
>
> Which server munged the name incorrectly?
> The sending server or the receiving server?

The error is solely yours!! /You/ deliberately removed the correct charset
and encoding declaration and therefore (implicitly for any MIME compliant
program) wrongly declared 7-bit US-ASCII.

As I wrote earlier (and Michael also emphasized): You *must not* remove
declaration headers /unconditionally/!. You /may/ remove 7-bit US-ASCII
declarations. And that's it. So you'd need to check first, whether /both/
declarations are set at the same time. And then (only then!) you may
remove both headers. In any other case those headers have to be left
untouched.

As for the MIME declaration: It is /required/ as long as charset/encoding
declarations are present. So, theoretically, it could be deleted, if
the charset/encoding declarations are deleted *and* an additional check,
whether /other/ MIME features are present (in the headers and/or the body)
results in "none". *But* that would ensure only compatibility with MIME
compliant default settings for networks. Since nobody /needs/ to default
to MIME, your postings can be interpreted as anything. Only with the
MIME declaration set, you can ensure, that your posting reaches the
reader undamaged. (As long as no error is introduced by transmission
or faulty settings, like the ones you currently try to use.)

To sum up: Fiddling with the MIME/charset/encoding headers isn't worth
the effort and is /very/ likely to produce malformed postings (at least
on a somewhat regular basis). Keeping an attitude like:

| I've never had a problem removing almost all those extraneous headers.
| Nobody has complained either.

should earn you a ban by your Usenet provider, when that's all that you
have to say, after you received a warning, that you deliberately create
malformatted postings...

Bernd

[Stijn De Jong]

On Mon, 9 Jan 2017 18:09:54 +0000 (UTC), you wrote:

> So, while I don't understand why just the one content-type line won't work,
> I'll try the first 2 of those 3 lines to see if the name works:

I'll try all three, but I don't see why the encoding matters since it's not something that I even do manually (so it's all automatic so how do I know what the encoding is?).

Here is the test set of names:
Marcus M?nnig
Michael B?uerle

Here are the lines taken out of Michael's header:

Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit

Here are those exact lines added to my header above:

Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version: ,Content-Type: ,Content-Transfer-Encoding: ';

Add_Headers := 'X-Comment: mime + charset + 8-bit encoding'#13#10 + 'Mime-Version: 1.0'#13#10 + 'Content-Type: text/plain; charset=ISO-8859-1; format=fixed'#13#10 + 'Content-Transfer-Encoding: 8bit'#13#10;

[Michael Bäuerle]

Stijn De Jong wrote:
>
> [...]

> // CASE-SENSITIVITY:

> // Michael B?uerle advised in news.software.readers on [...]
^
If you have no option to correctly encode my name, please use "ae" as
ASCII transcription for the umlaut "ä".

The complete list of ASCII transcriptions for non-ASCII special
characters used in german language looks like this:

Ä => Ae
ä => ae
Ö => Oe
ö => oe
Ü => Ue
ü => ue
ẞ => SS (sometimes "SZ" if ambigous otherwise, never "B"!)

ß => ss (sometimes "sz" if ambigous otherwise, never "B"!)

[Stijn De Jong]

On Mon, 9 Jan 2017 18:16:34 +0000 (UTC), you wrote:

> Here are the lines taken out of Michael's header:
> Mime-Version: 1.0
> Content-Type: text/plain; charset=UTF-8; format=fixed
> Content-Transfer-Encoding: 8bit

What makes no sense is that I copied, verbatim, Michael's headerlines!

Here are the lines from Michael's previous header:
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=fixed
Content-Transfer-Encoding: 8bit

Here are the lines from my prior header:
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=fixed
Content-Transfer-Encoding: 8bit

X-Comment: mime + charset + 8-bit encoding

So I don't see why the server munged the contents incorrectly.

I'll try this test without removing the headerlines:
Marcus M?nnig
Michael B?uerle


Remove_Headers := '';
Add_Headers := 'X-Comment: I did not change the mime, type, or encoding headerlines!'#13#10;

[Stijn De Jong]

On Mon, 9 Jan 2017 19:30:49 +0100, you wrote:

> Remove_Headers := '';
> Add_Headers := 'X-Comment: I did not change the mime, type, or encoding headerlines!'#13#10;

As I had stated before, the 3 headerlines don't seem to matter at all.

1. Notice in the first complaint, I had none of the 3 headerlines:
Mime-Version:
Content-Type:
Content-Transfer-Encoding:

2. Notice in the penultimate test I /exactly/ copied Michael's headerlines.

Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit

3. Notice in the last test, I didn't change my headerlines at all.
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

So the headerlines don't seem to make any difference in those tests.
Something is wrong, but the problem is not in the headerlines.

This has been my observation all along, which is that it doesn't matter
what I put in the headerlines, so why do they bother to exist?

[Michael Bäuerle]

Stijn De Jong wrote:
> On Mon, 9 Jan 2017 18:09:54 +0000 (UTC), you wrote:
> >
> > So, while I don't understand why just the one content-type line won't work,
> > I'll try the first 2 of those 3 lines to see if the name works:
>
> I'll try all three, but I don't see why the encoding matters since
> it's not something that I even do manually (so it's all automatic so
> how do I know what the encoding is?).

By looking in the header of the original message. If your software do
some type of character set conversion, your have to ask this piece of
software what it has created.

> Here is the test set of names:
> Marcus M?nnig
> Michael B?uerle

Somebody have replaced the non-ASCII characters by '?' again.

> Here are the lines taken out of Michael's header:
> Mime-Version: 1.0
> Content-Type: text/plain; charset=UTF-8; format=fixed
> Content-Transfer-Encoding: 8bit
>
> Here are those exact lines added to my header above:
>
> Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version: ,Content-Type: ,Content-Transfer-Encoding: ';
>
> Add_Headers := 'X-Comment: mime + charset + 8-bit encoding'#13#10 + 'Mime-Version: 1.0'#13#10 + 'Content-Type: text/plain; charset=ISO-8859-1; format=fixed'#13#10 + 'Content-Transfer-Encoding: 8bit'#13#10;

A fixed character set can't work.
You have to use the one from the article you quote or you must convert
the data to the fixed character set (not possible in the case of my
article <news:AABYc82AAKEAA...@WStation5.stz-e.de> that
contains the Unicode character U+1E9E that's not present in ISO 8859-1).

[Stijn De Jong]

On Mon, 9 Jan 2017 19:19:49 +0100, you wrote:

> The complete list of ASCII transcriptions for non-ASCII special
> characters used in german language looks like this:

That's clever!
How did you cancel the original message?

Looking at your headers, I see:
ORIGINAL:
Date: Mon, 9 Jan 2017 18:50:56 +0100
Message-ID: <AABYc82AAKEAA...@WStation5.stz-e.de>
Cancel-Lock: sha1:EmjA6jIHR97yvxEVd4VdYEMJxgA= sha1:ckiWxtQXGMzRi67eeTe9wT9f5ss=

SUBSEQUENT:
Supersedes: <AABYc82AAKEAA...@WStation5.stz-e.de>
Date: Mon, 9 Jan 2017 19:19:49 +0100
Message-ID: <AABYc9RFAKQAA...@WStation5.stz-e.de>
Cancel-Lock: sha1:6Jhjhn7Kev5tOG4A8RQPkn32KSI= sha1:zP70ulLerRZnbvmVmMlnZKLB7xw=
Cancel-Key: sha1:yiWTpjw2QsxAU54wwMOreE7AXCc= sha1:yIG5eRXg+SqAPIogwoLenRzr7Gk=

Based on your headers, you put something called a "cancel lock" on the original!

Looking up how to cancel my Usenet posts, I see there are 'control' messages:
https://en.wikipedia.org/wiki/Control_message#cancel

Looking up control messages, I see one type is a "cancel" message:
http://wiki.killfile.org/projects/usenet/faqs/cancel/

And looking up your "cancel-lock" header, I see information about it here:
https://www.eyrie.org/~eagle/usefor/drafts/draft-ietf-usefor-cancel-lock-01.txt

Looking up whether 40Tude Dialog has a native "cancel" capability, I find this:
http://dialog.datalist.org/scripts/CancelLock.html

But it specifically says public servers don't support the syntax.

[Dave Sparks]

On Mon, 09 Jan 2017 17:18:19 +0000, Stijn De Jong wrote:

...

> TESTCASE:
> 1. Marcus M?nnig <== the "o" in the surname is the test 2. Michael
> B?uerle <== the "a" in the surname is the test
>
>
> Remove_Headers := 'User-Agent: ,Message-ID: ,Date: ,Mime-Version:
> ,Content-Type: ,Content-Transfer-Encoding: ';
> Add_Headers := 'Content-Type: text/plain; charset=utf-8'#13#10 +
> 'X-Comment: No MIME, no encoding headerline; charset is set to
> utf-8'#13#10;

Since you've removed the Content-Encoding: header, the message body would
be treated as 7-bit, making the non-ASCII characters invalid. If the
encoding used by Dialog had been UTF-8, the ä and ö would have been
represented as two bytes, each with the most significant bit set,
resulting in ?? in each case. Since they have each been replaced by a
single ?, the encoding used by Dialog was probably ISO-8859-1 which for ä
and ö is marginally more efficient than UTF-8 (but wouldn't be good
enough for Stanisław Lem).

Why not just let Dialog get it right? Even if you discount personal
names, UK English needs characters outwith the ASCII subset (æ, œ, é, £,
€).

[Michael Bäuerle]

Stijn De Jong wrote:
> On Mon, 9 Jan 2017 19:19:49 +0100, you wrote:
> >
> > The complete list of ASCII transcriptions for non-ASCII special
> > characters used in german language looks like this:
>
> That's clever!
> How did you cancel the original message?

It was a supersede, requested with the "Supersedes" header field:
<https://tools.ietf.org/html/rfc5536#section-3.2.12>

> Looking at your headers, I see:
> ORIGINAL:
> Date: Mon, 9 Jan 2017 18:50:56 +0100
> Message-ID: <AABYc82AAKEAA...@WStation5.stz-e.de>
> Cancel-Lock: sha1:EmjA6jIHR97yvxEVd4VdYEMJxgA= sha1:ckiWxtQXGMzRi67eeTe9wT9f5ss=
>
> SUBSEQUENT:
> Supersedes: <AABYc82AAKEAA...@WStation5.stz-e.de>
> Date: Mon, 9 Jan 2017 19:19:49 +0100
> Message-ID: <AABYc9RFAKQAA...@WStation5.stz-e.de>
> Cancel-Lock: sha1:6Jhjhn7Kev5tOG4A8RQPkn32KSI= sha1:zP70ulLerRZnbvmVmMlnZKLB7xw=
> Cancel-Key: sha1:yiWTpjw2QsxAU54wwMOreE7AXCc= sha1:yIG5eRXg+SqAPIogwoLenRzr7Gk=
>
> Based on your headers, you put something called a "cancel lock" on the original!

Yes, this is used for authentication of cancel and supersede operations.

The Cancel-Locks are created with a secret and matching Cancel-Keys
can only be generated with knowledge of this secret.

The example above contains two Locks, one from my newsreader and one
from the server that injected the article (many servers do this for
their users so that things work automagically even with newsreaders
that don't have support for it).

> Looking up how to cancel my Usenet posts, I see there are 'control' messages:
> https://en.wikipedia.org/wiki/Control_message#cancel
>
> Looking up control messages, I see one type is a "cancel" message:
> http://wiki.killfile.org/projects/usenet/faqs/cancel/
>
> And looking up your "cancel-lock" header, I see information about it here:
> https://www.eyrie.org/~eagle/usefor/drafts/draft-ietf-usefor-cancel-lock-01.txt
>
> Looking up whether 40Tude Dialog has a native "cancel" capability, I find this:
> http://dialog.datalist.org/scripts/CancelLock.html
>
> But it specifically says public servers don't support the syntax.

Every server can decide to honor or ignore Cancel-Lock/Key.
But today many servers use this scheme (and most others likely ignore
all cancel and supersede requests).

[Stijn De Jong]

On Mon, 9 Jan 2017 19:24:21 +0000, you wrote:

> Since you've removed the Content-Encoding: header, the message body would
> be treated as 7-bit, making the non-ASCII characters invalid.

I don't think the headers make any difference since I'm not even touching
the headers and the server still munges the non-standard characters.

I found, by googling, that I can set the group default "charset" to any of
two dozen or so options so I just set it at UTF-8 for this test.

Remember, I'm not touching the MIME/Content/Encoding headers in the past
dozen or so messages and it was still screwed up so I'm not sure why those
headerlines even bother to exist.

Marcus M?nnig
Michael B?uerle

Remove_Headers := '';
Add_Headers := 'X-Comment: I did not change the mime, type, or encoding
headerlines!'#13#10;

All posts to this newsgroup are currently set at UTF-8 encoding (that
option of which I never noticed before I started looking how to do that).

BTW, I don't care what encoding it uses.
I just type on a US English keyboard and I stay standard, just as I wrap at
72 characters and I don't do non-standard things (which is the way it
should be, as the one doing non-standard things are the ones screwing
everything up).

Since I don't care what the default encoding or charset is for the newgroup
posts, what should I set it to so that everything I type (which is only
standard stuff) just works like it's supposed to work?

[Stijn De Jong]

On Mon, 9 Jan 2017 21:18:44 -0000, you wrote:

> Every server can decide to honor or ignore Cancel-Lock/Key.
> But today many servers use this scheme (and most others likely ignore
> all cancel and supersede requests).

Thanks. Since I'm using aioe, I asked on the alt.free.newsservers that
question, which is off topic here anyway so it best belongs there.

Thanks for being clever though - as that was a nice thing indeed.

[Stijn De Jong]

On Mon, 9 Jan 2017 19:48:23 +0100, you wrote:

> A fixed character set can't work.
> You have to use the one from the article you quote or you must convert
> the data to the fixed character set (not possible in the case of my
> article <news:AABYc82AAKEAA...@WStation5.stz-e.de> that
> contains the Unicode character U+1E9E that's not present in ISO 8859-1).

Thanks for describing the fact that even your own headers can't account for
the characters used in your post, and that no attempt on my part to rectify
that could possibly work.

This all points to the same thing that I originally said, which is that the
headers don't matter for those three lines.
Mime-Version (it's almost always 1.0 anyway so that should just default)
Content-Type & character set (doesn't matter what I put there anyway)
Encoding (why would text even be encoded unless that means ascii?)

[Bernd Rose]

On Tue, 10th Jan 2017 01:06:06 +0100, Stijn De Jong wrote:

Michael Bäuerle wrote:
>> A fixed character set can't work.
>> You have to use the one from the article you quote or you must convert
>> the data to the fixed character set (not possible in the case of my
>> article <news:AABYc82AAKEAA...@WStation5.stz-e.de> that
>> contains the Unicode character U+1E9E that's not present in ISO 8859-1).
>
> Thanks for describing the fact that even your own headers can't account for
> the characters used in your post, and that no attempt on my part to rectify
> that could possibly work.
>
> This all points to the same thing that I originally said, which is that the
> headers don't matter for those three lines.
> Mime-Version (it's almost always 1.0 anyway so that should just default)
> Content-Type & character set (doesn't matter what I put there anyway)
> Encoding

You can't insert arbitrary encoding information directly before sending and
expect the message to be encoded correctly! Dialog converts any received
message to Unicode and holds them inside an Unicode memory buffer (the same
with newly created messages) until you send it. Then Dialog tries any
character set and encoding you permitted Dialog to use in the sequence you
set up. If you don't change the default settings, Dialog will find at least
one suitable charset/encoding combination, that fits to the current chars
inside the memory text buffer. (When the original post contains characters
not supported by your operating system, that my still lead to question marks
or sth. like that, though.) As long as you don't fiddle with the settings,
the post will go out correctly. Changing settings requires correct knowledge
of the consequences. Else: Hands off!!

Inside OnBeforeSendingMessage, the charset/encoding declaration is already
decided by Dialog. /Any/ change you do has to be compliant with that
decision. Else, you need to re-encode the whole message (headers /and/ body)
to match your new encoding. Even if you /don't/ change charset/encoding,
you need to take care, that any change inside OnBeforeSendingMessage fits
the selected charset/encoding.

> (why would text even be encoded unless that means ascii?)

Different operating systems use different charset/encoding defaults. And
different system languages require a different set of characters. The
transmission capacity was historically (and to some degree still is) a
rare good. Therefore, out of the spectrum of possible encoding systems,
the most efficient is selected. For plain English text 7-bit usually is
sufficient. Other languages (need to) use 8-bit or more to represent
their characters. Since encoding requires overhead (and historically also
relevant: computation power), it is often more efficient to send messages
encoded in 8-bit. Since different languages use the upper half of such
a bit plane differently, the charset information has to be transmitted
alongside the bit-depth-encoding information.

Bernd

[Michael Bäuerle]

Stijn De Jong wrote:
> On Mon, 9 Jan 2017 19:24:21 +0000, you wrote:
> >
> > Since you've removed the Content-Encoding: header, the message body would
> > be treated as 7-bit, making the non-ASCII characters invalid.
>
> I don't think the headers make any difference since I'm not even touching
> the headers and the server still munges the non-standard characters.

It is very likely not the server that replaces the special characters.
I often use Aioe for testing and have never seen that it cripples
MIME articles.

> I found, by googling, that I can set the group default "charset" to any of
> two dozen or so options so I just set it at UTF-8 for this test.

For testing and european languages UTF-8 can be used. But be warned:
Dialog has a bug in the UTF-8 encoder for all codepoints beyond U+FFFF.

Example: 😀 (U+1F600 GRINNING FACE)

Dialog should cripple such characters with UTF-8 (by encoding them with
reserved surrogate codepoints), but works correctly with UTF-7. Because
itself it can correctly display the broken encoding, you may not notice
until you check with a different newsreader or a hex editor.

[Michael Bäuerle]

Michael Bäuerle wrote:
> Stijn De Jong wrote:
> >

> > [...]

> > I found, by googling, that I can set the group default "charset" to any of
> > two dozen or so options so I just set it at UTF-8 for this test.
>
> For testing and european languages UTF-8 can be used. But be warned:
> Dialog has a bug in the UTF-8 encoder for all codepoints beyond U+FFFF.
>
> Example: 😀 (U+1F600 GRINNING FACE)

To verify: This article was posted via Aioe.

[tlvp]

On Mon, 9 Jan 2017 16:44:19 +0000 (UTC), Stijn De Jong wrote, in part:

> // CASE-SENSITIVITY:

> // Michael B?uerle advised in news.software.readers ...

See? You garble poor Michael's surname by not allowing for the possibility
of characters outside the US-ASCII range. He knows (and so do I) that "?"
should be "ä" (a-umlaut), but you should have more respect for his name and
not allow it to become so disfigured in the first place. Cheers, -- tlvp
--
Avant de repondre, jeter la poubelle, SVP.

[Michael Bäuerle]

Stijn De Jong wrote:
> On Mon, 9 Jan 2017 19:30:49 +0100, you wrote:
> >
> > Remove_Headers := '';
> > Add_Headers := 'X-Comment: I did not change the mime, type, or encoding headerlines!'#13#10;
>
> As I had stated before, the 3 headerlines don't seem to matter at all.
>
> 1. Notice in the first complaint, I had none of the 3 headerlines:
> Mime-Version:
> Content-Type:
> Content-Transfer-Encoding:
>
> 2. Notice in the penultimate test I /exactly/ copied Michael's headerlines.
> Mime-Version: 1.0
> Content-Type: text/plain; charset=UTF-8; format=fixed
> Content-Transfer-Encoding: 8bit
>
> 3. Notice in the last test, I didn't change my headerlines at all.
> Mime-Version: 1.0
> Content-Type: text/plain; charset="us-ascii"
> Content-Transfer-Encoding: 7bit
>
> So the headerlines don't seem to make any difference in those tests.
> Something is wrong, but the problem is not in the headerlines.

No, it looks like you expect that these headerfields have some effect
on the sending side. That's not true, they are just labels that
describe the article format - in which the article must already be!

They can later be used by the receiver for correct interpretation.

[tlvp]

On Mon, 9 Jan 2017 19:41:24 +0100, Stijn De Jong wrote:

> On Mon, 9 Jan 2017 19:30:49 +0100, you wrote:
>
>> Remove_Headers := '';
>> Add_Headers := 'X-Comment: I did not change the mime, type, or encoding headerlines!'#13#10;
>
> As I had stated before, the 3 headerlines don't seem to matter at all.

Of course the header lines matter. But they're incapable of substituting
back, in place of the question mark (?) that you see, the original ö or ä
(o-umlaut or a-umlaut) that were used in the Mönnig or Bäuerle that your
newsreader reduced to a "?".

Let your newsreader display the full Unicode complement, and then when you
reply -- with those 3 headerlines in place -- you should no longer get "?",

HTH. Cheers, -- tlvp

[tlvp]

On Tue, 10 Jan 2017 09:50:09 +0100, Michael Bäuerle wrote, in part:

> Dialog has a bug in the UTF-8 encoder for all codepoints beyond U+FFFF.
>
> Example: �� (U+1F600 GRINNING FACE)
>
> Dialog should cripple such characters with UTF-8 (by encoding them with
> reserved surrogate codepoints), but works correctly with UTF-7.

Thanks very much for pointing that out: I never knew that before.

You yourself were using charset=UTF-8 for that posting. And I had imagined
for some years now that I too was using that, though I see now, checking
headers of recent messages elsewhere that I've sent, I seem to have
reverted to 7-bit US-ASCII (!) :-{ .

Yet if I consult my settings under

Dialog > Message > Charsets > Default Group Charset

I see UTF-8 highlighted; and the other three Charset types (Body, Subject,
and From) are all shown as Default. Am I doing something wrong?

Anway, under Settings > General Settings... > Charsets > ...for postings

I see "Use best matching charset out of: utf-8 (highlighted), utf-7,
us-ascii, iso-8859-2, iso-8859-1, ..."

So maybe Dialog uses US-ASCII when nothing more serious is needed, but does
use UTF-8 (or maybe UTF-7) when I use such European diacritically modified
characters as à, á, ä, ą, ç, ć, è, é, ê, ę, ŀ, ń, ñ, ó, ś, ź, ż, usw.
(Well, this post will inadvertently serve as a test, I guess. Apologies!)

To be followed up on if this posts with garbled characters. Cheers, -- tlvp

[tlvp]

On Tue, 10 Jan 2017 05:23:30 -0500, tlvp wrote:

> So maybe Dialog uses US-ASCII when nothing more serious is needed, but does
> use UTF-8 (or maybe UTF-7) when I use such European diacritically modified
> characters as à, á, ä, ą, ç, ć, è, é, ê, ę, ŀ, ń, ñ, ó, ś, ź, ż, usw.
> (Well, this post will inadvertently serve as a test, I guess. Apologies!)

And it passed the test :-) -- I see charset="utf-8" in its headers.

I guess Dialog uses the "least costly" charset option it can get away with.
Devilishly clever, that :-) . My thanks to MM and his collaborators!

(Now (OT) if only I could find a windows AltGr US-International keystroke
shortcut scheme for actually producing 2-key combinations for the uniquely
Polish characters ą, ć, ę, ł, ń, ś, ź, ż, similar to those for the other
more Western characters quoted above, I could die content :-) .)

[Dave Sparks]

On Tue, 10 Jan 2017 01:06:02 +0100, Stijn De Jong wrote:

> On Mon, 9 Jan 2017 19:24:21 +0000, you wrote:
>
>> Since you've removed the Content-Encoding: header, the message body
>> would be treated as 7-bit, making the non-ASCII characters invalid.
>
> I don't think the headers make any difference since I'm not even
> touching the headers and the server still munges the non-standard
> characters.

The headers in your message include:

Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

so the most significant bit of each byte in the message body will be
ignored, and may be stripped.

> I found, by googling, that I can set the group default "charset" to any
> of two dozen or so options so I just set it at UTF-8 for this test.

That may be the default for handling incoming messages with incomplete
headers.

Dialog *can* post messages with non-ASCII characters - see
<8rvhm2iwm3r5$.1rx0y710bzqpv$.d...@40tude.net> in this newsgroup, dated
Mon, 11 Jul 2016 21:06:25 -0400 .

> Remove_Headers := '';
> Add_Headers := 'X-Comment: I did not change the mime, type, or encoding
> headerlines!'#13#10;

There's no X-Comment: header in your message.

...

> BTW, I don't care what encoding it uses.
> I just type on a US English keyboard and I stay standard, just as I wrap
> at 72 characters and I don't do non-standard things (which is the way it
> should be, as the one doing non-standard things are the ones screwing
> everything up).

Non-ASCII characters are *not* non-standard. If you quote text from
other people's messages, or copy and paste text from a word processor
(even text you typed yourself using only ASCII characters) then you are
likely to include non-ASCII characters.

There was a time when NNTP was restricted to ASCII and 7-bit transfer
encoding, but the world moved on in a previous century.

> Since I don't care what the default encoding or charset is for the
> newgroup posts, what should I set it to so that everything I type (which
> is only standard stuff) just works like it's supposed to work?

UTF-8 will encode anything (including Klingon), and when the alphabet is
Latin the overhead is small (except possibly when the language is
Finnish). You don't need to consider any other encoding unless the
character set is non-Latin (Greek, Cyrillic, Thai, Japanese, ...).

[Michael Bäuerle]

tlvp wrote:
> On Tue, 10 Jan 2017 09:50:09 +0100, Michael Bäuerle wrote, in part:
> >
> > Dialog has a bug in the UTF-8 encoder for all codepoints beyond U+FFFF.
> >

> > Example: ? (U+1F600 GRINNING FACE)
^
Dialog tried to use a surrogate pair for UTF-8 (they are allowed only
for UTF-16) and encoded two 3-byte sequences for two surrogate code-
points:

0xED 0xA0 0xBD 0xED 0xB8 0x80

This in not valid UTF-8! The correct encoding for U+1F600 in UTF-8 is
the single 4-byte sequence:

0xF0 0x9F 0x98 0x80

> > Dialog should cripple such characters with UTF-8 (by encoding them with
> > reserved surrogate codepoints), but works correctly with UTF-7.

... because this is Base64 encoded UTF-16 (and therefore internally use
surrogate pairs).

> Thanks very much for pointing that out: I never knew that before.
>
> You yourself were using charset=UTF-8 for that posting. And I had imagined
> for some years now that I too was using that, though I see now, checking
> headers of recent messages elsewhere that I've sent, I seem to have
> reverted to 7-bit US-ASCII (!) :-{ .
>
> Yet if I consult my settings under
>
> Dialog > Message > Charsets > Default Group Charset
>
> I see UTF-8 highlighted; and the other three Charset types (Body, Subject,
> and From) are all shown as Default. Am I doing something wrong?

Normally UTF-8 should be used for Unicode in USENET articles. But
because of the bug in Dialog (see above), only UTF-7 work for all
codepoints without manual user intervention.

> Anway, under Settings > General Settings... > Charsets > ...for postings
>
> I see "Use best matching charset out of: utf-8 (highlighted), utf-7,
> us-ascii, iso-8859-2, iso-8859-1, ..."
>
> So maybe Dialog uses US-ASCII when nothing more serious is needed,

In general it is a good idea. This is also officially recommended by
MIME [1]:
|
| In general, composition software should always use the "lowest common
| denominator" character set possible. For example, if a body contains
| only US-ASCII characters, it SHOULD be marked as being in the US-
| ASCII character set, not ISO-8859-1, which, like all the ISO-8859
| family of character sets, is a superset of US-ASCII. More generally,
| if a widely-used character set is a subset of another character set,
| and a body contains only characters in the widely-used subset, it
| should be labelled as being in that subset. This will increase the
| chances that the recipient will be able to view the resulting entity
| correctly.

> but does
> use UTF-8 (or maybe UTF-7) when I use such European diacritically modified
> characters as à, á, ä, ą, ç, ć, è, é, ê, ę, ŀ, ń, ñ, ó, ś, ź, ż, usw.

They are all OK, because none use a codepoint beyond U+FFFF.


__________________
[1] <https://tools.ietf.org/html/rfc2046#section-4.1.2>

[Stijn De Jong]

On Tue, 10 Jan 2017 10:36:44 +0100, you wrote:

>> So the headerlines don't seem to make any difference in those tests.
>> Something is wrong, but the problem is not in the headerlines.
>
> No, it looks like you expect that these headerfields have some effect
> on the sending side. That's not true, they are just labels that
> describe the article format - in which the article must already be!
>
> They can later be used by the receiver for correct interpretation.

Thanks Michael.
I haven't written in a day because I'm completely confused by the results
of my tests of trying to see what Dialog does with the three headers.

In the 40tude GUI, I can't find any setting in the GUI that sets any of
those three headers.
1. Mime-Version:
2. Content-Type:
3. Content-Transfer-Encoding:

By default, 40tude seems to have the following:
1. Mime-Version: 1.0
2. Content-Type: text/plain; charset="us-ascii"
3. Content-Transfer-Encoding: 7bit

The only related setting I can find in 40tude is the "charset":
Selected Group Options > Charset > Default group charset=Default

The confusion arises when I change that default group charset to something
else. No matter what I change it to, the 3 related headerlines remain the
same.

That makes no sense to me whatsoever.

For example, I am changing the default group charset for this message in
the 40tude GUI to UTF-8 but I already tested it and the header will remain
as:
1. Mime-Version: 1.0
2. Content-Type: text/plain; charset="us-ascii"
3. Content-Transfer-Encoding: 7bit

If I change that Charset to, say, UTF-7, or ASCII, or ISO-8859-1, it still
won't change any of those three headers.

Why is that?
What is it that Dialog will always have those same three headers no matter
what "Charset" I set in the GUI?

[Stijn De Jong]

On Tue, 10 Jan 2017 04:39:19 -0500, you wrote:

> Let your newsreader display the full Unicode complement, and then when you
> reply -- with those 3 headerlines in place -- you should no longer get "?",

I'm sorry but I don't understand what you're suggesting.

The fact is that I have Dialog set to its defaults, with no
onbeforesendingmessage scripts, and Dialog still does two things no matter
what I set or don't set in the Dialog GUI.

1. Dialog sets these three header lines, always, no matter what I set in
the dialog GUI:

Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

2. Dialog (or the server?) makes Michael's name "B?uerle" no matter what I
set in the Dialog GUI.

Why is that?

What setting do you recommend I set in the Dialog GUI so that when I type
Michael's last name, it doesn't get substituted with a question mark?

[Stijn De Jong]

On Tue, 10 Jan 2017 06:59:52 +0100, you wrote:

> You can't insert arbitrary encoding information directly before sending and
> expect the message to be encoded correctly! Dialog converts any received
> message to Unicode and holds them inside an Unicode memory buffer (the same
> with newly created messages) until you send it. Then Dialog tries any
> character set and encoding you permitted Dialog to use in the sequence you
> set up.

I don't completely understand what you wrote because I don't know what
"unicode" is (all I know is ASCII and these things):
https://s28.postimg.org/xstzeio31/unicode.jpg

Looking up unicode:
https://en.wikipedia.org/wiki/Unicode

OK. It's just a better (newer) version of what ASCII did which is simply
that unicode (like ASCII did) defines every character that it wants to
represent into a set of ones and zeroes.

Unicode seems more complex than ASCII though, in that unicode seems to have
multiple parts such as this line in that Wikipedia article:
"UTF encodings include UTF-8, which is an 8-bit variable-width encoding
which maximizes compatibility with ASCII;"

So where in Dialog do I simply set it to use "unicode"?
There is only a single spot in the entire Dialog GUI that has anything
whatsoever to do with this stuff, and that's the "charsets" default for
groups, which makes no difference in the headerlines no matter what I set
it to.

So how do I use this charsets to set it to unicode when that's not even an
option? https://s27.postimg.org/b58w6zvhf/utf8.jpg

There's nothing there for unicode.
And, no matter what I set there, nothing changes in the header anyway.

> If you don't change the default settings, Dialog will find at least
> one suitable charset/encoding combination, that fits to the current chars
> inside the memory text buffer. (When the original post contains characters
> not supported by your operating system, that my still lead to question marks
> or sth. like that, though.) As long as you don't fiddle with the settings,
> the post will go out correctly. Changing settings requires correct knowledge
> of the consequences. Else: Hands off!!

Oh oh ...
Does that mean that the only setting there is in the entire Dialog GUI for
anything having to do with the "charsets" should be left at the "Default"?
https://s24.postimg.org/m0d7ukg7p/default.jpg

> Inside OnBeforeSendingMessage, the charset/encoding declaration is already
> decided by Dialog. /Any/ change you do has to be compliant with that
> decision.

My observation is that, with or without onbeforesendingmessage, absolutely
nothing I can set inside the Dialog GUI will change the default headers
which are *always* these three headers, no matter what I set the "charsets"
to in the DIalog GUI:

[Stijn De Jong]

On Tue, 10 Jan 2017 04:28:37 -0500, you wrote:

> See? You garble poor Michael's surname by not allowing for the possibility
> of characters outside the US-ASCII range. He knows (and so do I) that "?"

> should be "??" (a-umlaut), but you should have more respect for his name and

> not allow it to become so disfigured in the first place. Cheers, -- tlvp

If it's not clear yet, I must plainly state that there is absolutely no
setting that I can find in 40Tude dialog that won't garble Michael's name.

There is only one setting possible, and that doesn't change anything
whatsoever.

So, I admit I'm thoroughly confused.

[Stijn De Jong]

On Tue, 10 Jan 2017 05:23:30 -0500, you wrote:

> Thanks very much for pointing that out: I never knew that before.
>
> You yourself were using charset=UTF-8 for that posting. And I had imagined
> for some years now that I too was using that, though I see now, checking
> headers of recent messages elsewhere that I've sent, I seem to have
> reverted to 7-bit US-ASCII (!) :-{ .
>
> Yet if I consult my settings under
>
> Dialog > Message > Charsets > Default Group Charset
>
> I see UTF-8 highlighted; and the other three Charset types (Body, Subject,
> and From) are all shown as Default. Am I doing something wrong?
>
> Anway, under Settings > General Settings... > Charsets > ...for postings
>
> I see "Use best matching charset out of: utf-8 (highlighted), utf-7,
> us-ascii, iso-8859-2, iso-8859-1, ..."
>
> So maybe Dialog uses US-ASCII when nothing more serious is needed, but does
> use UTF-8 (or maybe UTF-7) when I use such European diacritically modified

> characters as ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, usw.

> (Well, this post will inadvertently serve as a test, I guess. Apologies!)
>
> To be followed up on if this posts with garbled characters. Cheers, -- tlvp

I'm very confused so I realize this may just confuse things more, but I
have found out from experimenting that it doesn't matter WHAT I set Dialog
to in the GUI, the three headers are *always* the same as what you see in
your header, which is:

1. Mime-Version: 1.0
2. Content-Type: text/plain; charset="us-ascii"
3. Content-Transfer-Encoding: 7bit

I do realize I'm confused, but the ONLY GUI items I can find in Dialog that
is related to these three things is the group default options, which I have
set to various things, but NONE of those settings show up in the header.
https://s28.postimg.org/xstzeio31/unicode.jpg

The header is always the same as those three lines above no matter what I
set in the Dialog GUI for "charsets" encoding.
https://s24.postimg.org/c3271i8lt/default.jpg

For example, I set, like you did, the UTF-8 encoding in the group defaults,
and I'm already positive that the header above will be the same as it
always is for all Dialog posts.
https://s27.postimg.org/b58w6zvhf/utf8.jpg

[Stijn De Jong]

On Tue, 10 Jan 2017 10:34:38 +0000, you wrote:

> UTF-8 will encode anything (including Klingon), and when the alphabet is
> Latin the overhead is small (except possibly when the language is
> Finnish). You don't need to consider any other encoding unless the
> character set is non-Latin (Greek, Cyrillic, Thai, Japanese, ...).

I'm fine with any encoding you suggest.
I have Dialog set to use UTF-8 for this message.
But it won't work.

So there must be a trick somewhere, since the ONLY GUI setting in Dialog
that has anything whatsoever to do with this stuff is the group options
default, which I have set to UTF-8 for this message, but it won't work.

Somehow, tlvp got his to work.
So there's a trick somewhere, in Dialog, to get it to use UTF-8.

[Stijn De Jong]

On Tue, 10 Jan 2017 05:43:14 -0500, you wrote:

>> So maybe Dialog uses US-ASCII when nothing more serious is needed, but does
>> use UTF-8 (or maybe UTF-7) when I use such European diacritically modified

>> characters as ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, usw.

>> (Well, this post will inadvertently serve as a test, I guess. Apologies!)
>
> And it passed the test :-) -- I see charset="utf-8" in its headers.
>
> I guess Dialog uses the "least costly" charset option it can get away with.
> Devilishly clever, that :-) . My thanks to MM and his collaborators!
>
> (Now (OT) if only I could find a windows AltGr US-International keystroke
> shortcut scheme for actually producing 2-key combinations for the uniquely

> Polish characters ??, ??, ??, ??, ??, ??, ??, ??, similar to those for the other

> more Western characters quoted above, I could die content :-) .)

Can you tell me how to pass this same test?

??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??

Should I set the news.software.readers group default charset to "Default"?
https://s24.postimg.org/c3271i8lt/default.jpg

Or should I set the news.software.readers group default charst to "UTF-8"?
https://s24.postimg.org/c3271i8lt/default.jpg

NOTE: The group default is set to "Default" for this message.

[Stijn De Jong]

On Tue, 10 Jan 2017 22:29:04 +0100, you wrote:

> Should I set the news.software.readers group default charset to "Default"?
> https://s24.postimg.org/c3271i8lt/default.jpg
>
> Or should I set the news.software.readers group default charst to "UTF-8"?
> https://s24.postimg.org/c3271i8lt/default.jpg
>
> NOTE: The group default is set to "Default" for this message.

Can you tell me how to pass this same test?
??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??, ??

Trying again with the news.software.readers group default "Charset" set to
"UTF-8".

[Bernd Rose]

On Tue, 10th Jan 2017 22:11:35 +0100, Stijn De Jong wrote:

> Unicode seems more complex than ASCII though, in that unicode seems to have
> multiple parts such as this line in that Wikipedia article:
> "UTF encodings include UTF-8, which is an 8-bit variable-width encoding
> which maximizes compatibility with ASCII;"

UTF-8 is one (out of several) encoding scheme for Unicode characters.
Unicode is a character set representation like ASCII, but with a far
wider scope of supported characters. (Aim: /Every/ character shall be
represented.) The different encoding schemes are optimized for different
purposes. UTF-8 can be considered more "efficient" for charsets mainly
from 7-bit or 8-bit ASCII characters. UTF-16 needs somewhat less overhead
for encoding (e.g.) Chinese chars. When it comes to networking, UTF-8
can often be transmitted in an unchanged way, while UTF-16 may need an
additional re-encoding. And there are lots of other considerations,
before the "best-fitting" encoding scheme for a usage scenario has been
selected...

> So where in Dialog do I simply set it to use "unicode"?
> There is only a single spot in the entire Dialog GUI that has anything
> whatsoever to do with this stuff, and that's the "charsets" default for
> groups, which makes no difference in the headerlines no matter what I set
> it to.

Of course not. The Dialog help file clearly states for this setting, that
it enables you to select a default charset for postings, that are missing
a charset declaration. It is meant for *reading* such erroneous messages,
like the ones you created when implicitly declaring 7-bit US-ASCII (by
deleting all declaration info) while sending posts containing German
Umlauts. It, therefore, is a means to simplify error correction. Apart
from this by-group setting, the same is possible per posting in Dialog:
If you receive an undeclared posting, which isn't decoded correctly by
the group-default charset (e.g. a Russian posting in a German group with
manually set default German charset), it is possible to switch decode
charset for the posting by Message->Charset menu. This way, it is also
possible to read messages, that have a /wrong/ charset declaration, with
a better fitting charset.

The posting settings are set via Settings->General_Settings->Charsets
menu: "Allowed/Forbidden charsets for posting" / "Encoding non-US-ASCII
messages". All charsets listed in the "Allowed charsets" list are tried
(in the listed order) against the posting, which is about to be sent.
The first charset, that fits to every character used in the posting
will be used. Therefore, it is recommended to have US-ASCII on top, and
it is necessary to have at least one Unicode representation (like utf-8)
included, to ensure, that Dialog finds at least one method to encode
any character, that can be represented by the operating system. (Whether
there's a font, that includes an actual drawing of all those characters
(= glyphs), doesn't matter in this context.)

If you often read groups with (e.g.) Russian postings sent from Windows
systems, the Cyrillic ANSI Windows charset will come handy in this list,
as well. This charset could be used for answering such postings, without
the need to use the "bulkier" UTF-8 or sth. like that...

And: It is *not* recommended to "tidy up" those settings, as long as
one isn't /absolutely/ sure about the consequences of every decision!!

Bernd

[Stijn De Jong]

On Tue, 10 Jan 2017 23:30:34 +0100, you wrote:

> UTF-8 is one (out of several) encoding scheme for Unicode characters.

Michael, I'll use whatever you say to use for the "encoding" scheme for the
"Unicode" characters.

If you say to use UTF-8, then I'm happy to use it.
The only problem is that setting UTF-8 in the Dialog settings for the group
defaults doesn't seem to actually set it to UTF-8, at least as evidenced by
two facts:
a. The header is always the same, and,
b. The funky characters in your name are still mangled

> Unicode is a character set representation like ASCII, but with a far
> wider scope of supported characters.

Again, I will use whatever "character set representation" you say to use,
so, Unicode it is.

However, I see no setting inside of Dialog that allows Unicode. I've even
dropped back to the older (more stable) dialog, and I don't see anything
there either.

>> So where in Dialog do I simply set it to use "unicode"?

> The Dialog ... setting ... is meant for *reading* such erroneous messages,

> like the ones you created when implicitly declaring 7-bit US-ASCII (by
> deleting all declaration info) while sending posts containing German
> Umlauts.

OK. Then you cleared up why Dialog never changed the /outgoing/ headers
when I set the default group options to UTF-8 (or to ASCII or to
ISO-8859-15 or to anything).

So what do you recommend I set the default "charsets" option to in Dialog?
I currently have it at "Default" for lack of a better idea.

> If you receive an undeclared posting, which isn't decoded correctly by
> the group-default charset (e.g. a Russian posting in a German group with
> manually set default German charset), it is possible to switch decode
> charset for the posting by Message->Charset menu.

I see Message > Charsets where the options are:
1. Default group charset
2. Body charset
3. Subject charset
4. From charset

But that doesn't help me with setting my outgoing message to whatever you
recommend (which I think is Unicode character set representation with UTF-8
encoding).

> The posting settings are set via Settings->General_Settings->Charsets
> menu: "Allowed/Forbidden charsets for posting" / "Encoding non-US-ASCII
> messages".

> All charsets listed in the "Allowed charsets" list are tried
> (in the listed order) against the posting, which is about to be sent.
> The first charset, that fits to every character used in the posting
> will be used. Therefore, it is recommended to have US-ASCII on top, and
> it is necessary to have at least one Unicode representation (like utf-8)
> included, to ensure, that Dialog finds at least one method to encode
> any character, that can be represented by the operating system. (Whether
> there's a font, that includes an actual drawing of all those characters
> (= glyphs), doesn't matter in this context.)

Here is what Settings > General Settings > Charsets looks like by default:
https://s30.postimg.org/5kirjvhy9/general_settings_charsets.jpg

> And: It is *not* recommended to "tidy up" those settings, as long as
> one isn't /absolutely/ sure about the consequences of every decision!!

All I want is to use whatever settings you recommend in Dialog.

The problem is that I'm *already* at the defaults, and they're not working.

For example, here is the most basic test of the default dialog settings:
TEST: Marcus M?nnig, Michael B?uerle

[Stijn De Jong]

On Wed, 11 Jan 2017 00:55:04 +0100, you wrote:

> For example, here is the most basic test of the default dialog settings:
> TEST: Marcus M?nnig, Michael B?uerle

Testing with the older version of Dialog.

[tlvp]

On Tue, 10 Jan 2017 20:51:12 +0000 (UTC), Stijn De Jong wrote:

> In the 40tude GUI, I can't find any setting in the GUI that sets any of
> those three headers.
> 1. Mime-Version:
> 2. Content-Type:
> 3. Content-Transfer-Encoding:

Visit Settings > General Settings... > Charsets . HTH. Cheers, -- tlvp

[tlvp]

On Tue, 10 Jan 2017 22:29:04 +0100, Stijn De Jong wrote:

> Should I set the news.software.readers group default charset to "Default"?
> https://s24.postimg.org/c3271i8lt/default.jpg
>
> Or should I set the news.software.readers group default charst to "UTF-8"?
> https://s24.postimg.org/c3271i8lt/default.jpg

My own setting under

Group > Default group options > Charsets

is: UTF-8. And under

Group > Selected group options > Charsets

it is: [checkmark] This group uses the default group options.

> NOTE: The group default is set to "Default" for this message.

More to the point, go visit Settings > General Settings... > Charsets.
That's where the settings affecting your *postings* are entered. HTH.

[tlvp]

I agree that the headers in the post I'm replying to invoke

: Content-Type: text/plain; charset="us-ascii"
: Content-Transfer-Encoding: 7bit

Do mine, in this message, once I've added a few non-US-ASCII bytes? -- like
ç, ö, ñ, or ê? I'd be surprised. Cheers, -- tlvp

[tlvp]

On Tue, 10 Jan 2017 22:25:33 +0100, Stijn De Jong wrote:

> I'm fine with any encoding you suggest.
> I have Dialog set to use UTF-8 for this message.
> But it won't work.

You did *not* "have Dialog set to use UTF-8": instead, you have only

: Content-Type: text/plain; charset="us-ascii"
: Content-Transfer-Encoding: 7bit

Sorry. Cheers, -- tlvp

[tlvp]

You might try this approach: press [F1] (or click Help > Help contents...),
click the *Index* tab, enter char (or charset) into the search box, and
then Display whichever of the character set topics found seem(s) relevant.

HTH. Cheers, -- tlvp

[Bernd Rose]

On Wed, 11th Jan 2017 00:55:04 +0100, Stijn De Jong wrote:

[Bernd Rose wrote]

> Michael, I'll use whatever you say to use for the "encoding" scheme for the

^^^^^^^
> "Unicode" characters.

Check your references.

> Here is what Settings > General Settings > Charsets looks like by default:
> https://s30.postimg.org/5kirjvhy9/general_settings_charsets.jpg

Severely broken. I guess, you already "tidied up". (Which you must not, as
long as you don't /really/ know the consequences!!) US-ASCII should appear
only once. And it seems, that you /already/ moved all other charsets to
the "Forbidden charsets" list.

Bernd

[Whiskers]

The ultimate cure for that would probably be to delete all the user
setting files (I don't know where they would be on a Windows system) and
perhaps even uninstall the newsreader program, and start again.

But first perhaps try moving all the 'Forbidden charsets' back into the
'Use best matching charset out of' box. Set as it is, the program
cannot use anything other than 'us-ascii' unless there are some other
options scrolled off the display. There should only be entries in the
'Forbidden charset' box if you know that a particular character set
causes problems and that putting it there fixes the problem.

Presumably the computer concerned is capable of using UTF-8 and is set up
to be able to. Any computer incapable of using UTF-8 must be very old.

--
-- ^^^^^^^^^^
-- Whiskers
-- ~~~~~~~~~~