40tude find & replace 4 characters before sending message!!!!!!

[Doyun Jahng]

Best would be script to remove & replace chars before sending!

40tude find & replace can't do multiple regexp for all at one command
1 -> (opening singlequote d145 to straight singlequote d39)
2 -> (closing singlequote d146 to straight singlequote d39)
3 -> (opening doublequote d147 to straight doublequote d34)
4 -> (closing doublequote d148 to straight doublequote d34)

"Allowed charsets for your postings" does not work to remove them.
utf-7 before UTF-8 does not work to prevent them.

I do not like or want curly fancy quotes.
I want straight quotes always.
They came with text cut from web pages.

Can 40tude find & replace work on all at same time?

40tude uses Unicode UTF-16 internally.
Allowed charsets are in order below but are red herrings
1 -> us-ascii
2 -> utf-7
3 -> utf-8
This is red herring as not what the question is about.

At sent time it looks at allowed charsets.
40tude picks first one that can fit all characters.
40tude converts to that charset before sending.

This is not about allowed charsets.
This is about doing multiple find and replace before sending.

Best would be script to remove & replace chars before sending!

[Grant Taylor]

On 09/08/2018 06:45 PM, Doyun Jahng wrote:
> Best would be script to remove & replace chars before sending!

I have a few different (Linux shell) scripts that I run that act on the
clipboard.

They read the clipboard, modify the text as I want, and write it back to
the clipboard.

So my workflow is to copy the text in the source, run the appropriate
script, and then paste the modified text in the destination. IMHO it
works quite well. I use the different scripts for all sorts of
different things. My most common one is to rewrap text so that it's ~74
characters and supports format=flowed in email & newsgroup postings.



--
Grant. . . .
unix || die

[VanguardLH]

Grant Taylor wrote:

> Doyun Jahng wrote:
>
>> Best would be script to remove & replace chars before sending!
>
> I have a few different (Linux shell) scripts that I run that act on the
> clipboard.

40tude Dialog is a Windows program, not Linux.

[VanguardLH]

What if you deleted all allowed charsets except us-ascii?

[Doyun Jahng]

Response to Grant Taylor <gta...@tnetconsulting.net>

Syntax is same inside Linux (except Linux control+v is Windows control+q).

May be you can help on gvim regexp syntax to get 1 command to work?
On Windows gvim "vy" copies & "ga" identifies the curly character.

The problem is to convert 4 character before sending 40tude message!
convert open single quote d145 to straight single quote d39 (')
convert close single quote d146 to straight single quote d39 (')
convert open double quote d147 to straight double quote d34 (")
convert close double quote d148 to straight double quote d34 (")

The "digraph" percent sign (%) only work one at a time.
open single quote (d145) to straight single quote (d39) -> :%s/\%d145/'/g
close single quote (d146) to straight single quote (d39) -> :%s/\%d146/'/g
open double quote (d147) to straight double quote (d34) -> :%s/\%d147/"/g
close double quote (d148) to straight double quote (d34) -> :%s/\%d148/"/g
I can not get the syntax correct to do all 4 conversion at once!

The "\d" reduce the 4 command above to 2 command:
open/close single quote to straight single quote -> :%s/[\d145,\d146]/'/g
open/close double quote to straight double quote -> :%s/[\d147,\d148]/"/g

But I can not understand yet syntax to make above to 1 command!

[Doyun Jahng]

Response to VanguardLH <V...@nguard.LH>

> What if you deleted all allowed charsets except us-ascii?

Red herring. Many hours waste on 40tude charsets and order!
Would not ask if was easy!

Search and replace is only solution to work!

Best is script just before sending.
Script would search & replace four character before sent!

[VanguardLH]

Doyun Jahng wrote:

> Response to VanguardLH <V...@nguard.LH>
>
>> What if you deleted all allowed charsets except us-ascii?
>
> Red herring. Many hours waste on 40tude charsets and order!
> Would not ask if was easy!

Did you even try by deleting ALL allowed charset for your posts except
us-ascii? Doesn't look like it because you are still mentioning order
(of the charsets). There is no order when there is only one item.

> Search and replace is only solution to work! Best is script just
> before sending.

Then you need to learn Delphi in which Dialog was written and upon which
its scripts are based. After writing the script, assign it to the
OnBeforeSendingMessage event in Dialog.

Probably the one most experienced with Dialog's code and writing scripts
is Bernd Rose. Either wait for him to show up or start an new thread
titled "Attention Bernd Rose - Need your help with scripting."

> Script would search & replace four character before sent!

There used to be examples of scripts at:

http://www.40tude.com/dialog/wiki/index.php/Script_Library

That could give clues on how to write scripts inside of Dialog. Alas,
there was no documentation from the author on the scope of global
objects defined within the context of scripts or the vars used as
arguments in the function declaration, so Dialog will often report
errors in scripts. However, that domain is long gone.

I have a script attached to Dialog's OnBeforeSavingMessage event which
might provide a clue how to write one for the OnBeforeSendingMessage
event. Basically my script (which I got from an example in the old
Dialog script library site) traverses through the lines of the body of a
message to strip out the worthless PGP signatures. You would modify the
script to walk through each line and in each line then walk through
character by character until the EOL. You could use a compound 'if'
statement looking for a set of characters that, if found, would get
replaced with the character you wanted. Delphi has the 'case' command
that you could use instead of multiple ifs: you would walk through each
character in a line looking for a match on a set of characters and, if
found then uses the case command to determine what to do depending on
which character was found.

Pascal (on which Delphi is based) has its ReplaceStr() function. Maybe
you could walk through each line in the message and pass that string to
ReplaceStr() to replace all occurences of one character with another.
You could have 4 ReplaceStr() commands, one for each Unicode character
you are changing to the ASCII7 character. Delphi apparently has a
differently named function: stringreplace. See:

http://www.delphibasics.co.uk/RTL.asp?Name=stringreplace

The following is what I use for the OnBeforeSavingMessage event. Just
use it for clues on how to walk through lines in the body of the
message.

//Remove PGP signature from newsgroup messages

program OnBeforeSavingMessage;

const pgp_headline='-----BEGIN PGP SIGNED';
const pgp_sig_start='-----BEGIN PGP SIGNATURE-----';
const pgp_sig_end='-----END PGP SIGNATURE-----';
const pgp_sig_fix='- -- ';
const maxint=2147483647;

procedure OnBeforeSavingMessage(var Message: TStringlist; Servername:
string; IsEmail: boolean);
var i,j:integer;
s:string;

begin
if isemail=true then exit;

for i:= 0 to message.count-1 do

begin
if pos(pgp_headline, message.strings[i]) > 0 then
begin

// delete the '...PGP SIGNED...' line
message.delete(i);

// delete the 'Hash: foo' line now at index "i".
message.delete(i-1);

// insert notice
// message.insert(i,'---PGP sig removed---');
end;

if pos(pgp_sig_fix, message.strings[i]) > 0 then
begin

// delete the broken signature delimiter '- -- ' and replace it
with '-- '
message.delete(i);
message.insert(i,'-- ');
// exit the "i" loop
break;

end;
end;


//setup to delete pgp signature
s:=message.text;

//locate start of pgp signature
i:=pos(pgp_sig_start,s);

begin
//find length of pgp signature string
j := i + pos(pgp_sig_end,copy(s, i, maxint));
if j > i then
begin
//delete pgp signature string
delete(s, i, j-i + length(pgp_sig_end) +1);
message.text:=s;
end;
end;
end;

begin
end.

[Doyun Jahng]

Response to VanguardLH <V...@nguard.LH>

> Did you even try by deleting ALL allowed charset for your posts except
> us-ascii?

Yes. The fix will has nothing to do with red herrings.
I tried everything all ready. Charsets is not the problem.

The only solution is a to modify before sending.

Your suggest of search-&-replace script is good.
Use to be scripts at http://dialog.datalist.org/ but it expired.

Will first get regexp to work as that workaround is manual.

[Bernd Rose]

On Sun, 9th Sep 2018 13:31:49 +0900, Doyun Jahng wrote:

> Your suggest of search-&-replace script is good.
> Use to be scripts at http://dialog.datalist.org/ but it expired.

That's a very recent situation. Last week the website was up and running.
Maybe it is just a temporary outage. Be this as it might be, the website
fortunately is saved on archive.org. Have a look at:

http://web.archive.org/web/20120127150224/http://dialog.datalist.org:80/scripts/ScriptreplaceUmlaut.html

This can be quite easily adjusted to your needs.

Bernd

[Anton Shepelev]

VanguardLH:

> Pascal (on which Delphi is based) has its Re-

> placeStr() function. Maybe you could walk through
> each line in the message and pass that string to
> ReplaceStr() to replace all occurences of one

> character with another. You could have 4 Re-
> placeStr() commands, one for each Unicode charac-

> ter you are changing to the ASCII7 character.

That would be terribly clumsy and inefficient. Just
scan the string are replace offending characters in
one go:

begin
l := Length(s);
for i := 1 to l do
begin
case Ord( s[i] ) of
145: s[i] := Chr(39);
146: s[i] := Chr(39);
147: s[i] := Chr(34);
148: s[i] := Chr(34);
end;
end;
end;

--
() ascii ribbon campaign -- against html e-mail
/\ http://preview.tinyurl.com/qcy6mjc [archived]

[Anton Shepelev]

Beg pardon for the incomplete source. Here is the
entire function:

procedure FixString( var s: WideString );
var l, i: integer;

[Frank Slootweg]

Doyun Jahng <ahng...@tbroad.com> wrote:
[...]

> Syntax is same inside Linux (except Linux control+v is Windows control+q).
>
> May be you can help on gvim regexp syntax to get 1 command to work?
> On Windows gvim "vy" copies & "ga" identifies the curly character.
>
> The problem is to convert 4 character before sending 40tude message!
> convert open single quote d145 to straight single quote d39 (')
> convert close single quote d146 to straight single quote d39 (')
> convert open double quote d147 to straight double quote d34 (")
> convert close double quote d148 to straight double quote d34 (")
>
> The "digraph" percent sign (%) only work one at a time.
> open single quote (d145) to straight single quote (d39) -> :%s/\%d145/'/g
> close single quote (d146) to straight single quote (d39) -> :%s/\%d146/'/g
> open double quote (d147) to straight double quote (d34) -> :%s/\%d147/"/g
> close double quote (d148) to straight double quote (d34) -> :%s/\%d148/"/g
> I can not get the syntax correct to do all 4 conversion at once!
>
> The "\d" reduce the 4 command above to 2 command:
> open/close single quote to straight single quote -> :%s/[\d145,\d146]/'/g
> open/close double quote to straight double quote -> :%s/[\d147,\d148]/"/g
>
> But I can not understand yet syntax to make above to 1 command!

Others have already given Delphi-based suggestions/solutions, which is
probably the way to go.

As you mention gvim and Linux, I assume you might have other Unix
commands available (on Windows). If so, have a look at the tr(1)
command, 'translate or delete characters'. tr(1) is designed for these
kind of tasks and in gvim you can pipe the current text to tr(1).

[VanguardLH]

> const pgp_sig_start='

Hmm, not sure why my post got truncated. Here is the script that I use
(attached to a different event) to get rid of PGP signatures:

[VanguardLH]

VanguardLH wrote:

> Hmm, not sure why my post got truncated. Here is the script that I use
> (attached to a different event) to get rid of PGP signatures:
>
> //Remove PGP signature from newsgroup messages
>
> program OnBeforeSavingMessage;

> const pgp_sig_start='

Damn, happened again. Oh well, uploaded it. Here's a link to it:

https://1drv.ms/t/s!AgmxLZ7PJal8y3rYsl_eR3BPLQ9E

[VanguardLH]

You won't need regexp. In a script, you code the ifs or case statements
to do string substitution. Since the script gets attached to an event
in Dialog (OnBeforeSendingMessage), there is no manual effort as the
script gets ran on the document before Dialog submits it to the server.

[VanguardLH]

Bernd Rose wrote:

> On Sun, 9th Sep 2018 13:31:49 +0900, Doyun Jahng wrote:
>
>> Your suggest of search-&-replace script is good.
>> Use to be scripts at http://dialog.datalist.org/ but it expired.
>
> That's a very recent situation. Last week the website was up and running.
> Maybe it is just a temporary outage.

Well, not really that recent.

https://www.whois.com/whois/datalist.org

Domain registration expired back on 02-Sep-2017. While registrars often
allow a month's grace to renew a domain registration, they don't offer
longer grace periods.

> Be this as it might be, the website
> fortunately is saved on archive.org. Have a look at:
>
> http://web.archive.org/web/20120127150224/http://dialog.datalist.org:80/scripts/ScriptreplaceUmlaut.html
>
> This can be quite easily adjusted to your needs.

Ah, I forgot about the Web Archive site. Can be very handy at times.

[Doyun Jahng]

Response to VanguardLH <V...@nguard.LH>

> Damn, happened again. Oh well, uploaded it. Here's a link to it:
>
> https://1drv.ms/t/s!AgmxLZ7PJal8y3rYsl_eR3BPLQ9E

Maybe this work to show full file?
Last line should be "end." if it work.

//Remove PGP signature from newsgroup messages

program OnBeforeSavingMessage;

[Bernd Rose]

On Sun, 9th Sep 2018 20:45:25 -0500, VanguardLH wrote:

>>> Use to be scripts at http://dialog.datalist.org/ but it expired.
>>
>> That's a very recent situation. Last week the website was up and running.
>> Maybe it is just a temporary outage.
>
> Well, not really that recent.
>
> https://www.whois.com/whois/datalist.org
>
> Domain registration expired back on 02-Sep-2017. While registrars often
> allow a month's grace to renew a domain registration, they don't offer
> longer grace periods.

Obviously, in this case grace period has been a year. I successfully
accessed the website only a couple of days ago.

Bernd

[Bernd Rose]

On Sun, 9th Sep 2018 20:32:50 -0500, VanguardLH wrote:

>> The following is what I use for the OnBeforeSavingMessage event. Just
>> use it for clues on how to walk through lines in the body of the
>> message.
>>
>> //Remove PGP signature from newsgroup messages
>>
>> program OnBeforeSavingMessage;
>> const pgp_sig_start='
>
> Hmm, not sure why my post got truncated.

For all people except you, it didn't. ;-) Your /inbound/ PGP removal
searches for a string which happens to be inside your script. Because
the script doesn't check position of the string inside the line, it
deletes starting from "pgp_sig_start" declaration. (Plus 1 line back.)
Your "broken signature"-check also replaces not only the PGP-Sig, but
other occurrences of the PGP-Sig inside the texts, as well... (Cf.:
"if pos ... > 0" instead of "if pos ... = 1".)

Bernd

[VanguardLH]

Bernd Rose wrote:

> VanguardLH wrote:
>
>> https://www.whois.com/whois/datalist.org
>>
>> Domain registration expired back on 02-Sep-2017. While registrars
>> often allow a month's grace to renew a domain registration, they
>> don't offer longer grace periods.
>
> Obviously, in this case grace period has been a year. I successfully
> accessed the website only a couple of days ago.

What do you see there now? I see a cybersquatter's page.

Just to be sure, I used ICANN's WhoIs lookup, which is:

https://whois.icann.org/en/lookup?name=datalist.org

I also redid the WhoIs at the above link. They both list a *new*
(auto-renewed) domain registration.

Updated: 2018-09-03
Expires: 2019-09-02

The cybersquatter (sedoparking.com) grabbed the defunct domain
registration and is waiting for someone to buy it at an elevated price.
If you click on the squatter's link, they'll sell you that domain for
$149 USD. If the prior owner had renewed in time at 1and1, they
could've gotten that domain for another 7.45 years at that price.

https://help.1and1.com/domains-c36931/renew-domains-c85143/what-is-the-redemption-period-a594796.html

So, I was right about the 30-day grace period. That means when a domain
becomes a candidate for deletion, not when it absolutely must happen.
The grace period was over a long time ago. The old site was obviously
stagnant and non-maintained for a long time, its registrant let the
domain slide away. Since it was a candidate for expiration but
[accidentally still] within the auto-renew period, a predator came along
to snatch it up to cybersquat on it hoping someone is willing to pay.
1and1 would've charged $90 (plus the domain registration fee) if the
original owner finally noticed the expiration and had renewed themself.

https://www.whois.com/whois/sedoparking.com

Looks like sedoparking takes serious their privacy. However, not
listing a registrant violates ICANN's requirement for such. ICANN
requires a responsible and active contact be listed for domain
registrations. Registrars may offer a private service, usually at an
extra fee, where they circumvent ICANN's requirement by listing the
registrar as the domain registrant and as the contact. Not listing the
real registrant or the registrar violates ICANN's rules. You get the
same shit (REDACTED FOR PRIVACY) when looking up registrygate.com. Even
the registrar refuses to identify themself. Makes me wonder if the two
(registrar and cybersquatter) are the same. In fact, if you go to
registrygate.com (and translate into English), it says:

The RegistryGate GmbH is an internationally operating purchasing
company with the focus on the procurement and administration of
domains.

Then I found:

http://domainincite.com/23007-icann-approves-messy-unfinished-whois-policy

So lookups on domain registrations to see who is the registrant will
eventually become worthless. Everyone can hide, by default. So WhoIs
will become and is becoming worthless. As such, ICANN accredidation of
registrars is also becoming worthless. This is a result of the EU's
General Data Protection Regulation (GDPR) fucking over the domain system
and ICANN becoming compliant despite private registrations already let
registrants hide their identity (which was a red flag for some domains).

https://gdpr-info.eu/
https://en.wikipedia.org/wiki/General_Data_Protection_Regulation

[VanguardLH]

Bernd Rose wrote:

> VanguardLH wrote:
>
>>> The following is what I use for the OnBeforeSavingMessage event. Just
>>> use it for clues on how to walk through lines in the body of the
>>> message.
>>>
>>> //Remove PGP signature from newsgroup messages
>>>
>>> program OnBeforeSavingMessage;
>>> const pgp_sig_start='
>>
>> Hmm, not sure why my post got truncated.
>
> For all people except you, it didn't. ;-) Your /inbound/ PGP removal
> searches for a string which happens to be inside your script.

Ooh, you're right (slaps forehead).

[VanguardLH]

Don't remember if I got this script from the Datalist script library for
Dialog or from elsewhere. I didn't write it.

Seems the script is wrong, as you pointed out. It finds the first line
*containing* "-----BEGIN PGP SIGNED" as a /substring/. It should check
that substring starts at column 1. I thought the PGP sig block was
always at the end of the message, so just delete everything from and
after the "-----BEGIN PGP SIGNATURE----" line.

Examples:
http://www.heavensgate.com/misc/bg2pgp.txt
https://philzimmermann.com/text/PGP_10thAnniversary.txt

Those have a PGP message block delimiter line as the first line in the
message and starting at column 1. Seems I only need to check if the
first line's entire string is the PGP message block delimiter line, not
by having to check all the lines in the message.

One example doesn't have a following "Hash: <type>" line, so I should
delete it only if a prior delete got rid of the PGP message block
delimiter line.

Looking for the pgp_sig_start substring anywhere within message.text
also seems wrong. That will look for the substring anywhere in the
message, not for it to start in column 1 and not only after the PGP
message block delimiter line. I don't know if the delete() string
command can be modified to look for the position of a concatenation of
EOL + pgp_sig_start (or, if EOL isn't recorded at the end of strings
then maybe NUL + pgp_sig_start if strings are NUL terminated). That
position + 1 (one character after the EOL or NUL) would mark the string
position where the PGP sig block started.

https://www.rossde.com/PGP/pgp_signatures.html

That shows the "-----BEGIN PGP SIGNED MESSAGE-----" delimiter line is
always in line 1 (starting in column 1) and the PGP sig block is always
at the end of the message.

I'm assuming in Delphi that message.strings[] is an array of strings
(don't know if terminated by EOL or NUL) where each string is a line in
the message with an index value starting at 0 (not 1). Seems the whole
"delete PGP sig" function should be conditional on first finding the PGP
message block delimiter line, and that every line from and after the
starting PGP sig delimiter can get deleted. Maybe the following works:

// Check if line 0 starts a PGP message block.
if message.strings(0) = '-----BEGIN PGP SIGNED MESSAGE-----' then
begin
// Delete PGP message block delimiter line.
message.delete(0);
// Replace with a comment that PGP sig was removed.
message.insert(0, '*****PGP SIGNATURE REMOVED*****'); // Optional.
// If present as the next line, delete the Hash specifier.
if message.strings(1) = 'Hash: ' then
message,delete(1);
// Delete the PGP signature block at the end.
sig_found := 0;
for i := 1 to message.count - 1 do
begin
if message.strings(i) = '-----BEGIN PGP SIGNATURE-----' then
sig_found := 1;
// If sig start delimiter line found, delete it and the rest.
if sig_found then message.delete(i);
end;
end;
end;

The 'if' checks if the first line is the PGP message block delimiter
line; else, there's no point in checking if the next line is a Hash line
and also no point looking for a PGP sig block. Everything rests on
finding that required delimiter line that delineates the start of the
PGP message block.

The for-loop checks for if and when the PGP sig block starts. While I
could've added another for-loop inside that for-loop to do the deletes,
I didn't see much impact in retesting each further line. I could
possibly short-circuit the 'if' by doing a boolean AND between the
string (line) value and sig_found assuming Pascal/Delphi supports short
circuiting of Boolean operators (e.g., FALSE AND ..., don't need to test
anything beyond FALSE since the entire Boolean will be FALSE).

If there can be mixed content where the PGP message block starts later
in the message (not in the first line) then I'll have to use the
for-loop to walk through lines in the message. If there can be
non-signed content after the PGP sig block, yeah, then I have to find
the length of just the sig block and delete just those lines.

I don't do Delphi, so I've probably made mistakes. I haven't touched
Pascal for over 40 years and that was before it got objectified.