OT: email woes

[Neil Rieck]

Had a problem this week where a specialized low-level app in my
OpenVMS system could no longer send SMTP messages to pagers (had been
doing this for 5 years). Next, I ran a DCL script from years ago that
would send 10 unique messages (with unique subject fields) to my phone
(without using the low-level app). This script used to work but now
only the first message got through. Obviously someone turned on some
sort of SPAM detection and/or message throttling at the pager company
but no one will fess-up to making any changes.

Next I decided to create 10 dissimilar emails with Windoz-Outlook then
send them out all in one go and this also worked. Okay, time to do a
little reverse engineering. One thing I noticed was some weird new (to
me) tags in the OUTLOOK mail message. They were “x-cr-puzzleid:” and
“x-cr-hashed-puzzle:”

A number of years ago I read a MS article about developing paid
postage for emails. I think the project was called “Penny Black”. The
theory goes like this: making SPAMMERS buy a stamp for each email will
stop spam. Knowing it would be difficult to collect the postage etc
even for fractional pennies, MS decided to create a CPU-expensive
stamp. The onus would be on the sending email client to waste a few
hundred CPU cycles creating a unique stamp for each message before
transmitting.

Anyway, that’s exactly what the “x-cr” tags appear to be and know we
know why OUTLOOK email clients are so damned slow :). I started
searching the web for the algorithm when I learned that Microsoft had
published the stuff and put it into the public domain (I think). Check
this these two links:

[MS-OXPSVAL]: E-Mail Postmark Validation Algorithm
http://msdn.microsoft.com/en-us/library/cc433493%28v=exchg.80%29.aspx

File: [MS-OXPSVAL].pdf
http://download.microsoft.com/download/5/D/D/5DD33FDF-91F5-496D-9884-0A0B0EE698BB/%5BMS-OXPSVAL%5D.pdf

Neil Rieck
Kitchener / Waterloo / Cambridge,
Ontario, Canada.
http://www3.sympatico.ca/n.rieck/OpenVMS.html

[Bob Gezelter]

> File: [MS-OXPSVAL].pdfhttp://download.microsoft.com/download/5/D/D/5DD33FDF-91F5-496D-9884-...

>
> Neil Rieck
> Kitchener / Waterloo / Cambridge,
> Ontario, Canada.http://www3.sympatico.ca/n.rieck/OpenVMS.html

Neil,

I for one would like to see a TCPDUMP (or similar) trace of the
conversation between your script and the mail server. Correlation is
not causality.

- Bob Gezelter, http://www.rlgsc.com

[Neil Rieck]

I wouldn't have messed around with the DCL script unless the customers
had complained that messages were not getting through to phones. To
make matters worse, there seems to be a lot more Microsoft email
servers around here than before (paranoia? yes!)

NSR

[JF Mezei]

Neil Rieck wrote:

> I wouldn't have messed around with the DCL script unless the customers
> had complained that messages were not getting through to phones. To
> make matters worse, there seems to be a lot more Microsoft email
> servers around here than before (paranoia? yes!)

My NTS service seems to have "misplaced" a whole bunch of messages
today. (tagged as expired or no longer availabl). But I think I have
been able to read enough quoted messages to see.


There are logicals you can set which will log conversations between the
SMTP server and the outside world.

Some HOTMAIL type of systems will pretend they receive the message and
confirm reception and then decided not to do anything with them.

But you need to have the logs for to prove that you succesfully sent the
message and that the receiving SMTP server accepted it.


http://www.vaxination.ca/vms/tcpip_smtp_systartup.com


you want to DEFINE/SYSTEM TCPIP$SMTP_NOSEY

and more importantly.
TCPIP$SYMB_TRACE


You need to TCPIP STOP MAIL
wait a few instants
TCPIP START MAIL

(if I recemember correctly) for the new logicals to take effect).

You can then run your tests, discover which message was not reeived by
the pager, and then look at the logs and you will see the exact
conversation which sent the message.

The SYMB_TRACE result infar easier logs to read than a TCP level tracing.

[Neil Rieck]

All the email messages sent by my low-level pager-app were logged (log
changes every day; we only keep the last 14 days). All the messages
transmitted by our system were accepted by the remote SMTP server.
Something (we think an internal SPAM detector) is diverting certain
messages into the bit-bucket.

Things have settled down since I added a "Message-id:" field to the
MIME block which is formatted like this:
Message-ID: <gu...@server-name.domain>
I suppose this makes each email more unique.

These emails (which go to employee cell phones) contain safety check-
in reminders. (In Ontario, all roving employees must check in every
two hours so we send out the first reminder at the 90 minute mark).
Things really stabilized (less email was lost) when I appended a
unique word to the end of each email body. The word looks like this:
"(CODE:"+ another GUID +")"
so the SPAM detector (if it exists) must be also scanning email bodies
looking for similarities.

NSR

[onedbguru]

>     Message-ID: <g...@server-name.domain>

> I suppose this makes each email more unique.
>
> These emails (which go to employee cell phones) contain safety check-
> in reminders. (In Ontario, all roving employees must check in every
> two hours so we send out the first reminder at the 90 minute mark).
> Things really stabilized (less email was lost) when I appended a
> unique word to the end of each email body. The word looks like this:
>     "(CODE:"+ another GUID +")"
> so the SPAM detector (if it exists) must be also scanning email bodies
> looking for similarities.
>
> NSR

A lot of ISP's have implemented things like they no longer allow out-
bound SMTP (port 25) and to transmit email you must use their server
which requires authenticated TLS or SSL outbound email and inbound pop
and IMAP on non-standard ports. A lot of companies are rejecting SMTP
port 25 (and have been for some time) as well. The spammers have
totally screwed up protocols by being such stupid morons - now the
"good guys" have to pay the price for their idiocy.

[Phillip Helbig---undress to reply]

In article
<7348bf0f-26da-4370...@y10g2000vbn.googlegroups.com>,

onedbguru <oned...@yahoo.com> writes:

> A lot of ISP's have implemented things like they no longer allow out-
> bound SMTP (port 25) and to transmit email you must use their server
> which requires authenticated TLS or SSL outbound email and inbound pop
> and IMAP on non-standard ports.

At least one dynamic-DNS provider, DynAccess.com, offers an SMTP relay
server which doesn't require anything fancy. Since they know your IP
address (from a dynamic-DNS update), you can connect to their SMTP
server and send from there. Incoming is fine (though they also offer
routing it through their computers and running a spam- and virus-checker
if you want.

> A lot of companies are rejecting SMTP
> port 25 (and have been for some time) as well. The spammers have
> totally screwed up protocols by being such stupid morons - now the
> "good guys" have to pay the price for their idiocy.

Right. I see no alternative but to use spamhaus.org as an RBL and
reject everything in there (as well as dropping connections for
non-existent users etc).

[Jan-Erik Soderholm]

My ISP dropped support from non-AUTH smtp a couple of months ago.
So my outgoing email from the VMS system (using the standard smtp
tools in TCP/Services) stopped.

I found that the Python port had some smtp tools included
that support smtp with optional AUTH and/or TLS/SSL. I use that
now for outgoing emails from my system using a short Python
script.

Support for AUTH in TCP/Services would have been nice...

Jan-Erik.

[George Cornelius]

Neil Rieck wrote:
> All the email messages sent by my low-level pager-app were logged (log
> changes every day; we only keep the last 14 days). All the messages
> transmitted by our system were accepted by the remote SMTP server.
> Something (we think an internal SPAM detector) is diverting certain
> messages into the bit-bucket.
>
> Things have settled down since I added a "Message-id:" field to the
> MIME block which is formatted like this:
> Message-ID: <gu...@server-name.domain>
> I suppose this makes each email more unique.
>
> These emails (which go to employee cell phones) contain safety check-
> in reminders. (In Ontario, all roving employees must check in every
> two hours so we send out the first reminder at the 90 minute mark).
> Things really stabilized (less email was lost) when I appended a
> unique word to the end of each email body. The word looks like this:
> "(CODE:"+ another GUID +")"
> so the SPAM detector (if it exists) must be also scanning email bodies
> looking for similarities.

Neil,

Is it possible it is just that they decided to change
to paging only on unique messages, perhaps due to cases
where multiple copies of the same message were being
generated at a site and annoying pager users?

Paging, after all, has some unique aspects to it. I remember
being new to my current team (Unix system administration),
when they were using a product called Big Brother to monitor
systems. The first time I had to take call I was awakened
at three in the morning by my pager and literally watched
the asterisks (one asterisk = one page) shoot across the
screen as pages came in rapid-fire. Ultimately I saw just
this on the pager screen:

MEMORY OVERFLOW

It could have been the application in a loop, but perhaps
it was a backlog of relatively low priority events/pages
that hed built up in a queue over several days and then
suddenly been released. I don't know if a duplicate
check would have fixed that one, but all it takes is one
instance where it does and you have a customer calling
in and asking the pager company why they even pass on the
duplicates.

George

[Neil Rieck]

On Feb 9, 4:08 pm, George Cornelius <cornel...@eisner.decus.org>
wrote:

> Neil Rieck wrote:
> > All the email messages sent by my low-level pager-app were logged (log
> > changes every day; we only keep the last 14 days). All the messages
> > transmitted by our system were accepted by the remote SMTP server.
> > Something (we think an internal SPAM detector) is diverting certain
> > messages into the bit-bucket.
>
> > Things have settled down since I added a "Message-id:" field to the
> > MIME block which is formatted like this:

> >     Message-ID: <g...@server-name.domain>

> > I suppose this makes each email more unique.
>
> > These emails (which go to employee cell phones) contain safety check-
> > in reminders. (In Ontario, all roving employees must check in every
> > two hours so we send out the first reminder at the 90 minute mark).
> > Things really stabilized (less email was lost) when I appended a
> > unique word to the end of each email body. The word looks like this:
> >     "(CODE:"+ another GUID +")"
> > so the SPAM detector (if it exists) must be also scanning email bodies
> > looking for similarities.
>
> Neil,
>
> Is it possible it is just that they decided to change
> to paging only on unique messages, perhaps due to cases
> where multiple copies of the same message were being
> generated at a site and annoying pager users?
>

I really think it has something to do with SPAM detection at one
company (although no one will admit to making any changes).

Here is what was going on in my case. Within one minute, my system
sent three different emails to three different technicians telling
them to report to the safety coordinator. The DESTINATION and SUBJECT
were unique but the BODYs were identical ("Safety Alert. Please call
1-800-xxx-xxxx ASAP"). According to our logs, all three messages were
accepted by the SMTP server but only the first one makes it all the
way to the field employee's cellphone. The other two are never
delivered (at least according to the employees).

Anyway, I seem to have solved the problem (at least for now) by
inserting a GUID into both the email header like this:

Message-ID: <EC346946-7651-E111...@www.bellics.com>

as well as appending a different GUID onto the message body like this:

(CODE:7DC26846-7651-E111-B9E2-00062B0160A2)

###

Now here is what is worrying me. For some reason I can't understand,
Microsoft Exchange servers (with an open SMTP port) seem to be popping
up all around me. Email coming to me from various directions many
times contain two new (to me) tags which are: “x-cr-puzzleid:” and “x-
cr-hashed-puzzle:”.

see: http://msdn.microsoft.com/en-us/library/cc433493(v=exchg.80).aspx
which contains a link to the very weirdly named file "[MS-
OXPSVAL].pdf"

Now I'm a mostly agnostic coder who realizes that resistance is
(mostly) futile. If Micro$oft has stumbled onto a solution to reduce
SPAM, and industries around me are buying into this solution, then I'm
going to need to to start coding routines to place a e-stamp on emails
issued by my OpenVMS software. That is, unless someone else has done
it first :-)

[George Cornelius]

Neil Rieck wrote:
> I really think it has something to do with SPAM detection at one
> company (although no one will admit to making any changes).
>
> Here is what was going on in my case. Within one minute, my system
> sent three different emails to three different technicians telling
> them to report to the safety coordinator. The DESTINATION and SUBJECT
> were unique but the BODYs were identical ("Safety Alert. Please call
> 1-800-xxx-xxxx ASAP"). According to our logs, all three messages were
> accepted by the SMTP server but only the first one makes it all the
> way to the field employee's cellphone. The other two are never
> delivered (at least according to the employees).
>
> Anyway, I seem to have solved the problem (at least for now) by
> inserting a GUID into both the email header like this:
>
> Message-ID: <EC346946-7651-E111...@www.bellics.com>
>
> as well as appending a different GUID onto the message body like this:
>
> (CODE:7DC26846-7651-E111-B9E2-00062B0160A2)

Yes, but those do not seem to make messages less likely, from a
recipient's point of view, to be spam, although you might imagine
filtering software docking you in some spam rating scheme for
missing certain standard headers.

> ###
>
> Now here is what is worrying me. For some reason I can't understand,
> Microsoft Exchange servers (with an open SMTP port) seem to be popping
> up all around me. Email coming to me from various directions many
> times contain two new (to me) tags which are: x-cr-puzzleid: and x-
> cr-hashed-puzzle:.
>
> see: http://msdn.microsoft.com/en-us/library/cc433493(v=exchg.80).aspx
> which contains a link to the very weirdly named file "[MS-
> OXPSVAL].pdf"

Organized in a way that is difficult to read, but it looks as if the
idea is to use a variant of SHA-1 (sosha1) on a collection of fields
derived from the message headers, and possibly from the message itself.

Surprising how little it is discussed outside Microsoft or Microsoft
Outlook discussion groups, even though they have made the specifications
public and, apparently, removed any intellectual property rights
restrictions. From what I can gather, if you enabled the option in
Outlook 2007 (and had applied appropriate patches to earlier versions),
it would automatically add the x-cr-puzzleid and x-cr-hashedpuzzle
headers to any message it considered might not pass spam filters
at the receiving end. This was billed as making spam filtering more
robust with regard to false positives.

Without more evidence, including a diagram of the path the mail
uses to get to the pager operator and the results of a few simple
tests you could try, I am not convinced that those headers really
have much to do with your problem, but perhaps your message format
looks like spam and tinkering with it, or adding these headers,
solves that.

Here at EISNER, out of hundreds of emails I have filed away since
2008, only three have these headers and they all come from the
same sender at chase.com . I see that of all the messages I
have saved away on my VMS system at work, only a few dozen have
the headers, and these are all messages going to a large
distribution list - both messages received and sent, presumably
all sent via Outlook.

George