We are now spam

[Jeremy Morse]

Hi,

The emails sent when account registration emails go out has the return
path terminating at my email address; thus I was distressed to see a
bounce with the following status message:

550 Email has been rejected as account...@studentrobotics.org is
blacklisted by Sorbs RBL

Sorbs claim to only blacklist hosts, which can't be right as we're
sending through gmail, but never mind. I've seen one other bounce with
the status message "Access denied", but that might not be due to spam
listing.

Last year, when people registered multiple accounts, they got one status
message per user registered. HRS won a prize for registering 19 people
in one go, but that led to the teacher there receiving a train of 19
emails to himself immediately. I'd suggest that it's this status email
train that could be what flagged us as spam.

The specific school that's returning these messages is HAM, looking
through the nemesis email log 4 users were registered, leading to 4
emails to individual students, and 4 to the teacher. The following trace
occurred (reading gmail's mail log), with students A, B, C, D

2014-09-19T12:10:02 Student-email A Accepted
2014-09-19T12:10:03 Teacher-email A Accepted
2014-09-19T12:10:04 Student-email B Bounced
2014-09-19T12:10:05 Teacher-email B Bounced
2014-09-19T12:10:06 Student-email C Bounced
2014-09-19T12:10:07 Teacher-email C Bounced
2014-09-19T12:10:08 Student-email D Accepted
2014-09-19T12:10:09 Teacher-email D Bounced

No emails have been sent to Hampton school before (this SR year, at
least). It also didn't happen last year. The headers suggest gmail
didn't store-and-reorder these emails.

This kind of suggests that what we're looking at is some form of step
response to a lot of incoming email that gets reset after five seconds.
Seeing how that's massively dumb, if it's true there's nothing we can do
about it.

I think for the moment we can deal with HAM manually. If more emails
start bouncing, that's much more problematic. I would prefer to
eliminate the email-train for teachers that gets generated, as it's only
going to make the situation worse.

There's also the risk that we've been silently flagged as spam by
google. Generating SR accounts for my plethora of gmail-based email
addresses suggests this isn't the case yet.

--
Thanks,
Jeremy

[Peter Law]

Jeremy wrote:
> Last year, when people registered multiple accounts, they got one status
> message per user registered. HRS won a prize for registering 19 people
> in one go, but that led to the teacher there receiving a train of 19
> emails to himself immediately. I'd suggest that it's this status email
> train that could be what flagged us as spam.

If we think that this is the likely cause, then given that we
currently cache the emails before sending them at 5 minute intervals,
we could possibly add something which would detect that there are
several emails on the same template being sent to the same person, and
collapse them into a single email with the relevant content tabulated.

Peter

[Jeremy Morse]

Hi,

On 19/09/14 19:12, Peter Law wrote:
> If we think that this is the likely cause, then given that we
> currently cache the emails before sending them at 5 minute intervals,
> we could possibly add something which would detect that there are
> several emails on the same template being sent to the same person, and
> collapse them into a single email with the relevant content tabulated.

This sounds like a plan. Speculation: perhaps a message queue or other
super-newfangled-tech is the most appropriate tool for this job in the
long run? Seeing how it's a queue of things to do that might get
condensed or expanded. Genuine question, as I hear about MQ's frequently
but never really know what they do.

I pumped our domain name and host address into a bunch of online
mx-reputation checking tools, and they all came back clean. This, along
with the fact that the HAM mail server accepted an email in the middle
of a chain of bounces has me leaning in the direction that their
mailserver is misconfigured or dumb. So I would now rate this as
slightly lower priority, until some other team reports trouble.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

We're now on a whitelist for Hampton. The sysadmin guy there asserts the
spam filter fired; I can't find badger in Sorbs' database anywhere, or
any other blacklist. I shall leave this as a mystery.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

Random traffic from trac is now turning up flagged as 'Spam' by google.
This is bad because gerrit and trac emails are sent for the same account
(trac@). This is not the same account as student/teacher emails, which
is account-mailer@. It's also not because of the competition-ticket
import, as the xmlrpc call for that doesn't have the send-notification
flag set.

Saffron is not on any blacklists, according to mxtoolbox.com.

It's unclear what the next steps are; configuring SPF correctly [0] will
reduce our spam score, but otherwise the spam-score world is extremely
murky from my perspective.

[0] Which I said I was going to do about 3 years ago but never did

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

A teacher replied back to me today with a 'spam score 8%' arbitrarily
added to a subject line by his mail system, which got me thinking about
spam again. After some searching I found mail-tester.com, which
apparently is a default spamassassin and blacklist/whitelist system that
you can email and then look at how you score for spam.

I did that with a pending email to students (g:2371), and got a score of
"6/10", which broke down into the following negative points:

* There's no SPF record in our DNS
* Same for 'Sender ID'
* Same for DKIM

Which all appear to be competing anti-spam sender-authentication
facilities. This was previously recognized (#1712) but I never had time
to do anything about it. Once again, we can't fix that in the
competition period and risk all our email not working for a week.

Slightly more concerningly, saffron is on a blacklist called
CASA-CBLPLUS. However checking blacklists on mxboolbox.com for saffron
says that the same blacklist does not contain saffrons address. So I'm
inclined to ignore this.

Conclusion: more DNS records equals less spam.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

I'm currently investigating whether google postmaster tools [0] will
yield some more information about spam scoring of our email. To verify
ownership, I've had to put a TXT record in DNS -- if you see that,
that's me.

[0] postmaster.google.com

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

On 11/07/15 14:43, Jeremy Morse wrote:
> I'm currently investigating whether google postmaster tools [0] will
> yield some more information about spam scoring of our email. To verify
> ownership, I've had to put a TXT record in DNS -- if you see that,
> that's me.

Turns out it's a dummy unless you send about 1000 emails a day. While I
was at it, I've inserted some SPF records that should improve our spam
scores. If your @studentrobotics.org email doesn't get through in the
next couple of days, it's because I've screwed that up somehow.

DKIM involves some kind of key generation situation that I'm not going
to touch without some thought about how we keep it in backups. It looks
like a public part goes in DNS, private in google apps config, so
nothing touches our systems at all.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

On 11/07/15 15:30, Jeremy Morse wrote:
> Turns out it's a dummy unless you send about 1000 emails a day. While I
> was at it, I've inserted some SPF records that should improve our spam
> scores. If your @studentrobotics.org email doesn't get through in the
> next couple of days, it's because I've screwed that up somehow.

Sitrep: linode is not serving the TXT record I registered for SPF [0].
This is odd, because google picked up the postmaster-authentication
record I put there. "dig @ns1.linode.com studentrobotics.org TXT", "host
-t TXT studentrobotics.org" don't report any TXT records either.

IMO, this justifies a support ticket. It's been 24h since I added that
record, TTL is set to 5 mins, after 48 hours there should be no excuse
for it not being propagated.

[0]
http://mxtoolbox.com/SuperTool.aspx?action=txt:studentrobotics.org&run=toolpage

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

On 12/07/15 16:45, Jeremy Morse wrote:
> Sitrep: linode is not serving the TXT record I registered for SPF [0].
> This is odd, because google picked up the postmaster-authentication
> record I put there. "dig @ns1.linode.com studentrobotics.org TXT", "host
> -t TXT studentrobotics.org" don't report any TXT records either.
>
> IMO, this justifies a support ticket. It's been 24h since I added that
> record, TTL is set to 5 mins, after 48 hours there should be no excuse
> for it not being propagated.

Remembered this; we're still not getting any TXT records served for
studentrobotics.org. Exactly the same configuration for jmorse.net works.

After eyeballing DKIM it looks like we just need to:
* Generate a keypair
* Load the private part into a gmail web form
* Load the public part into DNS TXT

Google will sign all outbound email; mailservers seeing the public key
in DNS will expect a corresponding signature. In terms of backups,
there's nothing on our own systems that needs this information. If one
part goes missing, we just re-generate a key, it doesn't need a signature.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

On 07/08/15 00:19, Jeremy Morse wrote:
> After eyeballing DKIM it looks like we just need to:
> * Generate a keypair
> * Load the private part into a gmail web form
> * Load the public part into DNS TXT
>
> Google will sign all outbound email; mailservers seeing the public key
> in DNS will expect a corresponding signature. In terms of backups,
> there's nothing on our own systems that needs this information. If one
> part goes missing, we just re-generate a key, it doesn't need a signature.

Oh: google already have a key generated and are already signing email
with it. Looks like we just put the public key in DNS to enable DKIM.
This also means there's actually no key management whatsoever that we
have to be concerned with.

--
Thanks,
Jeremy

[Jeremy Morse]

Hi,

Rob kindly pointed out to me that I'd DNS'd wrong; after wrangling that
a little further, both SPF and DKIM appear to work. I get the following
in mail headers sent via google:

Authentication-Results: mx.google.com;
spf=pass (google.com: domain of jmo...@studentrobotics.org
designates 2a00:1450:400c:c05::233 as permitted sender)
smtp.mail=jmo...@studentrobotics.org;
dkim=pass header.i=@studentrobotics.org

Which seems good. As mentioned, if you're sending email through
something that isn't google now, it might fail, or will be more likely
to be flagged as spam. If you have problem with email in the next week,
please let me know, via something that isn't student robotics mail.

This has improved our score on mail-tester.com to 8.5. We get an
additional -1.5 score because apparently we're on a blacklist -- closer
examination shows that it's googles addresses, not ours.

--
Thanks,
Jeremy