Trac notification emails

[Michael Orlitzky]

Can we please change how our Trac notification emails are sent? SendGrid
is absolutely atrocious. I currently have six of their shared IPs
whitelisted on our mail server to allow these notifications through,
because they would otherwise be blocked by the many many many blacklists
that SendGrid is always on for sending spam. Here are those IPs, and the
number of blacklists that they're on right now (it fluctuates).

Six blacklists:

https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a167.89.100.130
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a167.89.100.175

Four blacklists:

https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a167.89.100.176

Three blacklists:

https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a167.89.100.129
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a168.245.72.219
https://mxtoolbox.com/SuperTool.aspx?action=blacklist%3a198.21.6.101

Those are all IPs that are actively sending Trac notifications. There
are two problems with this:

1. I don't want to be whitelisting spammers on our mail server.

2. Every once in a while, SendGrid will pick a new IP to start
sending Trac notifications from, and the only way I know to
whitelist them is that I start missing important notifications.

It's impossible to do worse than this with a five-minute outgoing-only
local postfix instance. You get a PTR record for the server, make sure
it's not on any blacklists, and pick one poor sucker to receive the
"bounced" mail (when someone's Trac email address stops working, we need
to know and disable it). You might get rate-limited by Microsoft/Gmail
at first (what kind of volume are we talking about?), but those
notifications won't get lost forever, and that problem eventually
corrects itself unlike this one. And it's free.

Here's the entire postfix main.cf for such an instance:

compatibility_level = 2
inet_protocols = ipv4
home_mailbox = .maildir/
myhostname = hostname.example.com
smtp_skip_5xx_greeting = no
unknown_address_reject_code = 550
fast_flush_domains =
error_notice_recipient = postm...@example.com

Then postm...@example.com would go to whoever is in charge of the server.

[Dima Pasechnik]

We talk about 20-40K emails per month.
I believe that one would need a properly configured host
with correcrty set up DNS, SPF, DKIM, etc etc to make many mail servers
happy, as most people have no control over email whitelisting.
And it has to be on a reasonably fast and reliable network, where you
are able to ask admins to allow for an outgoing SMTP.
(here goes "free", unless you are able to utilize resources of some
organisation like a university or a company, or an non-profit
like a Linux distributor, say. A dedicated server
would still cost ~100$ per year at least, I guess, something we can
afford, though)
And it needs to be set up and maintained.

Any volunteers for this job? Michael?

Dima

>
> Here's the entire postfix main.cf for such an instance:
>
> compatibility_level = 2
> inet_protocols = ipv4
> home_mailbox = .maildir/
> myhostname = hostname.example.com
> smtp_skip_5xx_greeting = no
> unknown_address_reject_code = 550
> fast_flush_domains =
> error_notice_recipient = postm...@example.com
>
> Then postm...@example.com would go to whoever is in charge of the server.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/97bf2d72-6793-afc8-ef86-3a88acf07596%40orlitzky.com.

[Michael Orlitzky]

On 4/25/20 8:28 PM, Dima Pasechnik wrote:
>
> We talk about 20-40K emails per month.
> I believe that one would need a properly configured host
> with correcrty set up DNS, SPF, DKIM, etc etc to make many mail servers
> happy, as most people have no control over email whitelisting.
> And it has to be on a reasonably fast and reliable network, where you
> are able to ask admins to allow for an outgoing SMTP.
> (here goes "free", unless you are able to utilize resources of some
> organisation like a university or a company, or an non-profit
> like a Linux distributor, say. A dedicated server
> would still cost ~100$ per year at least, I guess, something we can
> afford, though)
> And it needs to be set up and maintained.
>
> Any volunteers for this job? Michael?
>

20-40k per month isn't much. SPF and reverse DNS are trivial to set up,
and you don't need DKIM. (That's not too hard either, but DKIM/DMARC
often just cause your messages to get rejected again.)

If the Trac web server has a static IP, all you would need to do is
install an MTA on it and tell Trac to use the local sendmail command
instead of connecting to SendGrid over SMTP to send these messages. That
shouldn't cost anything extra if Trac is on a dedicated server, but I've
no idea how that is set up. If it can send mail, and if we can change
the PTR record for the IP, that's all we'd need.

The goal here is just to do better than servers that are on six
blacklists... it doesn't take a dedicated team of full-time experts to
do that.

[Dima Pasechnik]

Trac server is hosted on Google Compute Engine (or whatever it's
called now), which
disallows outgoing SMTP.
(also, it seems that the IP may change over time).
That's why we went with SendGrid as Trac was being moved our of UW hosting.

>
> The goal here is just to do better than servers that are on six
> blacklists... it doesn't take a dedicated team of full-time experts to
> do that.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/abb652b8-c25c-2ced-f9f8-f3d26e250ede%40orlitzky.com.

[William]

On Saturday, April 25, 2020 at 6:27:21 PM UTC-7, Dima Pasechnik wrote:

Trac server is hosted on Google Compute Engine (or whatever it's
called now), which
disallows outgoing SMTP.

True. Docs here: https://cloud.google.com/compute/docs/tutorials/sending-mail

 

(also, it seems that the IP may change over time).

Fortunately not true.  I just checked the GCE console, and the ip addresses for our trac and wiki are static, so they do not change over time.   (If it were really changing over time then trac.sagemath.org would periodically completely break until either Harald Schilly or I manually change the DNS configuration, and fortunately I never remember doing that.)

We pay $15/month for a plan that lets trac.sagemath.org's send up to 40K emails (they charge when we go over).  During the last month, trac sent 23,237 emails.     Maybe there is a really good competitor for email today.  That page I linked to above says "SendGrid, Mailgun, and Mailjet offer a free tier for Compute Engine customers to set up and send email through their servers."   I have not investigated Mailgun or Mailjet, but would love to hear from Michael Orlitzky if either one of those are a reasonable option.

Thanks,

 William 

[Dima Pasechnik]

AWS SES is a better option money-wise, see https://aws.amazon.com/ses/pricing/
Also, note that GCE has an option of sending emails via G-suite,
perhaps if someone already has G-suite it might come out better.


Best
Dima

> Thanks,
>
> William

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/1b159b31-b87d-4422-a9af-99c159e18d19%40googlegroups.com.

[Harald Schilly]

On Sun, Apr 26, 2020 at 7:27 AM Dima Pasechnik <dim...@gmail.com> wrote:

Also, note that GCE has an option of sending emails via G-suite,
perhaps if someone already has G-suite it might come out better.

G-Suite has an email relay server, where the limit is 10k per day, more or less. It's not hard to set it up.
https://support.google.com/a/answer/2956491?hl=en

If you have 2FA enabled, you need to create an "app specific password" just for sending emails, and use that as the password.
https://support.google.com/accounts/answer/185833

-- harald

 

[Michael Orlitzky]

On 4/26/20 12:11 AM, William wrote:
>
> We pay $15/month for a plan that lets trac.sagemath.org's send up to 40K
> emails (they charge when we go over).  During the last month, trac sent
> 23,237 emails.     Maybe there is a really good competitor for email
> today.  That page I linked to above says "SendGrid, Mailgun, and Mailjet
> offer a free tier for Compute Engine customers to set up and send email
> through their servers."   I have not investigated Mailgun or Mailjet,
> but would love to hear from Michael Orlitzky if either one of those are
> a reasonable option.

"If what they're doing isn't working, tell them to do something else."

The other two can't be any worse.

I didn't know that the web server was unable to send email. That
precludes the easy solution. It looks like google is just passing the
buck to SendGrid/Mailgun/Mailjet who then allow GCE customers to send
whatever they want, so that any abuse that would have taken place on GCE
now takes place on those three platforms.

Now you need two servers to do one thing. Paying extra for a dedicated
IP with SendGrid/Mailgun/Mailjet would solve the problem (by separating
the trac emails from everybody else's junk), but it's just stupid that
you'd have to pay extra for a second copy of what you already have at
GCE. It's also possible that Mailgun/Mailjet are better at preventing
abuse than SendGrid, and that there would be no problem with them, but
as "bulk email for hire" services I'm skeptical.

[William Stein]

On Sun, Apr 26, 2020 at 2:01 AM Harald Schilly <har...@schil.ly> wrote:

On Sun, Apr 26, 2020 at 7:27 AM Dima Pasechnik <dim...@gmail.com> wrote:

Also, note that GCE has an option of sending emails via G-suite,
perhaps if someone already has G-suite it might come out better.

G-Suite has an email relay server, where the limit is 10k per day, more or less. It's not hard to set it up.
https://support.google.com/a/answer/2956491?hl=en

That sounds like what should definitely be done.  The dedicated SendGrid option is pretty expensive (well over $100/month).

We have a G-suite subscription for "sagemath.com" so it seems like we could use this for trac emails...

 

If you have 2FA enabled, you need to create an "app specific password" just for sending emails, and use that as the password.
https://support.google.com/accounts/answer/185833

-- harald

 

--

---
You received this message because you are subscribed to the Google Groups "sagemath-admins" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sagemath-admi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sagemath-admins/CAGG4CB5caT1O_J9uFwtqRP5rOwqbxrzXJROPC2sGzSaiAPC23g%40mail.gmail.com.

--

William (http://wstein.org)

[Reimundo Heluani]

I was wandering if anything happened to this. I have been receiving some trac e-mails and missing some others silently, some during the same day so it does not seem to me that trac is choosing these IPs sequentially.

Here's a typical entry for a blocked e-mail (blocked by spamcop):

Jul 20 14:06:04 whiskey postfix/smtpd[7254]: NOQUEUE: reject: RCPT from o1.3nn.shared.sendgrid.net[167.89.100.129]: 554 5.7.1 Service unavailable; Client host [167.89.100.129] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?167.89.100.129; from=<bounces+3351942-2958-heluani=potu...@mail.sagemath.org> to=<hel...@potuz.net> proto=ESMTP helo=<o1.3nn.shared.sendgrid.net>

Perhaps someone here has already written a postfix rule to allow traffic based on some metadata from the trac servers (say something like from=bounces+.*@mail.sagemath,org)? 

[Michael Orlitzky]

On 2020-07-29 08:02, 'Reimundo Heluani' via sage-devel wrote:
>
> I was wandering if anything happened to this. I have been receiving some
> trac e-mails and missing some others silently, some during the same day
> so it does not seem to me that trac is choosing these IPs sequentially.

No, and since I posted this, there have been a number of threads on
mailop, nanog, sdlu, and spamassassin-users (all mail admin lists)
criticizing how bad SendGrid has become.

As a result of those threads, two people at SendGrid stepped up to help
fix the problem. That gave me a bit of temporary hope, but one of them
immediately disappeared. The other guy made an honest effort, but I
think he realized that one person alone can't handle tens of thousands
of spam complaints when the perpetrators are allowed to sign up again
the next day. While I don't want to put words in his mouth (he did try),
he more or less gave up in frustration, and we're back to square one.
Nobody there is working on the problem.

[Dima Pasechnik]

I am willing to try to switch to using sending via a Google G-Suite relay. (Or using Harald's offer of such a setup).

Note that I am writing this on my phone, during the 2nd day of a 3day bicycle trip, so don't expect this done today.

I also have no clue about how trac emails work. Ideally I would rather let Erik handle this.

Dima

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c75ec790-60eb-f628-1cc3-4c51feb908b3%40orlitzky.com.

[Dima Pasechnik]

I've switched trac to send notifications via a host called
sagemath.openbsd.amsterdam
(after wasting a lot of time trying to use Google's G-suite, it's hell )

Let me know whether this works for you - I see it working for me
(sending trac notifications to my gmail address - and getting these).

Dima

[Reimundo Heluani]

On Aug 04, Dima Pasechnik wrote:
>I've switched trac to send notifications via a host called
>sagemath.openbsd.amsterdam
>(after wasting a lot of time trying to use Google's G-suite, it's hell )
>
>Let me know whether this works for you - I see it working for me
>(sending trac notifications to my gmail address - and getting these).

At what time did you switch? This is my log for today:

Aug 04 08:55:18 whiskey postfix/smtpd[26517]: NOQUEUE: reject: RCPT from o1.3nn.shared.sendgrid.net[167.89.100.129]: 554 5.7.1 Service unavailable; Client host [167.89.100.129] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?167.89.100.129; from=<bounces+3351942-2958-heluani=potu...@mail.sagemath.org> to=<hel...@potuz.net> proto=ESMTP helo=<o1.3nn.shared.sendgrid.net>

That's three hours ago here, it's the last ticket I should've gotten an update

R.

>To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq2n8cr6YBOXfTTF3gSaYpSTg%3DvSVhtmVSBc0Uoj_CTj-w%40mail.gmail.com.

[Dima Pasechnik]

On Tue, Aug 4, 2020 at 3:30 PM 'Reimundo Heluani' via sage-devel
<sage-...@googlegroups.com> wrote:
>
> On Aug 04, Dima Pasechnik wrote:
> >I've switched trac to send notifications via a host called
> >sagemath.openbsd.amsterdam
> >(after wasting a lot of time trying to use Google's G-suite, it's hell )
> >
> >Let me know whether this works for you - I see it working for me
> >(sending trac notifications to my gmail address - and getting these).
>
> At what time did you switch? This is my log for today:
>
> Aug 04 08:55:18 whiskey postfix/smtpd[26517]: NOQUEUE: reject: RCPT from o1.3nn.shared.sendgrid.net[167.89.100.129]: 554 5.7.1 Service unavailable; Client host [167.89.100.129] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?167.89.100.129; from=<bounces+3351942-2958-heluani=potu...@mail.sagemath.org> to=<hel...@potuz.net> proto=ESMTP helo=<o1.3nn.shared.sendgrid.net>
>
> That's three hours ago here, it's the last ticket I should've gotten an update

The switch went live 20 min ago or so
Please try doing some trac comments, in case.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/20200804142954.GB20773%40vertex.

[Reimundo Heluani]

On Aug 04, Dima Pasechnik wrote:
>On Tue, Aug 4, 2020 at 3:30 PM 'Reimundo Heluani' via sage-devel
><sage-...@googlegroups.com> wrote:
>>
>> On Aug 04, Dima Pasechnik wrote:
>> >I've switched trac to send notifications via a host called
>> >sagemath.openbsd.amsterdam
>> >(after wasting a lot of time trying to use Google's G-suite, it's hell )
>> >
>> >Let me know whether this works for you - I see it working for me
>> >(sending trac notifications to my gmail address - and getting these).
>>
>> At what time did you switch? This is my log for today:
>>
>> Aug 04 08:55:18 whiskey postfix/smtpd[26517]: NOQUEUE: reject: RCPT from o1.3nn.shared.sendgrid.net[167.89.100.129]: 554 5.7.1 Service unavailable; Client host [167.89.100.129] blocked using bl.spamcop.net; Blocked - see https://www.spamcop.net/bl.shtml?167.89.100.129; from=<bounces+3351942-2958-heluani=potu...@mail.sagemath.org> to=<hel...@potuz.net> proto=ESMTP helo=<o1.3nn.shared.sendgrid.net>
>>
>> That's three hours ago here, it's the last ticket I should've gotten an update
>
>The switch went live 20 min ago or so
>Please try doing some trac comments, in case.

Right, it's working fine here, thanks.

R.

>To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq0%3DhiTa%2BGC3cKXxDXyhUJ_doNeCSiRYsTgUcKVVqP5cuA%40mail.gmail.com.

[Samuel Lelièvre]

2020-08-04 14:21 UTC, Dima Pasechnik:

>
> I've switched trac to send notifications via a host called
> sagemath.openbsd.amsterdam
> (after wasting a lot of time trying to use Google's G-suite, it's hell )
>
> Let me know whether this works for you - I see it working for me
> (sending trac notifications to my gmail address - and getting these).

Thanks Dima, this appears to be working for me.
Just got an email due to activity on a Trac ticket I'm cc-ed on.

[Michael Orlitzky]

On 2020-08-04 10:21, Dima Pasechnik wrote:
> I've switched trac to send notifications via a host called
> sagemath.openbsd.amsterdam
> (after wasting a lot of time trying to use Google's G-suite, it's hell )
>
> Let me know whether this works for you - I see it working for me
> (sending trac notifications to my gmail address - and getting these).
>

This has been working wonderfully, thanks for taking the time to do it.