Bug#1043539: project: Forwarding of @debian.org mails to gmail broken

[Helge Kreutzmann]

Package: project
Severity: important

If I try to mail e.g. Marcos Fouces <mar...@debian.org>, this no
longer works. I get the following error message:

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

marcos...@gmail.com
host gmail-smtp-in.l.google.com [173.194.79.26]
SMTP error from remote mail server after pipelined end of data:
550-5.7.26 This mail is unauthenticated, which poses a security risk to the
550-5.7.26 sender and Gmail users, and has been blocked. The sender must
550-5.7.26 authenticate with at least one of SPF or DKIM. For this message,
550-5.7.26 DKIM checks did not pass and SPF check for [helgefjell.de] did not
550-5.7.26 pass with ip: [82.195.75.114]. The sender should visit
550-5.7.26 https://support.google.com/mail/answer/81126#authentication for
550 5.7.26 instructions on setting up authentication. v26-20020aa7d65a000000b005231f55294dsi4996663edr.385 - gsmtp

The IP 82.195.75.114 resolves to
114.75.195.82.in-addr.arpa is an alias for 114.64-26.75.195.82.in-addr.arpa.
114.64-26.75.195.82.in-addr.arpa domain name pointer mailly.debian.org.

And of course, SPF/DKIM checks for my domain (helgefjell.de) fail for
this IP, which is @debian.org.

I don't know how it worked so far, and the error could be on my side, as I
recently switched my e-mail setup; however, I don't see anything I can
do to make DKIM/SPF point to @debian.org instead of @helgefjell.de,
when transferring e-mail to gmail.

Greetings

Helge

--
Dr. Helge Kreutzmann deb...@helgefjell.de
Dipl.-Phys. http://www.helgefjell.de/debian.php
64bit GNU powered gpg signed mail preferred
Help keep free software "libre": http://www.ffii.de/

[Adam D. Barratt]

Hi,

On Sat, 2023-08-12 at 15:54 +0000, Helge Kreutzmann wrote:
> If I try to mail e.g. Marcos Fouces <mar...@debian.org>, this no
> longer works. I get the following error message:
>

Contacting DSA is generally a better way to ask about infrastructure
things than filing bugs on high-level pseudo-packages.

The DKIM signature warning has nothing to do with the forwarding, or
the involvement of debian.org at all. The reason that check fails is
that your mail has no DKIM signature, so obviously can't have a valid
one. Signing your mail would probably make gmail a lot happier with it
in general. (As a side note, the BTS breaks many common DKIM signature
strategies, but that's a different issue.)

The general issue is being worked on, as time and resources allow.

Regards,

Adam
(part of, but not on behalf of, DSA)

[Adam D. Barratt]

On Sat, 2023-08-12 at 17:08 +0000, Helge Kreutzmann wrote:
> Hello Adam,

> Am Sat, Aug 12, 2023 at 05:35:52PM +0100 schrieb Adam D. Barratt:
> > On Sat, 2023-08-12 at 15:54 +0000, Helge Kreutzmann wrote:

[...]

> > > 550-5.7.26 This mail is unauthenticated, which poses a
> > > security
> > > risk to the
> > > 550-5.7.26 sender and Gmail users, and has been blocked. The
> > > sender must
> > > 550-5.7.26 authenticate with at least one of SPF or DKIM. For
> > > this message,
> > > 550-5.7.26 DKIM checks did not pass and SPF check for
> > > [helgefjell.de] did not
> > >

[...]

> > > 550-5.7.26
> > > https://support.google.com/mail/answer/81126#authentication for
> > > 550 5.7.26 instructions on setting up authentication. v26-
> > > 20020aa7d65a000000b005231f55294dsi4996663edr.385 - gsmtp
> > >
> > > The IP 82.195.75.114 resolves to
> > > 114.75.195.82.in-addr.arpa is an alias for 114.64-
> > > 26.75.195.82.in-
> > > addr.arpa.
> > > 114.64-26.75.195.82.in-addr.arpa domain name pointer
> > > mailly.debian.org.
> > >
> > > And of course, SPF/DKIM checks for my domain (helgefjell.de) fail
> > > for this IP, which is @debian.org.
> > >
> >

> > The DKIM signature warning has nothing to do with the forwarding,
> > or the involvement of debian.org at all. The reason that check
> > fails is that your mail has no DKIM signature, so obviously can't
> > have a valid one. Signing your mail would probably make gmail a lot
> > happier with it in general. (As a side note, the BTS breaks many
> > common DKIM signature strategies, but that's a different issue.)
>

> Sigh.
>
> Directly gmail accepts it.
>

I'm not sure why the sigh, but in any case your direct mail presumably
succeeds because it passes the SPF check. I was simply clarifying that
the DKIM check would fail in both cases.

Regards,

Adam

[Helge Kreutzmann]

Hello Adam,
Am Sat, Aug 12, 2023 at 05:35:52PM +0100 schrieb Adam D. Barratt:

> On Sat, 2023-08-12 at 15:54 +0000, Helge Kreutzmann wrote:
> > If I try to mail e.g. Marcos Fouces <mar...@debian.org>, this no
> > longer works. I get the following error message:
> >
>
> Contacting DSA is generally a better way to ask about infrastructure
> things than filing bugs on high-level pseudo-packages.

Thanks, then I know this for the future.

Sigh.

Directly gmail accepts it.

> The general issue is being worked on, as time and resources allow.

Thanks a lot!

[Russ Allbery]

Helge Kreutzmann <deb...@helgefjell.de> writes:

> I don't know how it worked so far, and the error could be on my side, as
> I recently switched my e-mail setup; however, I don't see anything I can
> do to make DKIM/SPF point to @debian.org instead of @helgefjell.de, when
> transferring e-mail to gmail.

The mail to which I'm resonding also comes from your @helgefjell.de
domain, so I'm suspecting some DKIM/SPF issues there if you're using that
same address in your original mail message. But just in case you were
trying to send from your @debian.org address, one option is to send all of
your outgoing mail that is from your debian.org address through the
debian.org mail servers. See:

https://dsa.debian.org/user/mail-submit/

I don't think this is the direct answer to your original question, but I
suspect it would work around the problem.

--
Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>

[Helge Kreutzmann]

Hello Adam,

Well, I did have trouble sending directly to gmail accounts, which now
seems to work. Now the next e-mail problem arises, which I need to see
how much I can configure it to work. That's the sigh.

It's just that I never had this problem with mails to people with
@debian.org addresses, so either my new configuration or some other
change, I don't know.

I hope this explains it a little.

[Helge Kreutzmann]

Hello Russ,

Am Sat, Aug 12, 2023 at 11:31:35AM -0700 schrieb Russ Allbery:
> Helge Kreutzmann <deb...@helgefjell.de> writes:
>
> > I don't know how it worked so far, and the error could be on my side, as
> > I recently switched my e-mail setup; however, I don't see anything I can
> > do to make DKIM/SPF point to @debian.org instead of @helgefjell.de, when
> > transferring e-mail to gmail.
>
> The mail to which I'm resonding also comes from your @helgefjell.de
> domain, so I'm suspecting some DKIM/SPF issues there if you're using that
> same address in your original mail message. But just in case you were

Yes, this is my primary e-mail address

> trying to send from your @debian.org address, one option is to send all of
> your outgoing mail that is from your debian.org address through the
> debian.org mail servers. See:
>
> https://dsa.debian.org/user/mail-submit/
>
> I don't think this is the direct answer to your original question, but I
> suspect it would work around the problem.

Thanks for taking care, but I don't have an @debian.org address.

[Russ Allbery]

Helge Kreutzmann <deb...@helgefjell.de> writes:

> It's just that I never had this problem with mails to people with
> @debian.org addresses, so either my new configuration or some other
> change, I don't know.

The problem I suspect is with email forwarding, and specifically email
forwarding to Gmail, which has recently ramped up the amount of
verification it does on messages. Because of email forwarding, Gmail sees
a message purportedly from helgefjell.de but actually delivered by
debian.org mail servers, and has now decided to be suspicious of that.

If that's correct, you'll only have this problem with Debian developers
who forward their @debian.org addresses to Gmail. Gmail handles some
large percentage of all email on the Internet, so this probably isn't
rare, but Debian developers are less likely to use it than random Internet
users for obvious reasons, so it doesn't surprise me you've not run into
the problem before. (In other words, I doubt this is a problem with your
local configuration.)

[Mattia Rizzolo]

On Sat, Aug 12, 2023 at 01:41:46PM -0700, Russ Allbery wrote:
> The problem I suspect is with email forwarding, and specifically email
> forwarding to Gmail, which has recently ramped up the amount of
> verification it does on messages. Because of email forwarding, Gmail sees
> a message purportedly from helgefjell.de but actually delivered by
> debian.org mail servers, and has now decided to be suspicious of that.

This is the exact use case that SRS was developer for, however gmail's
documentation does not recommend that (but the situation, as you noted,
worsened, so I tried it in some other similar setups and everything is
great, so...).
My understanding is that several DSA members were opposed to using SRS
for @debian.org forwarding, but maybe it's now time?

Alternatively, I wonder if ARC nowadays is respected enough (and if
Google cares about it)... I personally don't have any system with ARC
under my care.

--
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`.
More about me: https://mapreri.org : :' :
Launchpad user: https://launchpad.net/~mapreri `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia `-

[Adam D. Barratt]

On Sat, 2023-08-12 at 23:13 +0200, Mattia Rizzolo wrote:
> On Sat, Aug 12, 2023 at 01:41:46PM -0700, Russ Allbery wrote:
> > The problem I suspect is with email forwarding, and specifically
> > email
> > forwarding to Gmail, which has recently ramped up the amount of
> > verification it does on messages. Because of email forwarding,
> > Gmail sees
> > a message purportedly from helgefjell.de but actually delivered by
> > debian.org mail servers, and has now decided to be suspicious of
> > that.
>
> This is the exact use case that SRS was developer for, however
> gmail's documentation does not recommend that (but the situation, as
> you noted, worsened, so I tried it in some other similar setups and
> everything is great, so...).

They sort of recommend it now. But also not. It's complicated. [tm]

> My understanding is that several DSA members were opposed to using
> SRS for @debian.org forwarding, but maybe it's now time?
>

That's essentially what's being worked on. But life, and free time, and
other priorities, keep getting in the way.

Regards,

Adam

[Cord Beermann]

Hallo! Du (Russ Allbery) hast geschrieben:

>The problem I suspect is with email forwarding, and specifically email
>forwarding to Gmail, which has recently ramped up the amount of
>verification it does on messages. Because of email forwarding, Gmail sees
>a message purportedly from helgefjell.de but actually delivered by
>debian.org mail servers, and has now decided to be suspicious of that.

>If that's correct, you'll only have this problem with Debian developers
>who forward their @debian.org addresses to Gmail. Gmail handles some
>large percentage of all email on the Internet, so this probably isn't
>rare, but Debian developers are less likely to use it than random Internet
>users for obvious reasons, so it doesn't surprise me you've not run into
>the problem before. (In other words, I doubt this is a problem with your
>local configuration.)

As listmaster i can confirm that it is a big problem to deliver Mails to
gmail/outlook/yahoo. Yahoo Subscribers are mostly gone by now because they
bounced a lot, for gmail it is so much that we just ignore bounces because of
those rules.

If you decide to handle your mails to be curated by someone else you have to
live with an incomplete mailbox.

| helgefjell.de descriptive text "v=spf1 ip4:142.132.201.35 mx ~all"

so you flagged your mail has to come from that IP (or the MX) and from other
sources it should be considered suspicious.

Thats the result.

SRS/ARC and so on are just dirty patches that try to fix things that were
broken before, but they will break even more things like Mail signing.

As long as we have this Oligopol that doesn't care about what they send out
(i.e. Spamfloods through Outlook) things will only get worse.

Cord

[Russ Allbery]

Cord Beermann <co...@debian.org> writes:

> As listmaster i can confirm that it is a big problem to deliver Mails to
> gmail/outlook/yahoo. Yahoo Subscribers are mostly gone by now because
> they bounced a lot, for gmail it is so much that we just ignore bounces
> because of those rules.

Yes, I gave up for the mailing lists I run and just rewrite the From
address to be the address of the list and move the actual sender to
Reply-To, and I see other technical mailing lists like the glibc lists
have started doing this as well (using the built-in Mailman feature, which
can optionally do this only if the sender domain has SPF/DMARC records).

[Stephen Frost]

Greetings,

* Cord Beermann (co...@debian.org) wrote:
> As listmaster i can confirm that it is a big problem to deliver Mails to
> gmail/outlook/yahoo. Yahoo Subscribers are mostly gone by now because they
> bounced a lot, for gmail it is so much that we just ignore bounces because of
> those rules.

As a maintainer or some pretty big lists ... we don't have *that* much
trouble delivering to gmail, or others for that matter.

> | helgefjell.de descriptive text "v=spf1 ip4:142.132.201.35 mx ~all"
>
> so you flagged your mail has to come from that IP (or the MX) and from other
> sources it should be considered suspicious.

... but if it's DKIM signed, then it'll generally get delivered
properly.

> SRS/ARC and so on are just dirty patches that try to fix things that were
> broken before, but they will break even more things like Mail signing.

ARC doesn't break DKIM signatures (unless someone's got a very broken
DKIM setup which over-signs ARC headers ... but if so, then that's on
them).

Thanks,

Stephen

[Stephen Frost]

Greetings,

* Mattia Rizzolo (mat...@debian.org) wrote:
> Alternatively, I wonder if ARC nowadays is respected enough (and if
> Google cares about it)... I personally don't have any system with ARC
> under my care.

Sadly, no, they don't seem to care one bit about ARC, except possibly if
it's their own ARC sigs.

If someone has some idea how to get them to care about ARC, I'd love to
hear about it, as I have folks on the one hand who view DKIM/DMARC as
too painful to set up but then they end up with bounces from gmail due
to my forwarding of messages through my server (which are being
ARC-signed by it and pass on that the SPF check was successful when they
arrived to my server)...

I'd encourage everyone running their own email servers to please get
DKIM/DMARC/ARC/SPF set up. Yeah, it's annoying, but it's not actually
all *that* bad to do.

Thanks,

Stephen

[Stephen Frost]

Greetings,

The answer that we (PostgreSQL folks, at least) went with was to stop
breaking DKIM because that's just a bad approach to take these days with
mailing lists. If you're curious about what PostgreSQL and now SPI are
using for our lists, it's called pgLister and is here:

https://gitlab.com/pglister/pglister

Others have hacked up mailman to make it stop breaking DKIM too (though
it's pretty grotty how they did it, I'll admit).

Yes, yes, I know that means a bunch of mailman features aren't
available. We've managed to survive even without them.

Thanks,

Stephen

[Marco d'Itri]

On Aug 14, Stephen Frost <sfr...@snowman.net> wrote:

>If someone has some idea how to get them to care about ARC, I'd love to
>hear about it, as I have folks on the one hand who view DKIM/DMARC as
>too painful to set up but then they end up with bounces from gmail due
>to my forwarding of messages through my server (which are being
>ARC-signed by it and pass on that the SPF check was successful when they
>arrived to my server)...

I do not know of any situation in which DMARC adoption would improve
deliverability, and most people that configure it are just engaging in
cargo cult sysadmining.
DMARC with p=reject is useful when the sender domain is a phishing
victim, e.g. a financial organization, but most users do not need it.

In other words: if these people want to support use cases like
forwarding and participating to mailing lists then they should adopt
DKIM and ignore DMARC.

--
ciao,
Marco

[Adam D. Barratt]

An initial version, rewriting mails to Google-hosted domains from
"external" e-mail addresses (those for which debian.org's mail relays
don't consider themselves authoritative, so mostly not *.debian.org and
*.debconf.org) is now live.

Please let DSA know if you encounter any issues.

Regards,

Adam

[Helge Kreutzmann]

Hello Adam,

Am Thu, Aug 17, 2023 at 06:52:11PM +0100 schrieb Adam D. Barratt:
> An initial version, rewriting mails to Google-hosted domains from
> "external" e-mail addresses (those for which debian.org's mail relays
> don't consider themselves authoritative, so mostly not *.debian.org and
> *.debconf.org) is now live.
>
> Please let DSA know if you encounter any issues.

Thanks a lot for the speedy fixing.

I'll report any issues (if any).

[Byung-Hee HWANG]

Hellow all,

Sorry for late feedback. By chance, i discovered this story
#1043539. Also i'm using Gmail via forwarding (Postfix+OpenDKIM). I love
Postfix and DKIM stuff.

Though i am not debian member, i would like to share this experience
with you. If your dkim signature is OK, then Gmail does accept all
mails. So never use SRS. DKIM is enough.

<quote: postfix log with debian-bugs-dist mailing>
Nov 25 09:51:14 yw-1204 postfix/smtpd[94851]: connect from yw-0919.doraji.xyz[2600:1900:4000:af49:0:3::]
Nov 25 09:51:14 yw-1204 postfix/smtpd[94851]: Trusted TLS connection established from yw-0919.doraji.xyz[2600:1900:4000:af49:0:3::]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256
Nov 25 09:51:14 yw-1204 postfix/smtpd[94851]: D6A8A6D3: client=yw-0919.doraji.xyz[2600:1900:4000:af49:0:3::]
Nov 25 09:51:14 yw-1204 postfix/cleanup[94856]: D6A8A6D3: resent-message-id=<handler.1056557.B1056...@bugs.debian.org>
Nov 25 09:51:15 yw-1204 postfix/cleanup[94856]: D6A8A6D3: message-id=<e98510e4-738c-1ea8...@debian.org>
Nov 25 09:51:15 yw-1204 opendkim[631]: RFC2822-From: Andreas Beckmann <an...@debian.org>
Nov 25 09:51:15 yw-1204 opendkim[631]: RFC2821-From: bounce-debian-bugs-dist=soyeomul=doraj...@lists.debian.org
Nov 25 09:51:15 yw-1204 opendkim[631]: RFC2821-To: soyeom...@gmail.com
Nov 25 09:51:15 yw-1204 opendkim[631]: D6A8A6D3: DKIM-Signature field added (s=yw-1204-doraji-xyz, d=doraji.xyz)
Nov 25 09:51:15 yw-1204 opendkim[631]: D6A8A6D3: DKIM-Signature field added (s=YW, d=doraji.xyz)
Nov 25 09:51:15 yw-1204 postfix/qmgr[91844]: D6A8A6D3: from=<bounce-debian-bugs-dist=soyeomul=doraj...@lists.debian.org>, size=7421, nrcpt=1 (queue active)
Nov 25 09:51:15 yw-1204 postfix/smtpd[94851]: disconnect from yw-0919.doraji.xyz[2600:1900:4000:af49:0:3::] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Nov 25 09:51:17 yw-1204 postfix/smtp[94857]: Verified TLS connection established to gmail-smtp-in.l.google.com[2607:f8b0:4023:1006::1b]:25: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256
Nov 25 09:51:18 yw-1204 postfix/smtp[94857]: D6A8A6D3: to=<soyeom...@gmail.com>, relay=gmail-smtp-in.l.google.com[2607:f8b0:4023:1006::1b]:25, delay=3.4, delays=0.25/0.02/2.2/0.98, dsn=2.0.0, status=sent (250 2.0.0 OK 1700905878 ep1-20020a056808444100b003b85c5d06d8si386914oib.242 - gsmtp)
Nov 25 09:51:18 yw-1204 postfix/qmgr[91844]: D6A8A6D3: removed
</quote>

Sincerely, Byung-Hee (Debian user in South Korea)