FYI for the PpyMilter group: Problems with DKIM verify

[Stephen Nightingale]

I have been building an email authenticator using the existing Python packages, which are:
- ppymilter.py to do header and body processing on sender and receiver messages.
- spf.py to authenticate From addresses in received messages, using the DNS txt records.
- dkim.py (the new dkimpy fork) to verify message integrity.
- Still to come: DMARC implementation to control SPF and DKIM deployment, and provide feedback to senders.

I did this by extending ppymilterbase.py directly, rather than implementing a child class, as I had some problems with the latter.
Also I'm using the AsyncPpyMilterServer. SPF and DKIM processing are both done in the 'OnEndBody' function.

There is a problem with the use of dkim.py. Whether it is directly a dkim problem, or whether it is a Sendmail problem, is one issue in question. PpyMilter would not seem to be at fault as it cleanly sends and receives data to Sendmail across the milter interface.  PpyMilter is doing the job: kudos to Eric.

The issue is described in the message I sent below to the dkimpy group on Launchpad. The short summary is: that Sendmail is corrupting the folded DKIM signature by replacing "\r\n" sequences with "\r<space>" sequences. DKIM verify fails as it unpacks the DKIM-Signature incorrectly.  I have a workaround that patches back the "\n"s in the signature before running through DKIM verify, though I suspect this is fragile.  My message to the dkimpy group is given below.

While this is not a problem with PpyMilter, I thought it would be of interest to users here, since SPF and DKIM processing are likely to be among the principal reasons for using it.

Stephen N.



New question #201799 on dkimpy:
https://answers.launchpad.net/dkimpy/+question/201799


This is the message emitted by the Sender end (bracketed by <BEGIN> <END>):

<BEGIN>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; 
d=good.emailtest.had-pilot.biz;
  i=@good.emailtest.had-pilot.biz; q=dns/txt; s=mailkey; t=1340661225;
  h=Date : From : To : Cc : Subject : Message-ID : MIME-Version :
  Content-Type : Content-Disposition : User-Agent;
  bh=2cilj9JgD3aYI49HQfXVKgyfMmRb1qUViiejVxoJG2w=;
  b=RmjJNiUcdVSsppai2DBpof3/L6GpqEpuPDVxJL4jyCgQKtclWYIs+5p/ZaEHDOfgW+rg9Q
  R1CRsp6AU5gQoHP9ICViZ+TUUHsErG6F1Hi8EJzHnfxrkDm41FNtI1KuU7Q9VWhbTkVKTrgX
  zMhFjrVRjKS7EFE+B3YFbS9LpwFfA=
Received: (from night@localhost)
     by had1.antd.nist.gov (8.14.4/8.14.4/Submit) id q5PLriME003627;
     Mon, 25 Jun 2012 17:53:44 -0400
Date: Mon, 25 Jun 2012 17:53:44 -0400
From: Fushimi Inari <ni...@good.emailtest.had-pilot.biz>
To: ni...@pythentic.emailtest.had-pilot.biz
Cc: ni...@nist.gov
Subject: Short01 test
Message-ID: <2012062521...@short.text>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.20 (2009-12-10)

all.
<END>

- I am using Sendmail 8.14.4 at the Sender (good.emailtest) and receiver 
(pythentic) ends, and dkimpy 0.5.2 in the milters.
- There is a Sender ppymilter, that calculates the signature and 
introduces the DKIM-Signature header at index 0,
and a Receiver ppymilter that does SPF and then verifies the DKIM signature.
- The signature is folded at the Sender end using dkim.fold(). This 
splits the signature with "\r\n<space>" sequences.
- The message as delivered through Sendmail has newlines stripped. 
Specifically, the "\r\n\" sequences in the DKIM-Signature are replaced 
by "\r<space> ".
- This signature fails dkim.verify.
- When I explicitly replace the "\r<space>" sequences in the signature 
with "\r\n" sequences, before giving the message to verify, the 
signature verifies.

So I suspect there is a problem with the way that the 
rfc822_parse(message) function 'unfolds' the signature.
Application of the 'lines = re.split(b"\r?\n", message)' statement would 
seem to leave a set of spurious spaces in the signature. The lines 
[i][0] in (<tab>, <space> ...) statement doesn't seem to patch it.  
dkim.util.parse_tag_value already assumes the signature is correctly 
unfolded when it isn't, and the computation fails.

I suspect the problem can be solved by fixing the way that rfc822_parse 
unfolds the signature. Or by fixing the newline stripping behavior of 
Sendmail. Or there's something even more arcane going on that I don't 
yet know.

Regards,

Stephen.

[Eric DeFriez]

FWIW, you should be able to turn up the sendmail logging to like 20
and i believe do the same in ppymilter (if not easy to patch it to
print each character as it's read off the wire) and confirm it's being
mangled by sendmail before the milter sees it.

if so, very odd/interesting development. make sure it's not another
milter you're running or some weird sendmail rulesets/spam checker,
etc.

FWIW, there are different levels of severity for DKIM
signature/verification iirc and some of them are OK with
whitespace-only changes. so you might want/have to use that to make
it all work more robustly.

eric

> --
> You received this message because you are subscribed to the Google Groups
> "ppymilter" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/ppymilter/-/2CcHgGRaW9wJ.
> To post to this group, send email to ppym...@googlegroups.com.
> To unsubscribe from this group, send email to
> ppymilter+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/ppymilter?hl=en.

[Stephen Nightingale]

Plenty of activity on Launchpad for dkimpy regarding this issue:  https://answers.launchpad.net/dkimpy/+question/202157

The problem is to work around a 'mis-design' in the libmilter specification. The solutions the dkimpy group are coming up with, so far, are mods to pymilter, that uses the libmilter library. Since ppymilter is an independent implementation of the interface, a different solution will have to be devised.

Since the folded whitespace, per RFC6376, is not part of the header, It cannot be expected that the WS introduced at the Sending milter is preserved at the Receiving milter after transmission through a bunch of MTAs. Therefore the functions that unpack the message have to be able to accommodate different forms of whitespace corruption. That is, either fix it in the milter, or fix it in the dkim implementation. Since dkimpy is used by other miltering systems, I am modifying 'OnHeader' in ppymilter to handle the dropped newlines.