Re: Using Amazon simple email service with Gerrit

[Lloyd Chang]

Hi Patrick,

Is your e-mail address in Gerrit case-insensitive? For example, are you using ${sAMAccountName.toLowerCase}@example.com against an Active Directory's LDAP interface, but using Amazon SES to send e-mail?

How is your ldap.accountEmailAddress set? What is the equivalent in your LDAP directory, e.g. the mail attribute?

http://gerrit-documentation.googlecode.com/svn/Documentation/2.5.1/config-gerrit.html#ldap.accountEmailAddress

Perhaps your Amazon SES differs by using Mixed...@example.com?

According to https://forums.aws.amazon.com/message.jspa?messageID=334504

SES seems to expect e-mail addresses to be case sensitive.

Another person encountered the same error message when registering a different e-mail address

http://stackoverflow.com/questions/12560447/gerrit-sendemail-server-rejected-body

The Gerrit code snippet you're interested in is in

http://code.google.com/p/gerrit/source/browse/gerrit-server/src/main/java/com/google/gerrit/server/mail/SmtpEmailSender.java

I hope above potential scenarios helps you debug and resolve the issue,

Lloyd

On Fri, Jan 11, 2013 at 3:10 PM, <pat...@judicata.com> wrote:

Hey everyone -- 

I am setting up Gerrit on an EC2 instance and, naturally, we want to use SES to send email.  I've gotten SES set up, but I just can't seem to get it to accept Gerrit's email.  When I add someone as a reviewer, the message I get is: "Server email-smtp.us-east-1.amazonaws.com rejected body"

Here is the [sendemail] portion from my gerrit.conf:

[sendemail]

    from = USER

    smtpServer = email-smtp.us-east-1.amazonaws.com

    smtpServerPort = 25

    smtpEncryption = tls

    smtpUser = (censored)

The smtpPass is also configured correctly in secure.conf.  I've tested these with a Python script running from the same machine and it works.  All relevant email addresses are verified in SES (both the from and to, even though the to address need not be verified).

I searched and searched on Google but could not find anyone who seems to have had the same problem.  I checked the Gerrit logs and could find no relevant information.  

Thanks,

Patrick

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Magnus Bäck]

On Friday, January 11, 2013 at 18:10 EST,

pat...@judicata.com wrote:

> I am setting up Gerrit on an EC2 instance and, naturally, we want to
> use SES to send email. I've gotten SES set up, but I just can't seem
> to get it to accept Gerrit's email. When I add someone as a reviewer,
> the message I get is: "Server email-smtp.us-east-1.amazonaws.com
> rejected body"

This error message from Gerrit is actually a bit misleading -- the
server isn't complaining about the message body itself but rather the
DATA command that the client issues to initiate the transmission of the
body. The only reason for this I can think of is if no recipients were
accepted in the preceding sequence of RCPT FROM commands. Gerrit does
record rejected recipients (and the server's reason for the rejection),
but they're not logged unless the message is actually accepted by the
server.

> Here is the [sendemail] portion from my gerrit.conf:
>
> [sendemail]
> from = USER
> smtpServer = email-smtp.us-east-1.amazonaws.com
> smtpServerPort = 25
> smtpEncryption = tls
> smtpUser = (censored)

Perhaps Amazon doesn't like 'from = USER'? Note the documented caveats
with that setting. Does the domain you're using have any DKIM or SPF
restrictions? I suppose you can't wiretap the SMTP session to see what
happens, but perhaps you can rebuild Gerrit to make sure recipient
errors always result in exceptions? In SmtpEmailSender.java, add

throw new EmailException(rejected.toString());

after this line in SmtpEmailSender.send():

rejected.append("Server " + smtpHost + " rejected recipient "
+ addr + ": " + error);

(Some SMTP servers, like Postfix, delay the rejection of MAIL FROM
commands until the RCPT TO stage. This could explain why it's the
RCPT TO command that fails even though it's actually the sender address
the server dislikes. Assuming me hypothesis of zero accepted recipients
holds, obviously.)

I'll look into refactoring this code during the coming week to make sure
all errors are properly logged.

[...]

--
Magnus Bäck
ba...@google.com

[Patrick Krecker]

Hey guys -- Thanks for the pointers.  We're using OpenID and in my sandbox I have been very careful to ensure there is no confusion about case sensitivity.

I happened across the StackOverflow question when I was doing my due diligence and I don't think it applies.  I can't totally decipher his answer, but I think he was using an unverified sendemail.from user.  I've been careful to avoid this situation by ensuing both sending and receiving users in my sandbox are verified.

I've tried different sendemail.from settings, all to no avail.  Even hardcoding to a known verified email address didn't do the job.

It looks like we have both DKIM and SPF set up for our domain, but wouldn't this only affect delivery of the mail, not sending?  Also, two other points: a) the Python script I used to test was able to successfully send the mail and did not do any DKIM signing and b) amazonses.com is in our SPF record.

Thanks for pointing out the location in the code to debug.  Tomorrow I will mess with it and report my results.

Thanks!

Patrick

[Patrick Krecker]

OK, so after a little poking around I figured out the issue.  It's related to this bug report: http://code.google.com/p/gerrit/issues/detail?id=1444.  It looks like the Mailing-List header is to blame.  When I try sending an email including this header with Python's smtplib I get a 554 with the message "Transaction failed: Illegal header 'Mailing-List'".  I can't find any evidence online that this a valid header, or even a common one.  Commenting out the line 

    setVHeader("Mailing-List", "list $email.listId");

in ChangeEmail.java solves the issue.  I suppose I will be rolling my own version until this can be resolved.

It looks like intent of Mailing-List is to provide the exact email address instead of an identifier for the list, but to me it seems like it should just be removed.

In addition to changing the error messages in SmtpEmailSender.java to be a little more elaborate, it would be greatly helpful to include the reply string from the server in the error message when client.completePendingCommand() fails.  Since completePendingCommand() fails only when there was an error, it should be safe to include the most recent reply string in the error message (I think).

[Magnus Bäck]

On Monday, January 14, 2013 at 19:51 EST,

Patrick Krecker <pat...@judicata.com> wrote:

> OK, so after a little poking around I figured out the issue. It's
> related to this bug report:
> http://code.google.com/p/gerrit/issues/detail?id=1444.
> It looks like the Mailing-List header is to blame. When I try
> sending an email including this header with Python's smtplib I
> get a 554 with the message "Transaction failed: Illegal header
> 'Mailing-List'". I can't find any evidence online that this a
> valid header, or even a common one. Commenting out the line
>
> setVHeader("Mailing-List", "list $email.listId");
>
> in ChangeEmail.java solves the issue. I suppose I will be rolling my
> own version until this can be resolved.
>
> It looks like intent of Mailing-List is to provide the exact email
> address instead of an identifier for the list, but to me it seems
> like it should just be removed.

I couldn't find any reason for keeping this header, so I uploaded a
change suggesting its deletion. If there's a good reason for keeping
it it'll be trivial to make its inclusion configurable.

https://gerrit-review.googlesource.com/41370

> In addition to changing the error messages in SmtpEmailSender.java to
> be a little more elaborate, it would be greatly helpful to include the
> reply string from the server in the error message when
> client.completePendingCommand() fails. Since

> completePendingCommand()fails only when there was an error, it should

> be safe to include the most recent reply string in the error message
> (I think).

This and a few other enhancements to the SMTP client error messages
were made in https://gerrit-review.googlesource.com/41352 (merged).

--
Magnus Bäck
ba...@google.com

[Martin Fick]

Did you guys inspect the emails from this list? It has that
header in it. My email client has a reply to mailing list
feature which I think this triggers. It seems odd to me that
it would complain about any header unless it were malformed.
Perhaps you are missing content in the header?

-Martin

[Magnus Bäck]

On Thursday, January 17, 2013 at 15:57 EST,

According to the previously posted Amazon documentation, only a limited
set of headers are allowed so it's not a question of a malformed header.
I haven't heard of any MUAs using the Mailing-List header for anything,
and having it in the messages for Gerrit change notifications just seems
gratuitous; what useful action could you or your MDA/MUA take with

Mailing-List: list gerrit...@gerrit-review.googlesource.com

in each message? Apart from message filtering, but there's List-Id for
that.

--
Magnus Bäck
ba...@google.com

[Patrick Krecker]

Thanks a lot for pushing these patches.  I'm assuming these will be included in 2.6.  Will they be in 2.5.2?

Patrick

--
Magnus Bäck
ba...@google.com

[Edwin Kempin]

2013/1/23 Patrick Krecker <pat...@judicata.com>

Thanks a lot for pushing these patches.  I'm assuming these will be included in 2.6.  Will they be in 2.5.2?

Yes, they are included in 2.6. I've now cherry-picked them for 2.5.2 [1,2]. If nobody objects we can include them there too.

[1] https://gerrit-review.googlesource.com/41520
[2] https://gerrit-review.googlesource.com/41521
 

[Wei Dai]

Hi Patrick,

     I'm trying to set up Gerrit on Amazon EC2 as well, but can't access Gerrit's web interface from the Internet. I've set the "canonicalWebURL" to the Elastic IP, and my Apache server works (which I stopped so the Gerrit server could run).  Do you have any suggestions?

logs/error_log:

[2013-07-18 01:46:34,694] INFO  com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /home/daiwei89/gerrit_review/cache

[2013-07-18 01:46:37,034] WARN  com.google.gerrit.httpd.GitWebConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)

[2013-07-18 01:46:38,063] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 228m

[2013-07-18 01:46:38,083] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /home/daiwei89/gerrit_review/plugins

[2013-07-18 01:46:38,213] INFO  com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD on *:29418

[2013-07-18 01:46:38,215] INFO  org.eclipse.jetty.server.Server : jetty-8.1.7.v20120910

[2013-07-18 01:46:38,239] INFO  org.eclipse.jetty.server.handler.ContextHandler : started o.e.j.s.ServletContextHandler{/,file:/home/daiwei89/gerrit_review/tmp/gerrit_5102500950734049320_app/gerrit_war/}

[2013-07-18 01:46:39,145] INFO  org.eclipse.jetty.server.AbstractConnector : StartedSelectChann...@0.0.0.0:8080

[2013-07-18 01:46:39,148] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 2.6.1 ready

$ sockstat

USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE

daiwei89 java                 13095    tcp4   *:29418                   *:*                       LISTEN

daiwei89 java                 13095    tcp4   *:8080                    *:*                       LISTEN

Thank you very much!

Dai Wei

On Friday, January 11, 2013 6:10:50 PM UTC-5, pat...@judicata.com wrote:

Hey everyone -- 

I am setting up Gerrit on an EC2 instance and, naturally, we want to use SES to send email.  I've gotten SES set up, but I just can't seem to get it to accept Gerrit's email.  When I add someone as a reviewer, the message I get is: "Server email-smtp.us-east-1.amazonaws.com rejected body"

Here is the [sendemail] portion from my gerrit.conf:

[sendemail]

    from = USER

    smtpServer = email-smtp.us-east-1.amazonaws.com

    smtpServerPort = 25

    smtpEncryption = tls

    smtpUser = (censored)