Long delay sending emails

[Daniel Becroft]

Hi,

We are noticing there is an unusual delay in sending the emails. We are noticing anything from 5-30 minutes between the build completing, and the emails being sent. The build seems to hang (still taking up an agent). The last lines in the log file are:

Email was triggered for: Success

Email was triggered for: Fixed

Trigger Success was overridden by another trigger and will not send an email.

Sending email for trigger: Fixed

The longer delays appear to be related to having lots of committers/culprits in the commit list. We are using Active Directory for authentication (caching enabled), and SVN for version control. I'm not sure if it's an issue with our LDAP configuration (we are doing something a bit unusual with a Virtual IP hiding the actual AD servers).

Are there any logs that I can look at specifically to determine whether there is a delay in (a) getting the list of committers; or (b) resolving the email addresses via AD, or if the delay is somewhere else in the line? I'm getting the Network guys to investigate the AD side, but want to check the Jenkins side as well.

Regards,

Daniel.

[Daniel Beck]

> On 2. Jan 2018, at 01:26, Daniel Becroft <djcbe...@gmail.com> wrote:
>
> Are there any logs that I can look at specifically

I think email-ext has a debug log option in the global preferences, but since it's a timing issue, https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump would probably be more helpful.

[Daniel Becroft]

Thanks, Daniel. I've enabled the debug mode for email-ext to see what it can provide. I'll also check out what the thread dump provides when the next build happens.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/DAD0D2CF-AF27-4628-BC6F-7F06F3CF0BE3%40beckweb.net.
For more options, visit https://groups.google.com/d/optout.

[Daniel Becroft]

The logging in the email-ext plugin didn't provide much information, other than that it takes approx. the same time for each committer in the build. ie 1min / committer.

Google helped me to enable logging on the Active Directory plugin, and I have found the following:

1) We have our domain name configured (ad.example.com), as well as the domain controller defined with the virtual IP (x.x.x.x:389)

2) The default lookup is via AD on the userPrincipalName, which it calculates as <committer-name>@<domain-name>.  In our case, our domain name and email addresses are slightly different (domain == ad.example.com, email == example.com). This meant that the first query was userPrincipleName == john...@ad.example.com, which was never going to be found (user is john...@example.com)

3) Based on (2), I changed the domain name to match the email address, but this then raised various javax.naming.CommunicationException errors, with connections attempted to example.com:389. The error makes sense, as that isn't how the domain controller is referenced, but I don't know why it's ignoring the value in the Domain Controller field (x.x.x.x:389).

4) It then seems to enumerate all groups of the user is a member of (which there are hundreds) even though the we have configured it to "Remove irrelevant groups" (or maybe I misunderstood what that flag is meant to do).

I've even got the cache configured for 1 hour, but it still does this lookup every build. 

Right now, it looks like the default options for the Active Directory plugin don't work for our environment, and we need to go down the LDAP configuration instead.

Cheers,

Daniel B.