The pipeline emailextrecipients step takes too long time

53 views
Skip to first unread message

Sverre Moe

unread,
Apr 24, 2020, 4:59:20 PM4/24/20
to Jenkins Users
Calling the pipeline step emailextrecipients when there are changes takes too much time.
A normal git checkout that takes 10-15 seconds, will usually take 1-2 minutes when there are changes, just because of emailextrecipients.
All I want is the authors from the current build changes. I see no reason why this should take so long.

I took a small look at the code for emailextrecipients, and it seems it looks also on previous builds.

def recipients = emailextrecipients([developers()])

Our developers want the builds to run as fast as it can. Time is precious.
I am considering removing this recipients. I already have the git repository maintainers I can send email notifications to if anything goes wrong.
The backside of removing this, is I cannot send email notifications for the developers, just the maintainers (and those only want notifications for release build, and not all continuous builds).

The reason I call emailextrecipients during the Checkout stage, is because that is the only stage where I have access to the git repository. Later in the build if it fails I do not have access to git anymore.

The emailextrecipients code does have some Debug logging. How can I enable this to see what it is actually doing?

I was thinking of implementing my own parsing of the change records. It cannot possibly take more than a few seconds to find the authors of all the changes.

Slide

unread,
Apr 24, 2020, 5:20:40 PM4/24/20
to Jenkins User Mailing List
You can turn on debug mode in the global config for Email Ext. The code isn't really doing anything major, it just looks at the changesets and get the authors, so I am not sure why it would be taking that long.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de0f53c9-9512-4fd6-a258-e604412ff77a%40googlegroups.com.


--
Website: http://earl-of-code.com

Sverre Moe

unread,
Apr 25, 2020, 2:58:41 PM4/25/20
to Jenkins Users
Tried to configure debug
Manage Jenkins > Configure System > Extended Email Notification >Enable Debug

I added a System Log for "hudson.plugins.emailext" and Log Level ALL.

Hope I can find something about why it takes so long.

fredag 24. april 2020 23.20.40 UTC+2 skrev slide følgende:
You can turn on debug mode in the global config for Email Ext. The code isn't really doing anything major, it just looks at the changesets and get the authors, so I am not sure why it would be taking that long.

On Fri, Apr 24, 2020 at 1:59 PM Sverre Moe <sver...@gmail.com> wrote:
Calling the pipeline step emailextrecipients when there are changes takes too much time.
A normal git checkout that takes 10-15 seconds, will usually take 1-2 minutes when there are changes, just because of emailextrecipients.
All I want is the authors from the current build changes. I see no reason why this should take so long.

I took a small look at the code for emailextrecipients, and it seems it looks also on previous builds.

def recipients = emailextrecipients([developers()])

Our developers want the builds to run as fast as it can. Time is precious.
I am considering removing this recipients. I already have the git repository maintainers I can send email notifications to if anything goes wrong.
The backside of removing this, is I cannot send email notifications for the developers, just the maintainers (and those only want notifications for release build, and not all continuous builds).

The reason I call emailextrecipients during the Checkout stage, is because that is the only stage where I have access to the git repository. Later in the build if it fails I do not have access to git anymore.

The emailextrecipients code does have some Debug logging. How can I enable this to see what it is actually doing?

I was thinking of implementing my own parsing of the change records. It cannot possibly take more than a few seconds to find the authors of all the changes.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

Slide

unread,
Apr 25, 2020, 3:51:25 PM4/25/20
to Jenkins User Mailing List
I think the debug logs will just go to the build log.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fbf04c99-b61b-4109-ac21-50a77fca08c0%40googlegroups.com.

Sverre Moe

unread,
Apr 26, 2020, 4:15:51 AM4/26/20
to Jenkins Users
Yes it did.

Just one simple little commit, and Checkout took a little over 1 minute

[Pipeline] emailextrecipients
  Collecting change authors...
    build: 168
      adding author: Sverre Moe

Adding Sverre Moe with address sve...@company.com
Analyzing: sve...@company.com
Looking for: sve...@company.com
	starting at: 0
	firstFoundIdx: 0
	firstFoundIdx-substring: sve...@company.com
	=> found type: 0
Analyzing: sve...@company.com
Looking for: sve...@company.com
	starting at: 0
	firstFoundIdx: 0
	firstFoundIdx-substring: sve...@company.com
	=> found type: 0
Analyzing: sve...@company.com
Looking for: sve...@company.com
	starting at: 0
	firstFoundIdx: 0
	firstFoundIdx-substring: sve...@company.com
	=> found type: 0

lørdag 25. april 2020 21.51.25 UTC+2 skrev slide følgende:
I think the debug logs will just go to the build log.

Sverre Moe

unread,
Apr 26, 2020, 10:37:33 AM4/26/20
to Jenkins Users
I added println how much time it used.
Mostly it uses only a few milliseconds.
In one case so far, It used a little over 60000 milliseconds.

The output from the emailextrecipients debug was no different from those build that only took a few milliseconds.

Slide

unread,
Apr 26, 2020, 12:30:59 PM4/26/20
to Jenkins User Mailing List
Can you add timestamps? It's hard to tell from what you posted when things are occurring.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d96643c8-b4d1-46bc-9339-c5094dd2d3c7%40googlegroups.com.

jeremy mordkoff

unread,
Apr 26, 2020, 2:12:03 PM4/26/20
to Jenkins Users
check that DNS is working. I've seen huge delays on basic ops when DNS lookups are timing out. Check forward and reverse lookups. 



On Sunday, April 26, 2020 at 12:30:59 PM UTC-4, slide wrote:
Can you add timestamps? It's hard to tell from what you posted when things are occurring.

Sverre Moe

unread,
Apr 26, 2020, 3:31:51 PM4/26/20
to Jenkins Users
How can I add timestamp to the debug output?

søndag 26. april 2020 18.30.59 UTC+2 skrev slide følgende:
Can you add timestamps? It's hard to tell from what you posted when things are occurring.

Sverre Moe

unread,
Apr 26, 2020, 3:32:42 PM4/26/20
to Jenkins Users
How so?

Does emailextrecipient do a DNS lookup on each author it can find?

Jan Monterrubio

unread,
Apr 26, 2020, 6:30:33 PM4/26/20
to jenkins...@googlegroups.com
Use the timestamps plugin , it has a mode to add global timestamps or you can configure options per pipeline 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6ea53ce0-e5a3-4d62-90a7-47ba43a482a1%40googlegroups.com.

Jeremy Mordkoff

unread,
Apr 26, 2020, 7:12:48 PM4/26/20
to Jenkins Users
sending email always involves a forward lookup and many mail servers do a reverse lookup on the sender's IP. 


---
Jeremy Mordkoff
Director of Engineering Services
RIFT, inc
900 Chelmsford Street,  4th floor, tower 3
Lowell, MA 01851
From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> on behalf of Sverre Moe <sverr...@gmail.com>
Sent: Sunday, April 26, 2020 3:32 PM
To: Jenkins Users <jenkins...@googlegroups.com>
Subject: Re: The pipeline emailextrecipients step takes too long time
 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/JvBIc0_EXsg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c30c15e2-6aa8-4016-a27e-c4711607642b%40googlegroups.com.

Sverre Moe

unread,
Apr 27, 2020, 4:39:30 AM4/27/20
to Jenkins Users
Well at this point I am not sending any email, yet.
I am looking up recipients from change set, so I can use that later when/if I send email for failed builds.

Does emailextrecipients do the lookup itself, or is it done when calling mail step?


mandag 27. april 2020 01.12.48 UTC+2 skrev Jeremy Mordkoff følgende:
sending email always involves a forward lookup and many mail servers do a reverse lookup on the sender's IP. 


---
Jeremy Mordkoff
Director of Engineering Services
RIFT, inc
900 Chelmsford Street,  4th floor, tower 3
Lowell, MA 01851
To unsubscribe from this group and all its topics, send an email to jenkins...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c30c15e2-6aa8-4016-a27e-c4711607642b%40googlegroups.com.

Sverre Moe

unread,
Apr 27, 2020, 4:40:55 AM4/27/20
to Jenkins Users
This plugin you are mentioning?


mandag 27. april 2020 00.30.33 UTC+2 skrev Jan Monterrubio følgende:
Use the timestamps plugin , it has a mode to add global timestamps or you can configure options per pipeline 

Sverre Moe

unread,
Apr 27, 2020, 10:19:28 AM4/27/20
to Jenkins Users
It seems emailextrecipients are using 60 seconds for each email it finds.

I am going to add the timestamper plugin to see what parts of the debug output from emailextrecipients are taking so long.

Slide

unread,
Apr 27, 2020, 10:25:37 AM4/27/20
to Jenkins User Mailing List
There is no lookup until the email is sent. There may be a call to resolve a username to the email address though. Are you using ldap or AD?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3ea8d7eb-4b22-48ca-9f05-5f7fcc86d0d8%40googlegroups.com.

Sverre Moe

unread,
Apr 27, 2020, 10:36:51 AM4/27/20
to Jenkins Users
Our Jenkins have enabled Login with LDAP.

We do occasionally experience LDAP timeout also. 60 seconds timeout.
Perhaps there is a relation to that, considering that each author found by emailextrecipients also takes 60 seconds each.


mandag 27. april 2020 16.25.37 UTC+2 skrev slide følgende:
There is no lookup until the email is sent. There may be a call to resolve a username to the email address though. Are you using ldap or AD?

Slide

unread,
Apr 27, 2020, 1:43:01 PM4/27/20
to Jenkins User Mailing List
That sounds like the culprit to me. Email-ext has to resolve usernames from the changeset so an email address. It provides an extension point for this that other plugins can implement to provide that service. If you are having long delays accessing LDAP, that would definitely point to that as a culprit for the length of time.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7aa49cb-cf0a-4fd1-8d52-547e54ea6e6c%40googlegroups.com.


--
Website: http://earl-of-code.com

Slide

unread,
Apr 27, 2020, 3:16:25 PM4/27/20
to Jenkins User Mailing List
Sorry, I misspoke about this. The email-ext plugin uses ChangeSet.Entry.getAuthor(), depending on the SCM plugin you are using, it can do lookups like I mentioned. The Git plugin will try and do the resolution of username -> email using the mail resolver from the Mailer plugin (which is where the extension point lives). I was confusing where the resolver was in the infrastructure. The TL;DR is that it is still the MailResolver feature that is taking the time, but the sequence was different than I remembered.
--
Website: http://earl-of-code.com

Sverre Moe

unread,
Apr 28, 2020, 7:58:23 AM4/28/20
to Jenkins Users
Any idea how to workaround this?

Any way to disable the resolving of email adresses?
Using Git SCM checkout in Pipeline

/Sverre


mandag 27. april 2020 21.16.25 UTC+2 skrev slide følgende:
Sorry, I misspoke about this. The email-ext plugin uses ChangeSet.Entry.getAuthor(), depending on the SCM plugin you are using, it can do lookups like I mentioned. The Git plugin will try and do the resolution of username -> email using the mail resolver from the Mailer plugin (which is where the extension point lives). I was confusing where the resolver was in the infrastructure. The TL;DR is that it is still the MailResolver feature that is taking the time, but the sequence was different than I remembered.

On Mon, Apr 27, 2020 at 10:42 AM Slide <slid...@gmail.com> wrote:
That sounds like the culprit to me. Email-ext has to resolve usernames from the changeset so an email address. It provides an extension point for this that other plugins can implement to provide that service. If you are having long delays accessing LDAP, that would definitely point to that as a culprit for the length of time.

Slide

unread,
Apr 28, 2020, 11:07:28 AM4/28/20
to Jenkins User Mailing List
How would emails get to the correct user if their email address can't be resolved from the username? I don't know of a way to disable the username -> email resolution. Did you try changing the search criteria for the LDAP plugin as related in one of your other posts?

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1b23d07c-39c0-4546-9875-c7e0e711f09b%40googlegroups.com.


--
Website: http://earl-of-code.com

Reinhold Füreder

unread,
Apr 28, 2020, 11:17:14 AM4/28/20
to jenkins...@googlegroups.com

Sorry for the following potentially too naïve hint:

 

In addition try to also play around with the LDAP caching, if not yet activated – because that may also reduce load on LDAP server (in case that is a problem); of course, other LDAP clients in your company should do the same!

 

(I remember that in our company a single badly configured server tool doing too many LDAP requests managed to led to quite interesting follow-up failure symptoms for other systems…)

 

HTH Reinhold

Sverre Moe

unread,
Apr 28, 2020, 4:52:06 PM4/28/20
to Jenkins Users
The LDAP Search criteria:
Not sure how the search criterias could be changed.
User search base: ou=users
User search filter: sAMAccountName={0}


Found this setting in Configure Global Security for LDAP:
Disable Ldap Email Resolver
I ticked it, now I will see if that can help the issue.


LDAP Caching:
I had not enabled cache.
Enabled it, and set Size=100, TTL=1 hour
Does this mean that the cache only lives for 1 hour? That seems a little.

/Sverre


tirsdag 28. april 2020 17.17.14 UTC+2 skrev r.fuereder følgende:

Sorry for the following potentially too naïve hint:

 

In addition try to also play around with the LDAP caching, if not yet activated – because that may also reduce load on LDAP server (in case that is a problem); of course, other LDAP clients in your company should do the same!

 

(I remember that in our company a single badly configured server tool doing too many LDAP requests managed to led to quite interesting follow-up failure symptoms for other systems…)

 

HTH Reinhold

 

From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> On Behalf Of Slide
Sent: Dienstag, 28. April 2020 17:07
To: Jenkins User Mailing List <jenkins...@googlegroups.com>
Subject: Re: The pipeline emailextrecipients step takes too long time

 

How would emails get to the correct user if their email address can't be resolved from the username? I don't know of a way to disable the username -> email resolution. Did you try changing the search criteria for the LDAP plugin as related in one of your other posts?


To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.


To view this discussion on the web visit

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.


To view this discussion on the web visit

Sverre Moe

unread,
Apr 29, 2020, 4:44:11 AM4/29/20
to Jenkins Users
This had no or very little impact on the time used by emailextrecipient.

Disbled email resolver against LDAP
Enabled LDAP cache

Checked a build that was built in Jenkins after changing these settings.
The emailextrecipients took 98274 milliseconds, almost 2 minutes, for 2 authors in the change sets.

/Sverre

Reinhold Füreder

unread,
Apr 29, 2020, 5:45:16 AM4/29/20
to jenkins...@googlegroups.com

Ad LDAP: we are also using another configuration element, but again, this may not be kriegs-entscheidend

userSearchBase: "CN=Users"

 

If you know that it is the emailextrecipients step and it is really that slow, then you could maybe find the problem when remote debugging into Jenkins?

 

But I guess that email-ext insider Slide might have much better hints…

 

HTH Reinhold

 

From: jenkins...@googlegroups.com <jenkins...@googlegroups.com> On Behalf Of Sverre Moe
Sent: Mittwoch, 29. April 2020 10:44
To: Jenkins Users <jenkins...@googlegroups.com>
Subject: Re: The pipeline emailextrecipients step takes too long time

 

This had no or very little impact on the time used by emailextrecipient.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/172bfdd6-f04e-4011-b0c0-5062585c80a2%40googlegroups.com.

Sverre Moe

unread,
Apr 29, 2020, 7:50:18 AM4/29/20
to Jenkins Users
The emailextrecipient gets the author by
ChangeSet getAuthor()

This gets the Name of the author. Then I guess it needs to find the email address.
But how does it do that?
Does it look up the name in LDAP?
What would it do if LDAP was not configured in Jenkins?

It would be so much easier if it just could call
ChangeSet getAuthorEmail()

A build that had changes with 11 authors, used a little over 10 minutes to run emailextrecipient.

I made my own code snippet to do pretty much the same, and it should be considerably faster (not yet tested).

def getAuthors(build) {
    final def authors = []
    final def changeSets = build.changeSets
    for (def cs: changeSets) {
        final def author = cs.getAuthorEmail()
        authors.add(author)
    }
    return authors.join(" ")
}

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.


To view this discussion on the web visit

Sverre Moe

unread,
Apr 29, 2020, 11:18:04 AM4/29/20
to Jenkins Users
My code for finding the changeset authors worked fine.
If any developers have wrong email address, then so be it.
This solution is much much faster.

def getAuthors(build) {
    final def authors = []
    final def changeSets = build.changeSets
    for (def cs: changeSets) {
        final def author = cs.getAuthorEmail()
        authors.add(author)
    }
    return authors.join(" ")
}

Though it would preffer to use emailextrecipients, because it can do so much more. Not just developers, but culprits, etc.

The emailextrecipient could use an configuration option to retrieve the AuthorEmail instead directly, no lookup/resolve needed.

Slide

unread,
Apr 29, 2020, 8:18:25 PM4/29/20
to Jenkins User Mailing List
The only method available via the ChangeSet API is getAuthor(), which internally does the lookup to make sure the email address is available, so no, it can't access author email directly.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/c6fa5643-4330-4b4f-a59f-0216f9f214f9%40googlegroups.com.


--
Website: http://earl-of-code.com

Sverre Moe

unread,
Apr 30, 2020, 3:42:26 PM4/30/20
to Jenkins Users
I see that getAuthorEmail is only part of the GitChangeSet not its super ChangeSet.Entry.

Seems better to get the Email rather the Name and then do lookup. The commit has both Name and Email, both should belong in the ChangeSet API.
What does it do if you don't have AD or LDAP on Jenkins, then there is nothing to lookup.

torsdag 30. april 2020 02.18.25 UTC+2 skrev slide følgende:
The only method available via the ChangeSet API is getAuthor(), which internally does the lookup to make sure the email address is available, so no, it can't access author email directly.

On Wed, Apr 29, 2020 at 8:18 AM Sverre Moe <sver...@gmail.com> wrote:
My code for finding the changeset authors worked fine.
If any developers have wrong email address, then so be it.
This solution is much much faster.

def getAuthors(build) {
    final def authors = []
    final def changeSets = build.changeSets
    for (def cs: changeSets) {
        final def author = cs.getAuthorEmail()
        authors.add(author)
    }
    return authors.join(" ")
}

Though it would preffer to use emailextrecipients, because it can do so much more. Not just developers, but culprits, etc.

The emailextrecipient could use an configuration option to retrieve the AuthorEmail instead directly, no lookup/resolve needed.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins...@googlegroups.com.

Slide

unread,
Apr 30, 2020, 5:14:18 PM4/30/20
to Jenkins User Mailing List
Some scm providers can map usernames to emails directly. So, it doesn't require AD or LDAP for email resolution. In the end, if none of the resolvers returns a valid response, then the default domain is appended to the username and that is used.

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/5691f5d0-dae8-4452-bf7e-3993f5df17fd%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages