"more SCM polling activities scheduled than handled..."

[Josh Santangelo]

Hi all -- I'm seeing the "There are more SCM polling activities scheduled than handled, so the threads are not keeping up with the demands" error consistently on my Jenkins setup (2.3, running as a Windows service).

Going to my http://jenkins/descriptor/hudson.triggers.SCMTrigger/ page does show jobs hanging for hours at a time, but always different jobs.

I currently have 14 jobs enabled which are polling git repos on BitBucket.org (I had more, but I disabled a bunch trying to troubleshoot this issue).

On the "Manage Jenkins" page I set "Max # of concurrent polling" to empty, which seems to mean an unlimited number of threads, but the issue remains.

Is there anything else I should look at?

thanks,

-josh

[Baptiste Mathus]

Hi,

The ideal target you should actually strive for is to "kill polling", and use push instead, as explained in http://kohsuke.org/2011/12/01/polling-must-die-triggering-jenkins-builds-from-a-git-hook/

If you really really cannot consider that, then in your place I would maybe try and mimic what Jenkins is supposedly doing: on your machine, write a script that clones repo, or updates it every minute (or something similar with what you put in Jenkins), during say an hour, or a day. Goal being to have data about connectivity.

Maybe it's just bitbucket.org that's inconsistent, and sometimes never answers. Or maybe not. IMO that's the first thing you want to look at. 

For example, looking at https://status.bitbucket.org/ I see issues around the date of your message, was your issue that day or something that's been consistently problematic across weeks?

-- Baptiste

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/03919522-cec0-43e7-8c9f-ed102234a3a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Josh Santangelo]

I’d love to stop polling, but with Jenkins behind our firewall and git outside of it, the conversation is only one way.

Here’s what I’m seeing in the git polling log on those processes. I would accept that if it timed out it would just try again later, but once it gets in this state, the project never automatically builds again. I can even manually do a build which will be successful, but polling never sorts itself out.

poll] Last Built Revision: Revision e5779a7ee33a8c58a4e13c9930f02b67a07c964a (refs/remotes/origin/multitaction)

 > C:\Program Files (x86)\Git\cmd\git.exe ls-remote -h g...@bitbucket.org:stimulant/DWD.git # timeout=10

ERROR: Timeout after 10 minutes

ERROR

: Failed to join a process

org.jvnet.winp.WinpException

: Failed to read environment variable table error=299 at .\envvar-cmdline.cpp:201

at org.jvnet.winp.Native.getCmdLineAndEnvVars(Native Method)

at org.jvnet.winp.WinProcess.parseCmdLineAndEnvVars(WinProcess.java:126)

at org.jvnet.winp.WinProcess.getCommandLine(WinProcess.java:102)

at hudson.util.ProcessTree$Windows$1.getArguments(ProcessTree.java:444)

at hudson.plugins.msbuild.MsBuildKillingVeto.vetoProcessKilling(MsBuildKillingVeto.java:55)

at hudson.util.ProcessTree$OSProcess.getVeto(ProcessTree.java:242)

at hudson.util.ProcessTree$Windows$1.killRecursively(ProcessTree.java:425)

at hudson.util.ProcessTree.killAll(ProcessTree.java:145)

at hudson.Proc$LocalProc.destroy(Proc.java:378)

at hudson.Proc$LocalProc.kill(Proc.java:370)

at hudson.Proc$1.run(Proc.java:157)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

-- 
S T I M U L Λ N T
Josh Santangelo, Technical Director
P 415 363 0336 / HQ 415 255 7081 / T @stimulant

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/vgdwV1et_oA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5CACK9LYB7D63r-S9J49RtiVXEhk6GPRveHiaDgAbKCQ%40mail.gmail.com.

[Donald Morton]

Maybe try setting it at 100 threads? That is the max it will do. Not sure if setting it blank does the same thing or not. 

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/03919522-cec0-43e7-8c9f-ed102234a3a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/vgdwV1et_oA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsub...@googlegroups.com.

[Josh Santangelo]

Setting it to 100 and restarting didn’t change the behavior. At the moment I have nine jobs hung with he same error log as below.

The fact that git is timing out is clearly a BitBucket issue (maybe they think I’m DOSing them) but Jenkins should recover from it correctly.

-- 
S T I M U L Λ N T
Josh Santangelo, Technical Director
P 415 363 0336 / HQ 415 255 7081 / T @stimulant

To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/246915d1-f5c0-4ebc-b947-63a7116ffc4e%40googlegroups.com.

[Josh Santangelo]

I'm still really stuck on this. I'm working with Bitbucket on it but it doesn't seem to be their problem. Things I've tried:

Reducing the number of polling jobs, now just down to 7.

Bitbucket suggested trying HTTPS instead of SSH. I am able to access the repos using HTTPS and git-bash on Windows, but using the regular command shell I get an error "Unable to find remote helper for 'https'" that I've been unable to resolve.

I change the "shell executable" setting on Jenkins to C:\Program Files\Git\git-bash.exe, but it had no effect.

Is there anything else I can try?

-- 

S T I M U L Λ N T

Josh Santangelo, Technical Director

P 206 737 5601 / HQ 415 255 7081 / T @stimulant

To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/03919522-cec0-43e7-8c9f-ed102234a3a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/vgdwV1et_oA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS5CACK9LYB7D63r-S9J49RtiVXEhk6GPRveHiaDgAbKCQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/vgdwV1et_oA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsub...@googlegroups.com.