[JIRA] [sauce-ondemand-plugin] (JENKINS-27111) Sauce Connect intermittently fails to launch

113 views
Skip to first unread message

nguilford@gmail.com (JIRA)

unread,
Feb 24, 2015, 2:06:34 PM2/24/15
to jenkinsc...@googlegroups.com
Issue Type: Bug Bug
Assignee: Kohsuke Kawaguchi
Components: sauce-ondemand-plugin
Created: 24/Feb/15 7:06 PM
Description:

Intermittently, SauceConnect fails to launch using the OnDemand plugin in Jenkins, failing builds. Usually, this does not appear to be a problem with SauceLabs service, as status.saucelabs.com does not report error, and I am able to launch SauceConnect manually on various other machines.

Unfortunately, this is not readily reproducible – it happens intermittently and seemingly randomly. Hopefully, the stack trace will give you some idea.

Details:

  • Sauce Connect launched on slave using credentials set in job
  • v3 checkbox is unchecked
  • Slave is 64bit linux box running Debian 6.0.10 in ec2
  • Configured to use webdriver

Log:
Starting Sauce Connect on slave node using tunnel identifier: default
Launching Sauce Connect on <hostname>
Error launching Sauce Connect
Flushing Sauce Connect Input Stream
Flushing Sauce Connect Error Stream
Closing Sauce Connect process
FATAL: Error launching Sauce Connect
com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:286)
at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:710)
at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:663)
at hudson.remoting.UserRequest.perform(UserRequest.java:121)
at hudson.remoting.UserRequest.perform(UserRequest.java:49)
at hudson.remoting.Request$2.run(Request.java:324)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
at ......remote call to build9.www.dash.d1.usw1.opendns.com(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
at hudson.remoting.Channel.call(Channel.java:752)
at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:223)
at hudson.model.Build$BuildExecution.doRun(Build.java:154)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
at hudson.model.Run.execute(Run.java:1718)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:89)
at hudson.model.Executor.run(Executor.java:240)
Environment: Sauce OnDemand Plug-in v1.109, SauceConnect v4,Jenkins v1.598, Debian v6.0.10 (64 bit)
Project: Jenkins
Priority: Major Major
Reporter: Nick Guilford
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

piaross@gmail.com (JIRA)

unread,
Mar 5, 2015, 1:10:32 PM3/5/15
to jenkinsc...@googlegroups.com
Change By: Ross Rowe (05/Mar/15 6:08 PM)
Assignee: Kohsuke Kawaguchi Ross Rowe

piaross@gmail.com (JIRA)

unread,
Mar 5, 2015, 1:12:32 PM3/5/15
to jenkinsc...@googlegroups.com
Ross Rowe commented on Bug JENKINS-27111

Hi Nick, would you be able to check the Sauce Connect log file to see what errors are being logged? By default, this is written to the /tmp/sc.log file, but you can change this location by specifying a value for the --logfile command line argument.

Cheers,

Ross

piaross@gmail.com (JIRA)

unread,
Mar 5, 2015, 1:12:32 PM3/5/15
to jenkinsc...@googlegroups.com
Ross Rowe started work on Bug JENKINS-27111
Change By: Ross Rowe (05/Mar/15 6:11 PM)
Status: Open In Progress

michael.wang@asti-usa.com (JIRA)

unread,
Mar 11, 2015, 4:34:33 PM3/11/15
to jenkinsc...@googlegroups.com

I'm getting something similar(I'm unsure if it's the exact same). This is the console output 

Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/protractor/workspace
Starting Sauce Connect on master node using identifier: default
Launching Sauce Connect on <<my host name>>
Time out while waiting for Sauce Connect to start, please check the Sauce Connect log
Flushing Sauce Connect Input Stream
Flushing Sauce Connect Error Stream
Closing Sauce Connect process
FATAL: Time out while waiting for Sauce Connect to start, please check the Sauce Connect log
com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Time out while waiting for Sauce Connect to start, please check the Sauce Connect log
	at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:310)
	at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:714)
	at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:237)
	at hudson.model.Build$BuildExecution.doRun(Build.java:154)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
	at hudson.model.Run.execute(Run.java:1718)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:89)
	at hudson.model.Executor.run(Executor.java:240)

and my Sauce Connect log file says this 

20150311 195842.975 [29352] Sauce Connect 4.3.6, build 1628 8a5c837
20150311 195842.976 [29352] Using CA certificate bundle /etc/ssl/certs/ca-bundle.crt.
20150311 195842.976 [29352] Warning, open file limit 4096 is too low.
20150311 195842.976 [29352] Sauce Labs recommends setting it to at least 8000.
20150311 195842.976 [29352] Trying to autodetect proxy settings
20150311 195842.976 [29352] can't lock pidfile /tmp/sc_client.pid. Please check if Sauce Connect is already running.
20150311 195842.976 [29352] Error creating pidfile /tmp/sc_client.pid.

piaross@gmail.com (JIRA)

unread,
Mar 11, 2015, 6:26:31 PM3/11/15
to jenkinsc...@googlegroups.com
Ross Rowe commented on Bug JENKINS-27111

Michael Wang, thanks for letting me know...can you let me know what version of the Sauce Jenkins plugin you're using? The latest version (v1.119) should detect if Sauce Connect is already running and if so, terminate the Sauce Connect process, which should also remove the pidfile. If the behaviour is occurring with the latest version, please let me know, and I'll try include a fix.

michael.wang@asti-usa.com (JIRA)

unread,
Mar 12, 2015, 8:46:31 AM3/12/15
to jenkinsc...@googlegroups.com

I was on 1.117 I will terminate SC upgrade and try again and let you know.

alex.white@oval.uk.com (JIRA)

unread,
Mar 20, 2015, 7:13:35 AM3/20/15
to jenkinsc...@googlegroups.com
Alex White commented on Bug JENKINS-27111

I appear to be having a similar issue with v1.120. I was able to tunnel to saucelabs through direct from Eclipse IDE, but am unable to run through Jenkins from the same machine.

Starting Sauce Connect on master node using identifier: default
Launching Sauce Connect on OV00039
Error launching Sauce Connect
Flushing Sauce Connect Input Stream
Flushing Sauce Connect Error Stream
Closing Sauce Connect process
FATAL: Error launching Sauce Connect
com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager$SauceConnectDidNotStartException: Error launching Sauce Connect
	at com.saucelabs.ci.sauceconnect.AbstractSauceTunnelManager.openConnection(AbstractSauceTunnelManager.java:286)
	at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper$SauceConnectHandler.call(SauceOnDemandBuildWrapper.java:726)
	at hudson.plugins.sauce_ondemand.SauceOnDemandBuildWrapper.setUp(SauceOnDemandBuildWrapper.java:237)
	at hudson.model.Build$BuildExecution.doRun(Build.java:154)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536)
	at hudson.model.Run.execute(Run.java:1751)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:89)
	at hudson.model.Executor.run(Executor.java:240)

here is the log file

20150320 105709.201 [03532] Sauce Connect 4.3.7, build 1660 249d939
20150320 105709.208 [03532] Trying to autodetect proxy settings
20150320 105709.212 [03532] Starting up; pid 3532
20150320 105709.213 [03532] Command line arguments: C:\sc-4.3.7-win32\bin\sc.exe -u alexwhiteoval -k **** -P 4445
20150320 105709.214 [03532] Using no proxy for connecting to Sauce Labs REST API.
20150320 105709.218 [03532] Resolving saucelabs.com to 162.222.73.28 took 4 ms.
20150320 105734.612 [03532] PROXY *** scproxy started ***
20150320 105734.614 [03532] PROXY found DNS server 192.168.1.10
20150320 105734.615 [03532] PROXY found DNS server 8.8.8.8
20150320 105734.615 [03532] PROXY found DNS server 192.168.1.10
20150320 105734.615 [03532] PROXY found DNS server 8.8.8.8
20150320 105734.616 [03532] PROXY found DNS server 192.168.1.10
20150320 105734.616 [03532] PROXY found DNS server 8.8.8.8
20150320 105734.781 [03532] Started scproxy on port 52462.
20150320 105734.781 [03532] Please wait for 'you may start your tests' to start your tests.
20150320 105734.781 [03532] Starting secure remote tunnel VM...
20150320 105828.580 [03532] POST https://saucelabs.com/rest/v1/alexwhiteoval/tunnels: Timeout was reached.
20150320 105828.580 [03532] Error creating tunnel.
20150320 105828.580 [03532] Sauce Connect could not establish a connection.
20150320 105828.580 [03532] Please check your firewall and proxy settings.
20150320 105828.580 [03532] You can also use sc --doctor to launch Sauce Connect in diagnostic mode.
20150320 105828.580 [03532] Cleaning up.
20150320 105855.609 [03532] Checking domain overlap for my domain sauce-connect.proxy, other tunnel domain sauce-connect.proxy
20150320 105855.609 [03532] Overlapping domain: sauce-connect.proxy, shutting down tunnel c4e64c2110804878a8d5dd136a69420b.
20150320 105925.604 [03532] DELETE https://saucelabs.com/rest/v1/alexwhiteoval/tunnels/c4e64c2110804878a8d5dd136a69420b: Timeout was reached.
20150320 105925.606 [03532] Goodbye.

piaross@gmail.com (JIRA)

unread,
Mar 20, 2015, 6:00:32 PM3/20/15
to jenkinsc...@googlegroups.com
Ross Rowe commented on Bug JENKINS-27111

Alex White - would you be able to raise a support issue for the Sauce Connect startup problem? I think the issue is related to Sauce Connect being able to connect to the Sauce Labs servers from the workstation/server that Jenkins is running from - are you using a proxy server to connect to the internet? If so, then you will need to include the proxy information in the command line arguments sent to Sauce Connect

alex.white@oval.uk.com (JIRA)

unread,
Mar 23, 2015, 4:56:31 AM3/23/15
to jenkinsc...@googlegroups.com
Alex White commented on Bug JENKINS-27111

Thanks Ross, I have logged a support ticket, although, we are not running through a proxy as our connection is direct. I have noticed that there is a sauce-ondemand plugin update which I am installing to see if this will resolve the issue.

alex.white@oval.uk.com (JIRA)

unread,
Mar 23, 2015, 8:07:31 AM3/23/15
to jenkinsc...@googlegroups.com
 
Alex White edited a comment on Bug JENKINS-27111

Thanks Ross, I have logged a support ticket, although, we are not running through a proxy as our connection is direct. I have noticed that there is a sauce-ondemand plugin update which I am installing to see if this will resolve the issue.

edit:
1.121 has resolved the issue.

nguilford@gmail.com (JIRA)

unread,
Mar 24, 2015, 7:59:31 PM3/24/15
to jenkinsc...@googlegroups.com

Hi Ross, I am now on version 1.120 and still reproduce this from time to time. Like Michael Wang, this appears to be due to not being able to get a lock on the pidfile:

20150324 232500.032 [16723] Sauce Connect 4.3.7, build 1658 d347dd3
20150324 232500.032 [16723] Using CA certificate bundle /etc/ssl/certs/ca-certificates.crt.
20150324 232500.032 [16723] Trying to autodetect proxy settings
20150324 232500.032 [16723] can't lock pidfile /tmp/sc_client.pid. Please check if Sauce Connect is already running.
20150324 232500.032 [16723] Error creating pidfile /tmp/sc_client.pid.
20150324 232611.603 [27513] Cleaning up.

piaross@gmail.com (JIRA)

unread,
Mar 25, 2015, 4:02:33 AM3/25/15
to jenkinsc...@googlegroups.com
Ross Rowe commented on Bug JENKINS-27111

Nick Guilford - thanks for letting me know...can you tell me if you have multiple Jenkins jobs which are configured to launch Sauce Connect? Would you mind raising a support ticket with Sauce Labs referencing this issue and attach the complete sc.log file (and also indicate that the ticket should be assigned to me? If it's possible, when you next get the error could you check to see if there are any other Sauce Connect processes running on the Jenkins instance?

mmerritt1010@gmail.com (JIRA)

unread,
Apr 1, 2015, 7:32:10 PM4/1/15
to jenkinsc...@googlegroups.com

@Ross - Logged a sauce labs support ticket with requested information on behalf of nick.

mmerritt1010@gmail.com (JIRA)

unread,
Apr 7, 2015, 7:15:06 PM4/7/15
to jenkinsc...@googlegroups.com
Michael Merritt resolved Bug JENKINS-27111 as Fixed

This appears to be resolved with Sauce Connect 4.3.8

Will continue to test and re-open if we see it again. Thanks for your help Ross.
Change By: Michael Merritt (07/Apr/15 11:13 PM)
Status: In Progress Resolved
Resolution: Fixed
Reply all
Reply to author
Forward
0 new messages