difficulty connecting to Mac slave

[Dan Tenenbaum]

Hi all,

I have tried a number of methods to connect to a Mac slave and have had no success.

My master is also a mac and it's called dhcp151178. The slave I'm trying to connect to is called pelham. These of course are machines on my internal network. 

The first method I tried was "Launch slave agents on Unix machines via SSH".

It looks like the master can connect to the slave via ssh but then I get the following (excerpted; full stack traces at end of message):

java.io.IOException: Remote call on pelham failed

...

Caused by: java.lang.InternalError: Can't connect to window server - not enough permissions.

Then I tried "Launch slave agents via Java Web Start"
And on the slave I navigated to http://dhcp151178:8080/computer/pelham/slave-agent.jnlp
This gave me:

java.lang.Exception: The server rejected the connection: Unauthorized access
at hudson.remoting.Engine.onConnectionRejected(Engine.java:257)
at hudson.remoting.Engine.run(Engine.java:232)

Also, based on the following post:
http://stackoverflow.com/questions/11824486/setting-up-a-hudson-slave-on-windows-xp

I tried downloading the jnlp file from the URL above, replacing all instances of 'localhost' with 'dhcp151178' and launching the jnlp with javaws, this either gives me no console output at all (but the slave agent gui says Terminated) or it says:

...
Exception in thread "main" java.lang.InternalError: Can't connect to window server - not enough permissions.

I tried starting in headless mode, both as jenkins and as my regular admin user. I did this on the slave:

java -jar slave.jar -jnlpUrl http://dhcp151178:8080/computer/pelham/slave-agent.jnlp
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://localhost:8080/, http://dhcp151178:8080/]
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to localhost:49161
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: Unauthorized access
java.lang.Exception: The server rejected the connection: Unauthorized access
	at hudson.remoting.Engine.onConnectionRejected(Engine.java:257)
	at hudson.remoting.Engine.run(Engine.java:232)

There is no security set up on the master, so I'm not sure what the unauthorized access is about, and there is no -auth username:password that I can add to that command line.

I'm not sure how to set up "Launch slave via execution of command on the Master" because I can't find much documentation for that. 

So I'm stuck. I was able to set up a windows slave so I know that it is possible. But the Mac slave does not want to start. Can you help?

Thanks,
Dan

Addenda: full stack trace from trying to launch slave agents on the master using ssh:

04/30/13 17:23:29] [SSH] Opening SSH connection to pelham:22.
[04/30/13 17:23:29] [SSH] Authentication successful.
[04/30/13 17:23:29] [SSH] The remote users environment is:
BASH=/bin/bash
BASH_ARGC=()
BASH_ARGV=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="3" [1]="2" [2]="48" [3]="1" [4]="release" [5]="x86_64-apple-darwin10.0")
BASH_VERSION='3.2.48(1)-release'
DIRSTACK=()
EUID=216
GROUPS=()
HOME=/Users/Shared/Jenkins
HOSTNAME=pelham.fhcrc.org
HOSTTYPE=x86_64
IFS=$' \t\n'
LOGNAME=jenkins
MACHTYPE=x86_64-apple-darwin10.0
MAIL=/var/mail/jenkins
OPTERR=1
OPTIND=1
OSTYPE=darwin10.0
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PPID=545
PS4='+ '
PWD=/Users/Shared/Jenkins
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='140.107.151.178 56267 22'
SSH_CONNECTION='140.107.151.178 56267 140.107.150.101 22'
TERM=dumb
UID=216
USER=jenkins
_=bash
[04/30/13 17:23:29] [SSH] Checking java version of java
[04/30/13 17:23:29] [SSH] java -version returned 1.6.0_45.
[04/30/13 17:23:29] [SSH] Starting sftp client.
[04/30/13 17:23:29] [SSH] Copying latest slave.jar...
[04/30/13 17:23:29] [SSH] Copied 284,251 bytes.
Expanded the channel window size to 4MB
[04/30/13 17:23:29] [SSH] Starting slave process: cd '/Users/Shared/Jenkins/master' && java  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave.jar version: 2.23
This is a Unix slave
Copied maven-agent.jar
Copied maven3-agent.jar
Copied maven3-interceptor.jar
Copied maven-interceptor.jar
Copied maven2.1-interceptor.jar
Copied plexus-classworld.jar
Copied classworlds.jar
Evacuated stdout
Tue Apr 30 17:23:31 pelham.fhcrc.org java[551] <Error>: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
java.io.IOException: Remote call on pelham failed
    at hudson.remoting.Channel.call(Channel.java:681)
    at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32)
    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:472)
    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:340)
    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:678)
    at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:472)
    at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.InternalError: Can't connect to window server - not enough permissions.
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1827)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1724)
    at java.lang.Runtime.loadLibrary0(Runtime.java:823)
    at java.lang.System.loadLibrary(System.java:1045)
    at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:38)
    at sun.awt.DebugHelper.<clinit>(DebugHelper.java:29)
    at java.awt.Component.<clinit>(Component.java:566)
    at org.jenkinsci.modules.slave_installer.impl.InstallerGui.call(InstallerGui.java:47)
    at org.jenkinsci.modules.slave_installer.impl.InstallerGui.call(InstallerGui.java:29)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:326)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    ... 5 more
[04/30/13 17:23:31] [SSH] Connection closed.
ERROR: Connection terminated
java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2557)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1294)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
ERROR: [04/30/13 17:23:31] slave agent was terminated
java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2557)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1294)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
    at hudson.remoting.Command.readFrom(Command.java:92)
    at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

[Scott Evans]

Dan,

I'm not a mac person so can't help you in detail, but can you, from your mac, browse to your Jenkins server with a web browser?  It almost sounds like your slave mac isn't giving the jenkins slave process enough permissions to reach out, or there is some sort of firewall blocking connections to the jenkins server on port 8080 perhaps.

Scott

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dan Tenenbaum]

On Tue, Apr 30, 2013 at 5:55 PM, Scott Evans <milw...@gmail.com> wrote:
> Dan,
>
> I'm not a mac person so can't help you in detail, but can you, from your
> mac, browse to your Jenkins server with a web browser?

Yes.

> It almost sounds
> like your slave mac isn't giving the jenkins slave process enough
> permissions to reach out, or there is some sort of firewall blocking
> connections to the jenkins server on port 8080 perhaps.
>

There is no firewall.


To me it seems like the slave can contact the master, but is having an
authorization problem once it does.

Dan

> You received this message because you are subscribed to a topic in the

> Google Groups "Jenkins Users" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/jenkinsci-users/wYLQESSDjag/unsubscribe?hl=en.
> To unsubscribe from this group and all its topics, send an email to

[Kenny Ayers]

You're experiencing at least two issues here.  First there's the OSX bug with your window manager permissions.  You need to run headless on OSX, as you already determined, so make sure you continue to do that. (The bug is reported here: https://issues.jenkins-ci.org/browse/JENKINS-15697).

Add this to the slave command line options: -Djava.awt.headless=true

Secondarily it seems like perhaps the slave node may not be properly configured within Jenkins?  Have you tried a combination of SSH + running headless (perhaps that was your error message you included)?

I hope this helps,

Kenny

[Dan Tenenbaum]

On Tue, Apr 30, 2013 at 6:49 PM, Kenny Ayers
<theother...@gmail.com> wrote:
> You're experiencing at least two issues here. First there's the OSX bug
> with your window manager permissions. You need to run headless on OSX, as
> you already determined, so make sure you continue to do that. (The bug is
> reported here: https://issues.jenkins-ci.org/browse/JENKINS-15697).
>
> Add this to the slave command line options: -Djava.awt.headless=true
>

This did the trick.
Thanks!
Dan

[Dan Tenenbaum]

On Wed, May 1, 2013 at 10:04 AM, Dan Tenenbaum <dand...@dandante.com> wrote:
> On Tue, Apr 30, 2013 at 6:49 PM, Kenny Ayers
> <theother...@gmail.com> wrote:
>> You're experiencing at least two issues here. First there's the OSX bug
>> with your window manager permissions. You need to run headless on OSX, as
>> you already determined, so make sure you continue to do that. (The bug is
>> reported here: https://issues.jenkins-ci.org/browse/JENKINS-15697).
>>
>> Add this to the slave command line options: -Djava.awt.headless=true
>>
>
> This did the trick.
> Thanks!
> Dan
>
>
>> Secondarily it seems like perhaps the slave node may not be properly
>> configured within Jenkins? Have you tried a combination of SSH + running
>> headless (perhaps that was your error message you included)?
>>

I still have a problem, because I ultimately I want to have a scenario
where the master is in the cloud (EC2) and the Mac slave is on my
internal network, behind a firewall.

So I won't be able to use ssh in this scenario to connect from the
master to the slave.
It sounds like I will have to start the slave agent on the slave.
But that isn't working, even with the headless option to the JVM, I am getting

$ java -Djava.awt.headless=true -jar slave.jar -jnlpUrl
http://dhcp151178:8080/computer/pitt/slave-agent.jnlp -secret <secret
hidden>
May 1, 2013 2:59:10 PM hudson.remoting.jnlp.Main$CuiListener error

SEVERE: The server rejected the connection: Unauthorized access
java.lang.Exception: The server rejected the connection: Unauthorized access
at hudson.remoting.Engine.onConnectionRejected(Engine.java:257)
at hudson.remoting.Engine.run(Engine.java:232)

This happens even if I provide the "-secret" suggested to me on the
slave's page on the master's Jenkins.


I get the same error if I try and start the JNLP from a browser, after
logging in to jenkins on the master as a user with admin rights.

I also downloaded the JNLP manually and tried this:
javaws -Jjava.awt.headless=true slave-agent.jnlp
...but I get:
...
Splash: Can't connect to window server.: No such file or directory
Wed May 1 15:04:49 pitt.fhcrc.org java[349] <Error>: kCGErrorFailure:

Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are
logged.

Exception in thread "main" java.lang.InternalError: Can't connect to
window server - not enough permissions.

...

I thought the java.awt.headless switch was supposed to take care of
this. I have added that switch to the master's config of this node.

I *do* now have security on the master, so is there a way to pass a
username/password when I try and invoke the JNLP from the command
line? I assume that's what the -secret option is supposed to do.

I am able to start the slave from the master using both "Launch slave
agents on Unix machines via SSH" and "Launch slave via execution of
command on the Master" (using the command ssh jenkins@nameofslave
"java -Djava.awt.headless=true -jar
/Users/Shared/Jenkins/Home/war/WEB-INF/slave.jar")

BUT, I cannot establish communication between the slave and master if
the master is outside my network firewall.

There is an ssh gateway and I'll investigate whether tunneling is a
possibility. But I'd like to be able to start the slave agent--that's
my real problem.

Also I'm wondering if there is a way to start the slave agent on the
slave without being logged in. Because what if the slave machine
reboots, there is no guarantee that someone will log into it and start
the appropriate Java Web Start. There is probably a way to automate
that but I'd prefer if the necessary processes ran as daemons so being
logged in was not necessary.

Thanks in advance,
Dan

[Kenny Ayers]

Hey Dan,

  I noticed that in part of your reply, you said you had an issue starting the slave, but it looks there is a syntax error here:

I also downloaded the JNLP manually and tried this: 

javaws -Jjava.awt.headless=true slave-agent.jnlp 

...but I get: 

... 

Splash: Can't connect to window server.: No such file or directory 

Wed May  1 15:04:49 pitt.fhcrc.org java[349] <Error>: kCGErrorFailure: 

Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are 

logged. 

Exception in thread "main" java.lang.InternalError: Can't connect to 

window server - not enough permissions. 

  Make sure you use -D, not -J.

  Setting up an SSH tunnel from the master to slave should be a workaround for the security error you're getting.  Any reason your master has to be outside of your network in the EC2 cloud (I'm guessing this is part of an enterprise support scheme)?

  There are a few ways to run a Jenkins slave on startup for OSX.  Here is a link describing how to setup the Jenkins service (not slave): http://jrenard.info/blog/a-quick-but-working-startup-item-for-hudson-for-mac-os-x.html  - you should be able to modify the contents of the plist file to run the slave service.

  Does the master node have any information in its log when OSX has the security crash?  Have you considered setting up a test instance of Jenkins and debugging in real-time?  I realize this is a bit more technical than most users want to get, but if you've got some understanding of Java it is fairly simple to do using NetBeans: https://groups.google.com/forum/?fromgroups=#!topic/jenkinsci-dev/9a4F7jN1JNI

Hope this helps some,

Kenny

[Dan Tenenbaum]

On Wed, May 1, 2013 at 6:48 PM, Kenny Ayers <theother...@gmail.com> wrote:
> Hey Dan,
>
> I noticed that in part of your reply, you said you had an issue starting
> the slave, but it looks there is a syntax error here:
>
> I also downloaded the JNLP manually and tried this:
>
> javaws -Jjava.awt.headless=true slave-agent.jnlp
>
> ...but I get:
>
> ...
>
> Splash: Can't connect to window server.: No such file or directory
>
> Wed May 1 15:04:49 pitt.fhcrc.org java[349] <Error>: kCGErrorFailure:
>
> Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are
>
> logged.
>
> Exception in thread "main" java.lang.InternalError: Can't connect to
>
> window server - not enough permissions.
>
>
> Make sure you use -D, not -J.
>

For javaws (as opposed to java) the command to pass an option to the
JVM is -J, not -D.

> Setting up an SSH tunnel from the master to slave should be a workaround
> for the security error you're getting. Any reason your master has to be
> outside of your network in the EC2 cloud (I'm guessing this is part of an
> enterprise support scheme)?
>

I would like all the build machines to be in EC2 but there are no
virtual Mac machines because of licensing issues, so in order to build
on the Mac I need to contact a machine on our internal network.

> There are a few ways to run a Jenkins slave on startup for OSX. Here is a
> link describing how to setup the Jenkins service (not slave):
> http://jrenard.info/blog/a-quick-but-working-startup-item-for-hudson-for-mac-os-x.html
> - you should be able to modify the contents of the plist file to run the
> slave service.
>

OK, so that ensures that the jenkins web app is always running, but
I'm not clear whether that means that the slave will always be running
(and in contact with the master).

> Does the master node have any information in its log when OSX has the
> security crash?

Any idea where I should look? I'm not sure where jenkins' logs are.
The master is running on OSX as well if that helps you answer the
question.

> Have you considered setting up a test instance of Jenkins
> and debugging in real-time? I realize this is a bit more technical than
> most users want to get, but if you've got some understanding of Java it is
> fairly simple to do using NetBeans:
> https://groups.google.com/forum/?fromgroups=#!topic/jenkinsci-dev/9a4F7jN1JNI
>

I'll try it if other avenues don't work out.

Thanks,
Dan

[Kenny Ayers]

Hey Dan,

  Oh, I didn't notice you were using javaws.  Thank you for the correction.

OK, so that ensures that the jenkins web app is always running, but 

I'm not clear whether that means that the slave will always be running 

(and in contact with the master). 

  Presumably if the application successfully starts up successfully, that's what you'll get.

  The Jenkins log file location can vary according to how you started the service.  Can you grep out the startup command from the list of running services?  There is a logfile option.  For example you should see something like:

$ ps aux | grep -i jenk

java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --daemon --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

  I looked over your JNLP startup command, and found that your OSX build agent seems to be trying to connect to localhost.  Are you running the master and slave on the same host?  If not, is there another instance of Jenkins master running on the build agent that you can kill (presumably this was accidentally setup and left running during testing)?  I've highlighted the line below where it shows that the JNLP agent is connecting to localhost:

java -jar slave.jar -jnlpUrl http://dhcp151178:8080/computer/pelham/slave-agent.jnlp
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://localhost:8080/, http://dhcp151178:8080/]
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to localhost:49161
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Apr 30, 2013 5:19:02 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: Unauthorized access
java.lang.Exception: The server rejected the connection: Unauthorized access
	at hudson.remoting.Engine.onConnectionRejected(Engine.java:257)
	at hudson.remoting.Engine.run(Engine.java:232)

  So that seems strange.

  A few more things to check: is the remote fs root specified correctly, and does it have the correct permissions?  Is the Jenkins URL specified correctly on the master node?

Maybe getting closer...

-Kenny

[Dan Tenenbaum]

On Wed, May 1, 2013 at 8:12 PM, Kenny Ayers <theother...@gmail.com> wrote:
> Hey Dan,
>

OK, one thing I guess I did not understand was that the Jenkins webapp
does not need to be running on the slave. In fact, shutting it down
solved my problem. I was then able to start the slave by doing

java -Djava.awt.headless=true -jar slave.jar -jnlpUrl

http://master:8080/computer/slave/slave-agent.jnlp -secret
<secret-key>

And in fact I could start this command with 'nohup' and pipe its
output to a file (and put & on the end of it) and I don't need to be
logged in on the slave for my builds to work. I'll set up this command
to be run automatically at boot time and then everything will 'just
work' if the machine is rebooted.

Thanks very much for your help.
Dan