can't start slaves anymore, ssh auth failing

[Haszlakiewicz, Eric]

I recently upgraded my Jenkins installation and now none of my slaves will start. I changed nothing in my environment other than upgrading Jenkins. I am still able to ssh successfully from the command line as the user that is running Jenkins to all of the remote slaves.

Here's the error I get:

[05/02/13 10:21:55] [SSH] Opening SSH connection to myhost.example.com:22.
ERROR: Server rejected the private key for dev1appl (credentialId:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/method:publickey)
[05/02/13 10:21:55] [SSH] Authentication failed.
hudson.AbortException: Authentication failed.
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:923)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:479)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
[05/02/13 10:21:55] [SSH] Connection closed.

Does anyone have any suggestions about how to fix this?

Thanks,
Eric

[CHAVANNE Robert]

Hello Eric,
We had the same kind of problem.
I think you should take a look at "manage Credentials" in manage Jenkins.
We had to create credential to fix the issue.
I hope it can help.

Regards,
Robert

[Stephen Connolly]

Ensure you are on ssh-slaves 0.25 or newer as 0.23 and 0.24 can forget the credentials any time you reboot due to a race condition

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[jcsirot]

You should also take a look at the tool locations on your slaves. My configurations were messed up with, for instance, maven configured with the JDK path and so on.

[Haszlakiewicz, Eric]

eh? I now need to manage my credentials within Jenkins, instead of just the contents of my .ssh directory? Well that's a pretty huge change, and I didn't see any mention of it in the main changelog. :(
I guess I'll take a look and see what I need to configure there. Thanks for the pointer.

Eric

[Haszlakiewicz, Eric]

> -----Original Message-----
> From: jenkins...@googlegroups.com [mailto:jenkinsci-
>

> Ensure you are on ssh-slaves 0.25 or newer as 0.23 and 0.24 can forget the
> credentials any time you reboot due to a race condition

I actually rolled back to 0.22 to get things working, but I'll try upgrading again when I get a chance.

Eric

[Kevin Fleming (BLOOMBERG/ 731 LEXIN)]

This change would not have been in the Jenkins changelog, because it wasn't technically a Jenkins change. It's a change in the SSH Slaves plugin.

[Les Mikesell]

On Thu, May 2, 2013 at 1:26 PM, Kevin Fleming (BLOOMBERG/ 731 LEXIN)
<kpfl...@bloomberg.net> wrote:
> This change would not have been in the Jenkins changelog, because it wasn't
> technically a Jenkins change. It's a change in the SSH Slaves plugin.

Shouldn't something note when the things distributed with the core
package change in ways that affect behavior? Or are all users
supposed to know the internal workflow across all the plugins?

--
Les Mikesell
lesmi...@gmail.com

[JonathanRRogers]

I'm glad I came across this thread while I'm still on version 0.22. I'll make sure not to upgrade to a version which breaks things so fundamentally.

[Stephen Connolly]

On Thursday, 2 May 2013, Haszlakiewicz, Eric wrote:

eh?  I now need to manage my credentials within Jenkins, instead of just the contents of my .ssh directory?  

If you have all they keys Jenkins needs in your .ssh directory then this makes your life easier... You just tell it to use those credentials.

If you don't then this makes your life easier.

It is a great pity that nobody tested 0.23 (or the -SNAPSHOT with the credentials change available since jan.)

With greater testing the race condition that can cause migrated credentials to get lost would have been caught and fixed.

As soon as I spotted the issue, I fixed it and cut the 0.25 release.

Well that's a pretty huge change, and I didn't see any mention of it in the main changelog. :( 

I guess I'll take a look and see what I need to configure there. 

Shouldn't have to configure anything if you go 0.22->0.25

It just makes it easier to share credential config across build slaves and also use those credentials for other ssh related build steps

 

 Thanks for the pointer.

Eric

> -----Original Message-----
> From: jenkins...@googlegroups.com [mailto:jenkinsci-
> us...@googlegroups.com] On Behalf Of CHAVANNE Robert
> Sent: Thursday, May 02, 2013 10:43 AM
> To: jenkins...@googlegroups.com
> Subject: Re: can't start slaves anymore, ssh auth failing
>
> Hello Eric,
> We had the same kind of problem.
> I think you should take a look at "manage Credentials" in manage Jenkins.
> We had to create credential to fix the issue.
> I hope it can help.
>
> Regards,
> Robert
>

--
Sent from my phone

[Stephen Connolly]

0.23 and 0.24 have a race condition where there is a 10-15% chance that you will loose your credentials after rebooting the first time post installation.

0.25 should be fine.

This is mostly a UI change, thought the consolidation makes life easier when you need to change credentials in bulk for slaves.

A similar change will be coming down the line for, eg subversion, git, etc (the blocker for now being getting a consistent credential type for the many auth mechanisms for the scm systems) 

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jonathan Rogers]

Stephen Connolly wrote:
>
>
> On Thursday, 2 May 2013, JonathanRRogers wrote:
>
> On Thursday, May 2, 2013 2:50:44 PM UTC-4, LesMikesell wrote:
>
> On Thu, May 2, 2013 at 1:26 PM, Kevin Fleming (BLOOMBERG/ 731
> LEXIN)
> <kpfl...@bloomberg.net> wrote:
> > This change would not have been in the Jenkins changelog,
> because it wasn't
> > technically a Jenkins change. It's a change in the SSH
> Slaves plugin.
>
> Shouldn't something note when the things distributed with the
> core
> package change in ways that affect behavior? Or are all users
> supposed to know the internal workflow across all the plugins?
>
>
> I'm glad I came across this thread while I'm still on version
> 0.22. I'll make sure not to upgrade to a version which breaks
> things so fundamentally.
>
>
> 0.23 and 0.24 have a race condition where there is a 10-15% chance
> that you will loose your credentials after rebooting the first time
> post installation.
>
> 0.25 should be fine.
>

Are you saying that upgrading from 0.22 to 0.25 is transparent?
-- Jonathan Rogers

[Stephen Connolly]

Should be... The plugin will pickup your credentials from your slave configuration and add them to the credentials store, consolidating where that makes sense. IOW the upgrade should result in everything still working as before

If it doesn't then that's a bug, but nobody has alerted me to issues with that upgrade path.

 

-- Jonathan Rogers

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Les Mikesell]

On Fri, May 3, 2013 at 2:15 AM, Stephen Connolly
<stephen.al...@gmail.com> wrote:
>
>> Are you saying that upgrading from 0.22 to 0.25 is transparent?
>
>
> Should be... The plugin will pickup your credentials from your slave
> configuration and add them to the credentials store, consolidating where
> that makes sense. IOW the upgrade should result in everything still working
> as before
>
> If it doesn't then that's a bug, but nobody has alerted me to issues with
> that upgrade path.

Can you describe what is "supposed" to happen in this upgrade? I
just did an rpm update from a 1.5.00-1.1 (non LTS) to LTS 1.509.1-1
and none of the ssh-started slaves are working. The first one I've
looked at has a configuration under 'Launch slave agents on Unix
machines via SSH' that says Host: 1722.22.181.77, Credentials:
jenkins (172.22.181.78) so I suspect something went wrong there....

This is a non-critical test instance, but I'd like to understand what
is supposed to happen before breaking a production system. I might
have had a mix of systems using the ~jenkins/.ssh/ key and explicitly
entered passwords before in case that matters, but none of them work
now

--
Les Mikesell
lesmi...@gmail.com

[Haszlakiewicz, Eric]

> -----Original Message-----
> From: jenkins...@googlegroups.com [mailto:jenkinsci-

> On Friday, 3 May 2013, Jonathan Rogers wrote:
> Stephen Connolly wrote:
> > On Thursday, 2 May 2013, JonathanRRogers wrote:
> > 0.23 and 0.24 have a race condition where there is a 10-15% chance
> > that you will loose your credentials after rebooting the first time
> > post installation.
> >
> > 0.25 should be fine.
>
> Are you saying that upgrading from 0.22 to 0.25 is transparent?
>
> Should be... The plugin will pickup your credentials from your slave
> configuration and add them to the credentials store, consolidating where
> that makes sense. IOW the upgrade should result in everything still working
> as before
>
> If it doesn't then that's a bug, but nobody has alerted me to issues with that
> upgrade path.

I seem to have ended up with 9 entries on the Manage Credentials page, most with "From a file on Jenkins master" selected, but with no way to indicate the file to use.
Is this what happens when that race condition is hit?
I guess I'll try upgrading again, then removing and re-adding some credentials.

Eric

[Les Mikesell]

On Fri, May 3, 2013 at 2:19 PM, Les Mikesell <lesmi...@gmail.com> wrote:

>> If it doesn't then that's a bug, but nobody has alerted me to issues with
>> that upgrade path.
>
> Can you describe what is "supposed" to happen in this upgrade? I
> just did an rpm update from a 1.5.00-1.1 (non LTS) to LTS 1.509.1-1
> and none of the ssh-started slaves are working.

Part of this may have involved the peculiar upgrade path here. The
initial error (which I can't reproduce now) had to do with some method
not being found. Updating the Credentials and SSH Credentials plugins
fixed that error, but only one slave started with the auto-converted
credentials. I had to re-enter them for the others. Some of the
production slaves are maintained by others so I don't have the
passwords or keys. Is there something I can do to fix things up from
the existing file contents if the production upgrade has similar
problems?

--
Les Mikesell
lesmi...@gmail.com

[Stephen Connolly]

Hmmm, not sure as once it has tried to transition to the new config it saves the new config so that it can be applied. I'll have a look through the code again. Most likely it will be Tuesday before I can take a look though

--
   Les Mikesell
     lesmi...@gmail.com