Leverage Kerberos/Wallet for non-interactive SSH and script execution

[Andreas Ntaflos]

Hi,

I'd like to leverage our Kerberos (and Wallet) infrastructure to enable
non-interactive SSH/SCP between two servers for a given user. Is this
possible? Using MIT Kerberos 1.10 on Ubuntu 12.04 everywhere, currently
still with Wallet from prior to 1.0 (but after 0.12).

The scenario is this: We have a Jenkins build server (build01) and an
APT repo server (apt01, using Freight [1]). Jenkins does what it does
and in the end creates DEB packages. Those DEB packages should land on
the APT repo server and the APT repo should be updated with the new
packages. This works as expected using SSH public key authentication.

On the shell it looks like this:

jenkins@build01:~$ scp *.deb jenkins@apt01:/path/to/packages
jenkins@build01:~$ ssh jenkins@apt01 "/usr/local/bin/update-apt-repo"

After that the APT repo server has the new packages, signed and ready
for installation.

I have implemented a Wallet infrastructure according to Jan-Piet Mens's
excellent article [2] and distributed Keytabs for all servers (using
Puppet). I can interactively and without passwords log into any of those
servers after doing a "kinit" as my user.

So what can I do to avoid SSH public key authentication and instead use
Kerberos and possibly Wallet to implement the described scenario?

Thanks,

Andreas

[1] https://github.com/rcrowley/freight
[2]
http://jpmens.net/2012/06/25/streamlining-distribution-of-kerberos-keytabs-and-other-secure-data/

[Russ Allbery]

Andreas Ntaflos <da...@pseudoterminal.org> writes:

> The scenario is this: We have a Jenkins build server (build01) and an
> APT repo server (apt01, using Freight [1]). Jenkins does what it does
> and in the end creates DEB packages. Those DEB packages should land on
> the APT repo server and the APT repo should be updated with the new
> packages. This works as expected using SSH public key authentication.

> On the shell it looks like this:

> jenkins@build01:~$ scp *.deb jenkins@apt01:/path/to/packages
> jenkins@build01:~$ ssh jenkins@apt01 "/usr/local/bin/update-apt-repo"

> After that the APT repo server has the new packages, signed and ready
> for installation.

What we do for a similar case is that we put the two commands in a script,
and then create a keytab on the build server that will be used for
authentication to upload the commands. You can just use the
host/build01.example.com host key, but we usually create a special
principal for the Jenkins build service (we'd call it service/jenkins), or
if you want each build server to have a unique keytab (not a bad idea),
create a keytab like jenkins/build01.example.com.

Then, use wallet to create that keytab on the build server, and then have
your Jenkins server end its tasks by running:

k5start -qUf /path/to/keytab/file -- /path/to/upload/script

This will run the script with a Kerberos ticket cache. ssh will then be
able to use GSS-API authentication using that Kerberos ticket cache, and
once you add a .k5login file in the home directory of the jenkins user on
apt01 containing the principal you use on the build server, everything
should just work with Kerberos and no ssh public keys.

--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>

[Ken Dreyer]

On Wed, May 22, 2013 at 1:20 PM, Russ Allbery <r...@stanford.edu> wrote:
> Then, use wallet to create that keytab on the build server, and then have
> your Jenkins server end its tasks by running:
>
> k5start -qUf /path/to/keytab/file -- /path/to/upload/script

I recently set up something just like this to do Jenkins deploys out
of an SCM into AFS (instead of SSH or SCP). k5start works like a charm
and I'd highly recommend it.

Also, I'd second Russ's point about separate keytabs per build
"server". Out of the box, Jenkins doesn't do privilege separation well
at all. I worked around this by using separate Jenkins shell accounts
on the build servers, one account per project, with separate keytabs
for each shell account/project. They are all prefixed by "jenkins/",
so the keytab that can deploy to an Apache virtualhost in AFS is named
"jenkins/vhost.example.com". It's a pain to manage all these extra
pieces at scale, although Puppet helps a bit.

- Ken

[Andreas Ntaflos]

On 2013-05-22 21:20, Russ Allbery wrote:
> Andreas Ntaflos <da...@pseudoterminal.org> writes:
>
>> The scenario is this: We have a Jenkins build server (build01) and an
>> APT repo server (apt01, using Freight [1]). Jenkins does what it does
>> and in the end creates DEB packages. Those DEB packages should land on
>> the APT repo server and the APT repo should be updated with the new
>> packages. This works as expected using SSH public key authentication.
>
>> On the shell it looks like this:
>
>> jenkins@build01:~$ scp *.deb jenkins@apt01:/path/to/packages
>> jenkins@build01:~$ ssh jenkins@apt01 "/usr/local/bin/update-apt-repo"
>
>> After that the APT repo server has the new packages, signed and ready
>> for installation.
>
> What we do for a similar case is that we put the two commands in a script,
> and then create a keytab on the build server that will be used for
> authentication to upload the commands. You can just use the
> host/build01.example.com host key, but we usually create a special
> principal for the Jenkins build service (we'd call it service/jenkins), or
> if you want each build server to have a unique keytab (not a bad idea),
> create a keytab like jenkins/build01.example.com.
>

> Then, use wallet to create that keytab on the build server, and then have
> your Jenkins server end its tasks by running:
>
> k5start -qUf /path/to/keytab/file -- /path/to/upload/script
>

> This will run the script with a Kerberos ticket cache. ssh will then be
> able to use GSS-API authentication using that Kerberos ticket cache, and
> once you add a .k5login file in the home directory of the jenkins user on
> apt01 containing the principal you use on the build server, everything
> should just work with Kerberos and no ssh public keys.

Russ, thanks very much for this idea, I implemented it almost exactly
like that and it works flawlessly. It also helps to have Puppet modules
and types for Wallet and k5login.

I'll be using this approach for a few other aspects in our
infrastructure as well.

Thanks again,

Andreas

[Andreas Ntaflos]

On 2013-05-22 21:37, Ken Dreyer wrote:
> On Wed, May 22, 2013 at 1:20 PM, Russ Allbery <r...@stanford.edu> wrote:

>> Then, use wallet to create that keytab on the build server, and then have
>> your Jenkins server end its tasks by running:
>>
>> k5start -qUf /path/to/keytab/file -- /path/to/upload/script
>

> I recently set up something just like this to do Jenkins deploys out
> of an SCM into AFS (instead of SSH or SCP). k5start works like a charm
> and I'd highly recommend it.
>
> Also, I'd second Russ's point about separate keytabs per build
> "server". Out of the box, Jenkins doesn't do privilege separation well
> at all. I worked around this by using separate Jenkins shell accounts
> on the build servers, one account per project, with separate keytabs
> for each shell account/project. They are all prefixed by "jenkins/",
> so the keytab that can deploy to an Apache virtualhost in AFS is named
> "jenkins/vhost.example.com". It's a pain to manage all these extra
> pieces at scale, although Puppet helps a bit.

Ken, thank you, too, for your input! k5start indeed works great. Our
build server infrastructure is not as complex as yours seems to be but
I've taken your approach and have given each jenkins system account on
each of our four build servers its own keytab and set up k5login on the
APT repo server (with Puppet modules and types for Wallet and k5login
this is easy enough). On the APT repo server the jenkins user can call a
single script via sudo to deploy new .deb packages. This works exactly
as I had hoped it would.

Thanks again,

Andreas

[Richard E. Silverman]

In case it's of interest: we take a different approach. We've factored
all of this out into PAM. All automation accounts have corresponding
principals whose keys are in the system keytab. We wrote a PAM module
which creates a per-session ccache, sets KRB5CCNAME, kinits the matching
principal, and removes the ccache when the PAM session closes. We apply
this module via PAM configuration to services like cron and sudo, and
then cron jobs and sudo sessions automatically have credentials. For
sudo, it has the option to copy the applicant's ccache, so that you have
your own Kerberos identity while using another uid (if that's more
useful in a given case).

--
- Richard