Gitolite support of SSHD Certificate Authority Keys (specifically the AuthorizedPrincipals file)

[James Shannon]

Hi,

I've used gitolite in the past, really liked it, and wanted to install it again on a new server. However, I just set the server up using SSH cert authority key signing model.

Roughly, that means that authorized_keys isn't used and sshd instead it looks to the principals key of a signed public key. I understand that gitolite relies heavily on the authorized_keys file so I'm trying to figure out if it has any support for CA via the AuthorizedPrincipalsFile.

This looks like it'd be a pretty straightforward implementation. In default mode, sshd looks at the principals key in the signed public key and makes sure that that matches the user you're trying to authenticate as. So I could have a public key with principals=james,git,root, and sshd would allow me to login as any of those. I think there's even a command key that could be added by the CA and force a particular command. However, that's really messy.

sshd also has a AuthorizedPrincipalsFile [1] which is typically set up one-per-user (so authorized/root, authorized/git, authorized/james) and lists principals. So rather than my public key having principals=jshannon,root,git it would be something like principals=fleetwide-jshannon,server-specific-root,git-repo-access. So more like groups than users. Then the root principals file would specify that anybody with fleetwide-root and thisserver-root can access. The docs say that:

> Names are listed one per line preceded by key options (as described in AUTHORIZED_KEYS FILE FORMAT in sshd(8)).

So it sounds like the command= key which gitolite relies on in authorized_users would have the same functionality. So it might be as easy as updating the principals file rather than the authorized_keys file and simply writing the principal name instead of the key? Has anybody done this?



[1] https://man.openbsd.org/sshd_config#AuthorizedPrincipalsFile

[Sitaram Chamarty]

OK this is a long mail. And it's best read in a mono-space
font, if your mail client can cut it.

Here's what I found. (This is all new to me so please forgive
any details you find to be too trivial/basic! I am writing this
as much for myself as anyone else, really!)

Background
==========

Lots more detail is available in various parts of the internet
(refs [3] and [4] are courtesy James, though I admit I have not
yet read [4]).

[3]: https://code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/
[4]: https://framkant.org/2017/07/scalable-access-control-using-openssh-certificates/

But basically, I would describe this as:

- setup a CA to sign keys
- sign your users' pubkeys with this CA's private key
- tell sshd to accept such signed keys (instead of, or in
addition to, the current mechanism of accepting keys found
in `~/.ssh/authorized_keys`).

In among all this is the concept of a "principal". For the
moment, think of the principal as a "role" of some kind.

That is, a user may have several roles (specified when the CA
signs her key), and a server may allow several roles (see below)
to log in to it using a cert. As long as these two role-lists
have at least one role (i.e., "principal") in common, you're in.
That's the main advantage of using certs in ssh, as far as I
understand.

(Server principals come from either sshd config's
"AuthorizedPrincipalsFile" or the "principals=" option in the
users `~/.ssh/authorized_keys` file)

Caveats in what I tried
=======================

1. I'm only looking at the user-certification part of this.
Hosts can also be certified, but that is orthogonal to
gitolite so I didn't bother testing that. (This is the part
that adds a new line to ~/.ssh/known_hosts the first time
you ssh to a host; using host certs can make that a bit more
secure).

2. I did **not** test revocation lists. Any production use of
this MUST deal with that. This includes adding a serial
number to each signed cert, and various settings in ssd
config (and maybe more; I don't know). Also see SUMMARY
section at the end for more on this.

3. Don't try these experiments remotely; I have not checked if
some intermediate step will lose ssh access for you! I did
all this on my laptop so I was safe from getting locked out
inadvertently.

(That said, I don't think you will lose access; the existing
authorized keys functionality seemed to work fine all
along.)

How I tested
============

I basically took my standard test bed, which contains 6
"gitolite" users (u1 thru u6), all running on the same Unix
userid ("g3"). (If that's confusing, think of "u1", "u2", etc.,
as "alice", bob", and "g3" as "git").

This is all done using the IdentityFile directive in
`~/.ssh/config` to select different users, so I can simulate
different gitolite users while sitting in the same Unix user,
which is also the gitolite hosting user!

I then removed the first 2 pubkey files (u1.pub and u2.pub) from
keydir and pushed. (As a result, they also disappear from
`~/.ssh/authorized_keys`).

At this point "ssh u1 info" fails, as does "ssh u2 info". The
others work as usual; and they continued to work all through
this testing. That is, whatever I do to the certificate stuff,
the contents of `~/.ssh/authorized_keys` was always available as
a fallback.

Method 1: no root access
========================

Despite how many people probably use gitolite via RPM/DEB, it
was always designed so that you don't need root to run it
(modulo installing git, perl, and openssh-server).

As such, I was *most* interested in this mode. Here's how I
tested this:

(All this is in `~g3/.ssh`):

1. Generate a CA key:

ssh-keygen -f ca -C "CA for testing gitolite"

Note that normally the CA (or at least its private key) is
on some completely different machine.

2. edit the `authorized_keys` file and copy the ca pubkey
(ca.pub from step 1) to it. Put it right at the start of
the file (*before* the `# gitolite start` line).

Then edit that line to prefix

cert-authority,principals="g3"

with a trailing space. So the whole line will look
something like:

cert-authority,principals="g3" ssh-rsa AAAAB3Nza[...]

3. Sign **each user's** key ("u1" in this example):

ssh-keygen -s ca \
-n g3 \ # this is the principal
-I g3-u1 \ # this is the key ID; note the "u1"
-O no-user-rc \ # this and the next 5 are restrictions
-O no-port-forwarding \
-O no-X11-forwarding \
-O no-agent-forwarding \
-O no-pty \
-O 'force-command=/home/g3/gitolite/src/gitolite-shell u1' \
# note the "u1" as the argument to gitolite-shell
u1.pub

Notice the "gitolite-shell" command and other restrictions are
hardcoded in the certificate, which means certs signed in this
mode can't be used to access any other (normal) server.

As a result, this method quickly looked like a "why bother?" :)

(On top of that, it seems revocation is not doable without
getting into root somewhere... have not played with it but

Method 2: AuthorizedPrincipalsFile
==================================

This method requires root, and fiddles with
`/etc/ssh/sshd_config`. You may want to use this if your CA is
certifiying not just gitolite user keys but keys for accessing
other systems also (i.e., you don't want the certificate to have
the "gitolite-shell" hardcoded in it).

(All this in /etc/ssh):

1. Generate a CA key (same as earlier, except now it is placed
in /etc/ssh).

2. Edit `/etc/ssh/sshd_config` and add these lines somewhere:

TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u

Then restart sshd.

3. Create the authorized principals file and add a symlink to
it:

touch /home/g3/.ssh/principals
chown g3:g3 /home/g3/.ssh/principals
ln -sf /home/g3/.ssh/principals /etc/ssh/principals/g3

We do it this way so that gitolite can directly update the
principals list; otherwise you'd have to do it via root.

4. Sign **each user's** key ("u1" in this example):

ssh-keygen -s ca -n g3-u1 -I g3-u1 u1.pub

and give the resulting *-cert.pub files to the respective
users; they have to put it in their `~/.ssh` along side the
pubkey and privkey. They may have to rename it to
`id_rsa-cert.pub` or whatever, too.

Note that the principal is now "g3-u1", not "g3".
Similarly, user "u2" will have a principal of "g3-u2", etc.

Note also there are no restrictions. You probably want to
add other principals to the "-n" to give access to other
servers, so adding restrictions may be a problem.

5. Add **each user's** principal to the authorized principals
file:

(this is as g3, in ~g3/.ssh)

For each user, a line like this ("u1" shown here) is
appended to `~g3/.ssh/principals`:

no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,command="/home/g3/gitolite/src/gitolite-shell u1" g3-u1

As you can guess, the "g3-u1" is the principal (matching the
"-n" in step 4). The argument to gitolite-shell ("u1" here)
is the gitolite username.

You'll have one such line for each gitolite user (say, u2
and g3-u2, bob and g3-bob, etc.), to match each of the
different principals.

The disadvantage here is that, apart from signing the pubkey,
you also have to update the principals file.

Method 3: AuthorizedPrincipalsCommand
=====================================

This method also requires root, and fiddles with
`/etc/ssh/sshd_config`. The gitolite-shell is not hardcoded
into the certs, but adding a new key to gitolite is just a
matter of signing.

(All this in /etc/ssh):

1. Generate a CA key (same as earlier).

2. Edit `/etc/ssh/sshd_config` and add these lines at the top:

TrustedUserCAKeys /etc/ssh/ca.pub
AuthorizedPrincipalsCommand /root/apc %u %i
AuthorizedPrincipalsCommandUser root

(I was too tired to do better here, but in production you
should not use root for this).

The script `/root/apc` is this:

#!/bin/bash

NO="no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc"

# args: %u %i
# also, the way we sign it, all IDs start with "g3-"
if [ $1 == g3 ]
then
gitolite_user=${2#g3-}
echo "$NO,command=\"/home/g3/gitolite/src/gitolite-shell $gitolite_user\" g3"
fi

exit 0

3. Sign **each user's** key ("u1" here):

ssh-keygen -s ca -n g3 -I g3-u1 u1.pub

Notice we're back to using "g3" as the principal, instead of
"g3-u1", "g3-u2", etc.

SUMMARY
=======

As you can see, the biggest issue comes from the need to have
different arguments (to gitolite-shell) for different users.

Method 1 puts it in the certificate directly, making the cert
unusable for other servers and users. Method 2 puts it in the
principals file, so you can have a generic certificate that also
has other principals, but you have to maintain the principals
file. Method 3 is the best of both worlds.

(Best if signing is scripted so the IDs etc are consistent, of
course!)

The other thing to note is revocation. In all cases, you need
to take care of revocation. Unlike X509-style PKI, there is no
OCSP, and no "CRL ditribution point" URL that the client will
query. It's not clear to me how the CRL (called KRL here) is to
be updated as fast as possible after a revocation. If someone
knows, please post.

In general, revocation doesn't seem as well thought out. For
example, it's not clear how revocation is handled if you have
multiple CAs (because the RevokedKeys setting does not take
tokens, so there's exactly one file for the entire system!

What can we do within gitolite?
===============================

Sadly, not much. And happily, that's alright; not much is
*needed*.

1. I will not be adding any code that signs keys for anything
that affects any other service or user on the system. So,
if you want signing also to be done by gitolite, method 1 is
the only one that will work. However, revocation looks like
it will be a pain, and may need root!

More than that, though, if you're using a CA and giving out
certs that work(s) only with gitolite, why use a CA at all?

2. For method 2, as long as the signing is done by someone
else, gitolite could update the principals file, akin to how
it currently keeps the `authorized_keys` file updated.

3. For method 3, I don't see anything to be done by gitolite.
It's actually the most generic and maintenance-free of the
lot. (Notice how you can add more code to that script to
handle other odd-ball users too!)

----------------------------------------------------------------------

Phew! Hope that was useful!

regards
sitaram

> --
> You received this message because you are subscribed to the Google Groups "gitolite" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to gitolite+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[James SHANNON]

And I was concerned my message was too long.... :)

I didn't realize that sshd falls back to authorized_keys. The sshd_config doc mentions that if there's no AuthorizedPrincipalsFile then it falls back to principal == username, but didn't mention additional fallback.

I (also?) don't see a ton of value to implementing a CA for gitolite purposes; I'm using the CA approach for other reasons and want to also run gitolite. Because of my wrong assumption that authorized_keys wouldn't work I thought I was out of luck. However, now that I know that authorized_keys does work, I'd still prefer that gitolite supports the CA AuthorizedPrincipalsFile rather than needing to think through two different authentication mechanisms.

I think Approach #2 makes the most sense for gitolite integration. It's already managing the local authorized_keys file, and being able to manage a symlinked AuthorizedPrincipalsFile seems pretty close. The biggest change, I think, would be to signify to gitolite that you're using principals instead of keys so that it doesn't, e.g., rely on the keys in keydir. Maybe that's a gitolite-wide config variable?

I did try to upload a .pub file with a principal string instead of a key, so that gitolite would write it out into what it thinks is the authorized_keys file but is actually AuthorizedPrincipalsFile. However, there's a check in Common that tries to parse the key and throws an error. I found that gitolite does something different if the file doesn't end in .pub. I thought that might be the right approach, but it didn't seem to write out to authrorized_keys at all. It'd take me a while to wrap my head around what's going on, but it seems straightforward enough.

I had considered AuthorizedPrincipalsCommand but it's all or nothing -- you can't use it for gitolite but not for other users. I don't see a huge benefit to gitolite producing a Command binary since it's already dealing with file-writing anyways. However... it did occur to me that I could create a script that -- for the git user -- would read the authorized_keys file and rewrite it to a AuthorizedPrincipalsFile format while -- for all other users -- it simply reads out the principals file I would be using otherwise.

James

> To unsubscribe from this group and stop receiving emails from it, send an email to gitolite+unsubscribe@googlegroups.com.

[Sitaram Chamarty]

On Mon, May 14, 2018 at 08:17:41AM -0700, James SHANNON wrote:

> I think Approach #2 makes the most sense for gitolite integration. It's
> already managing the local authorized_keys file, and being able to manage a
> symlinked AuthorizedPrincipalsFile seems pretty close. The biggest change,
> I think, would be to signify to gitolite that you're using principals
> instead of keys so that it doesn't, e.g., rely on the keys in keydir. Maybe

Why either-or? You can have both.

Let me think about how best to do it, but at the moment I am
leaning toward a simple directory called "principals" in the
gitolite-admin repo that contains just empty files with the name
of each principal. That would then translate directly into a
simple principals file, within ~/.ssh.

I am pretty sure this will work just fine, and there's no need
to disable the keydir handling if you're using principals.

However, this will be in contrib, with instructions for enabling
it, rather than enabled by default. To be honest, until I see
some serious improvements in revocation handling, I'm inclined
to be a little wary of this whole thing.

> I had considered AuthorizedPrincipalsCommand but it's all or nothing -- you
> can't use it for gitolite but not for other users. I don't see a huge

I just tested with one key served up by AuthorizedPrincipalsFile
and one by AuthorizedPrincipalsCommand. Works fine. I don't
see any reason it can't use Command for one user and File for
others, when it can happily do both within the same user.

Haven't tried it though.

rgds
sitaram

[James SHANNON]

On Mon, May 14, 2018 at 8:44 AM, Sitaram Chamarty <sita...@gmail.com> wrote:

On Mon, May 14, 2018 at 08:17:41AM -0700, James SHANNON wrote:

> I think Approach #2 makes the most sense for gitolite integration. It's
> already managing the local authorized_keys file, and being able to manage a
> symlinked AuthorizedPrincipalsFile seems pretty close. The biggest change,
> I think, would be to signify to gitolite that you're using principals
> instead of keys so that it doesn't, e.g., rely on the keys in keydir. Maybe

Why either-or?  You can have both.

True. I think that it could get messy to manage two types of keys.

However, I can see the benefits where you have your standard CA-signed keys that are deployed to, e.g., your employees and other devices, while third parties use the more traditional key infrastructure.

One benefit of being either-or is that you wouldn't need to differentiate principals from "users" (pub keys) within gitolite. 

 

Let me think about how best to do it, but at the moment I am
leaning toward a simple directory called "principals" in the
gitolite-admin repo that contains just empty files with the name
of each principal.  That would then translate directly into a
simple principals file, within ~/.ssh.

Why botheer with these empty files? That seems somewhat vestigial to how gitolite works today and I don't see the necessity?

Why not something like p:principal_name in the ACL? So:

RW+ = bob,alice,p:readers

 (Even if you use files in a principals directory then you still need to somehow differentiate between principals and pub keys because there could be duplicates.)

 

I am pretty sure this will work just fine, and there's no need
to disable the keydir handling if you're using principals.

However, this will be in contrib, with instructions for enabling
it, rather than enabled by default.  To be honest, until I see
some serious improvements in revocation handling, I'm inclined
to be a little wary of this whole thing.

Yeah. I did notice this as a weakness, and I'm probably not as concerned about it as I should be, but I have a pretty simple use case.

If I were deploying this within an enterprise I'd probably use short-lived (5 minute) keys that are issued for a specific reason. That's basically what my day-job company does. 

> I had considered AuthorizedPrincipalsCommand but it's all or nothing -- you
> can't use it for gitolite but not for other users. I don't see a huge

I just tested with one key served up by AuthorizedPrincipalsFile
and one by AuthorizedPrincipalsCommand.  Works fine.  I don't
see any reason it can't use Command for one user and File for
others, when it can happily do both within the same user.

Huh. I could have sworn that I read that if AUthorizedPrincipalsCommand was specified then that would be read instead of the File. But on re-reading that doesn't seem to be the case.

Seems to be lots of UNIONs here -- combine Command output + File and look for at least one row where at least one of the principals listed in the key. Fall back to authorized_keys.

[Sitaram Chamarty]

On Mon, May 14, 2018 at 10:08:42AM -0700, James SHANNON wrote:
> On Mon, May 14, 2018 at 8:44 AM, Sitaram Chamarty <sita...@gmail.com>
> wrote:
>
> > On Mon, May 14, 2018 at 08:17:41AM -0700, James SHANNON wrote:
> >
> > > I think Approach #2 makes the most sense for gitolite integration. It's
> > > already managing the local authorized_keys file, and being able to
> > manage a
> > > symlinked AuthorizedPrincipalsFile seems pretty close. The biggest

Gitolite "already managing" the authkeys file should not be used
as a precedent :-) As [1] says, that's only being done as a
convenience for the most common case. Gitolite can't/won't do
anything for authentication via LDAP-based keys (I don't even
know how to do that, honestly; never played with LDAP), or via
HTTP auth. I'd regard certificate-based keys as one more such
authentication mechanism.

[1]: http://gitolite.com/gitolite/ssh/#but-but-you-have-all-that-ssh-stuff-in-gitolite

And to be perfectly frank, I favour #3
(AuthorizedPrincipalsCommand). A lot fewer steps, so a lot less
to go wrong. And even the signing step that is common to both,
is itself simpler here -- all users get the same principal.

> > change,
> > > I think, would be to signify to gitolite that you're using principals
> > > instead of keys so that it doesn't, e.g., rely on the keys in keydir.
> > Maybe
> >
> > Why either-or? You can have both.
> >
>
> True. I think that it could get messy to manage two types of keys.

What I meant was if you don't want to rely on the keys in keydir
then you simply don't put any keys there. Gitolite can't
process what's not there :)

> Why not something like p:principal_name in the ACL? So:
>
> RW+ = bob,alice,p:readers

Gitolite's main job is authorisation. Authentication has always
been, and will always be, a "side" thing where we'll do what's
possible for convenience, but it's not really gitolite's main
job. I think the link [1] I gave above says this too.

This sounds like an authentication feature impacting the core
authorisation code (i.e., the ACLs). Sorry but that won't be
happening.

rgds
sitaram

[Sitaram Chamarty]

On 05/14/2018 10:38 PM, James SHANNON wrote:

> > I had considered AuthorizedPrincipalsCommand but it's all or nothing -- you
> > can't use it for gitolite but not for other users. I don't see a huge
>
> I just tested with one key served up by AuthorizedPrincipalsFile
> and one by AuthorizedPrincipalsCommand.  Works fine.  I don't
> see any reason it can't use Command for one user and File for
> others, when it can happily do both within the same user.
>
>
> Huh. I could have sworn that I read that if AUthorizedPrincipalsCommand

> was specified then that would be read *instead of* the File. But on

> re-reading that doesn't seem to be the case.
>
> Seems to be lots of UNIONs here -- combine Command output + File and
> look for at least one row where at least one of the principals listed in
> the key. Fall back to authorized_keys.

I just realised that AuthorizedPrincipals{File,Command} are both usable
within Match. So, even if the above were not true, you could just do:

Match User git
AuthorizedPrincipalsCommand /home/git/apc %u %i
AuthotizedPrincipalsCommandUser git

And it should just work without impacting any other user.

Might actually be the best way to segregate different requirements of
different ssh-based services on a box, I suppose.

[James SHANNON]

Thanks a ton. You've given me a lot of ideas, both for gitolite and ssh.

I realized I could turn off the ssh-authkeys contrib trigger; gitolite is happily accepting updates without a key file.

Then I set up a simple APC:

NO="no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc"

echo "$NO,command=\"/home/git/gitolite/src/gitolite-shell ${1%%-*}\" git-users"

Any signed key with a git-users principal in their key gets authenticated. The username passed to gitolite is the key ID up to the first dash (or entire key ID if there's no dash). This works quite well with my existing ID structure of [user]-[machine]. 

 

[Sitaram Chamarty]

Good to hear!

--
sitaram

[Radmilo Milojković]

Hello,

I’m trying to setup forced command only for root user with authorized_principals file in its home directory, because I’m using ssh user and host CA in my environment. My attempt results with an error “Certificate does not contain an authorized principal”, which is not the case as principal name is the same in authorized_principals file and ssh user certificate - us...@domain.com. Principle name can be anything you like and it’s only important to match in both ssh user certificate and authorized_principals/authorized_keys, depending what you use. I have also tried to change the principal to “user” to avoid using “@” and it still throws the same error “Certificate does not contain an authorized principal”. In my case authorized_keys file should not exist in user home directory, because authorized_principals file should be sufficient.

It’s interesting that the same setup works if authorized_keys file is used instead and it has similar syntax as authorized_principals file. Maybe I’m missing something because it looks like syntax issue…

authorized_principals file in home directory of root user:

command=”/sbin/ifconfig”,no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding us...@domain.com


authorized_keys file in home directory of root user:

cert-authority,command=”/sbin/ifconfig”,no-agent-forwarding,no-port-forwarding,no-pty,no-X11-forwarding,principals=”us...@domain.com” ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDFJU7IPlVkSWvMTFVEEaWAuF/6VehtVBEmr0Q4ihqqfnx/JCOzCmhrIcJ+pR+ifOMMriL/6yd7nMQXI8ZFG4j/mTAzbTbV1qK1rhhF5Cjn/Y+zmk47rDD8zANcgdzJ6KyVzjGo6hXdcLuH80QC+OUAttZF0fCNil+48LclsA6tdtCXcl8gkOytbHKYLsT8cYvVdp6vNqBpn7EN7CupQ7+cjfb3iaGOn6LopHwPHA/ccCGTccsRMdqryE085seyoNmGM6bd8K2vkWWzQb/QNQ9NqK+vpzZWKtaY48/F80RAndUQiY5t6am74VLjJrN9VRpKaa1fP+lHahM7nOIi5BT/q2wnKzMzErMAfypy1jGq5PupusZ+CLYHc/qzHm09HCGYYYH6nM0CxhDo1MsUtcyZMDgEr4X8Q41ygdlL+8XIAzY7Oqt18dZPPIvY1mWfiE5z2OdLsyOI7hjsOSoCkLN65rr+z39Una+E66g9GbkTlaFoRXnoEbiuIpASQQY40v2Vz+k0tMqFxaI1VMsxvpg2zOfpRtrRUpOuq2tmNqkOhaHrM6RU4lyV5sgzZLCS1zTdK/9764jTkLYorP3/8GAFSn950sP7bjzWH0ONcWA5kauoyK86BN0ncW/njanxReY0pUbpbtCNo8hjZGH3ZwIfLf4/l5hDUdosCQgd8nnCQ== UserSigningCA


SSH user certificate:

user_key-cert.pub:
Type: ssh-rsa-...@openssh.com user certificate
Public key: RSA-CERT SHA256:61zNSnUJ/2gyjo838P0U8H8eqQR1EkhJPj7pF9CaxUU
Signing CA: RSA SHA256:AsEE0T/P7Z0o/s6q8egBquay8WLL2sJHOLzYfc3N484
Key ID: “user-0001”
Serial: 0
Valid: from 2018-07-31T11:54:00 to 2020-07-30T11:55:56
Principals:
us...@domain.com
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc


Related lines from sshd_config file:

TrustedUserCAKeys /etc/ssh/user_ca_key.pub
PermitRootLogin forced-commands-only
AuthorizedKeysFile %h/.ssh/authorized_keys
AuthorizedPrincipalsFile %h/.ssh/authorized_principals


My conclusion is that if TrustedUserCAKeys is defined in sshd_config file of remote server, sshd service will firstly check authorized_principals file for principal match and only if there is no match it will proceed to authorized_keys. This can be seen in server logs.

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 898
debug2: parse_server_config: config /etc/ssh/sshd_config len 898
debug3: /etc/ssh/sshd_config:21 setting Protocol 2
debug3: /etc/ssh/sshd_config:23 setting HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
debug3: /etc/ssh/sshd_config:29 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:32 setting RevokedKeys /etc/ssh/revoked_keys
debug3: /etc/ssh/sshd_config:33 setting TrustedUserCAKeys /etc/ssh/user_ca_key.pub
debug3: /etc/ssh/sshd_config:41 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:42 setting SyslogFacility AUTHPRIV
debug3: /etc/ssh/sshd_config:43 setting LogLevel DEBUG
debug3: /etc/ssh/sshd_config:48 setting PermitRootLogin forced-commands-only
debug3: /etc/ssh/sshd_config:54 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:55 setting AuthorizedKeysFile %h/.ssh/authorized_keys
debug3: /etc/ssh/sshd_config:59 setting AuthorizedPrincipalsFile %h/.ssh/authorized_principals
debug3: /etc/ssh/sshd_config:74 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:78 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:88 setting GSSAPIAuthentication no
debug3: /etc/ssh/sshd_config:104 setting UsePAM yes
debug3: /etc/ssh/sshd_config:107 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:108 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:109 setting AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
debug3: /etc/ssh/sshd_config:110 setting AcceptEnv XMODIFIERS
debug3: /etc/ssh/sshd_config:116 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:129 setting UseDNS no
debug3: /etc/ssh/sshd_config:139 setting Subsystem sftp /usr/libexec/openssh/sftp-server
debug1: sshd version OpenSSH_5.3p1
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key-cert.pub.
debug1: ssh_rsa_verify: signature correct
debug1: host certificate: #0 type 5 RSA-CERT
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug3: oom_adjust_setup
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 898
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from X.X.X.X port 49780
debug1: Client protocol version 2.0; client software version OpenSSH_for_Windows_7.7
debug1: match: OpenSSH_for_Windows_7.7 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 15856
debug3: preauth child monitor started
debug3: mm_request_receive entering
debug3: privsep user:group 74:74
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-rsa-...@openssh.com
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 856 bytes for a total of 877
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-rsa-...@openssh.com
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnda...@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnda...@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,uma...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ri...@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,uma...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ri...@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zl...@openssh.com
debug2: kex_parse_kexinit: none,zl...@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256,curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: kex_parse_kexinit: ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ssh-ed2551...@openssh.com,ssh-rsa-...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: kex_parse_kexinit: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com
debug2: kex_parse_kexinit: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com
debug2: kex_parse_kexinit: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found uma...@openssh.com
debug1: kex: client->server aes128-ctr uma...@openssh.com none
debug3: mm_request_send entering: type 78
debug3: mm_request_receive_expect entering: type 79
debug3: mm_request_receive entering
debug3: monitor_read: checking request 78
debug3: mm_request_send entering: type 79
debug3: mm_request_receive entering
debug2: mac_setup: found uma...@openssh.com
debug1: kex: server->client aes128-ctr uma...@openssh.com none
debug3: mm_request_send entering: type 78
debug3: mm_request_receive_expect entering: type 79
debug3: mm_request_receive entering
debug3: monitor_read: checking request 78
debug3: mm_request_send entering: type 79
debug3: mm_request_receive entering
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug3: mm_request_send entering: type 0
debug3: mm_choose_dh: waiting for MONITOR_ANS_MODULI
debug3: mm_request_receive_expect entering: type 1
debug3: mm_request_receive entering
debug3: monitor_read: checking request 0
debug3: mm_answer_moduli: got parameters: 2048 3072 8192
debug3: mm_request_send entering: type 1
debug2: monitor_read: 0 used once, disabling now
debug3: mm_request_receive entering
debug3: mm_choose_dh: remaining 0
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug3: Wrote 408 bytes for a total of 1285
debug2: dh_gen_key: priv key bits set: 133/256
debug2: bits set: 1555/3072
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug2: bits set: 1538/3072
debug3: mm_key_sign entering
debug3: mm_request_send entering: type 5
debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN
debug3: mm_request_receive_expect entering: type 6
debug3: mm_request_receive entering
debug3: monitor_read: checking request 5
debug3: mm_answer_sign
debug3: mm_answer_sign: signature 0x7fb71cdd13c0(271)
debug3: mm_request_send entering: type 6
debug2: monitor_read: 5 used once, disabling now
debug3: mm_request_receive entering
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 2216 bytes for a total of 3501
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug3: Wrote 40 bytes for a total of 3541
debug1: userauth-request for user root service ssh-connection method none
debug1: attempt 0 failures 0
debug3: mm_getpwnamallow entering
debug3: mm_request_send entering: type 7
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM
debug3: mm_request_receive_expect entering: type 8
debug3: mm_request_receive entering
debug3: monitor_read: checking request 7
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 898
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 8
debug2: monitor_read: 7 used once, disabling now
debug3: mm_request_receive entering
debug2: input_userauth_request: setting up authctxt for root
debug3: mm_start_pam entering
debug3: mm_request_send entering: type 50
debug3: mm_inform_authserv entering
debug3: mm_request_send entering: type 3
debug3: mm_inform_authrole entering
debug3: mm_request_send entering: type 4
debug2: input_userauth_request: try method none
debug3: Wrote 56 bytes for a total of 3597
debug3: monitor_read: checking request 50
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "X.X.X.X"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 50 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 3
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 3 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authrole: role=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 1 failures 0
debug2: input_userauth_request: try method publickey
debug1: ssh_rsa_verify: signature correct
debug1: test whether pkalg/pkblob are acceptable
debug3: mm_key_allowed entering
debug3: mm_request_send entering: type 21
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED
debug3: mm_request_receive_expect entering: type 22
debug3: mm_request_receive entering
debug3: monitor_read: checking request 21
debug3: mm_answer_keyallowed entering
debug1: ssh_rsa_verify: signature correct
debug3: mm_answer_keyallowed: key_from_blob: 0x7fb71cddd940
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying authorized principals file /root/.ssh/authorized_principals
debug1: fd 4 clearing O_NONBLOCK
debug3: secure_filename: checking '/root/.ssh'
debug3: secure_filename: checking '/root'
debug3: secure_filename: terminating check at '/root'
debug1: restore_uid: 0/0
Certificate does not contain an authorized principal
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys '/root/.ssh/authorized_keys': No such file or directory
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys '/root/.ssh/authorized_keys': No such file or directory
debug1: restore_uid: 0/0
Failed publickey for root from X.X.X.X port 49780 ssh2
debug3: mm_answer_keyallowed: key 0x7fb71cddd940 is not allowed
debug3: mm_request_send entering: type 22
debug3: mm_request_receive entering
debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa-...@openssh.com
debug3: Wrote 56 bytes for a total of 3653