SSH backend - bastion hosts?

[Brian Rodgers]

I'm very excited about the ssh backend coming in 0.3.  One question I had though is that we are setting up SSH for our VPCs modeled around the bastion host concept.  That is, we have a server that only exists to proxy ssh traffic to the "real" boxes, and in fact we turn it off when not needed.  We use this with ssh ProxyCommand entries in ssh config files to jump through the bastion host transparently.  Right now we use a single SSH key for all hosts including the bastion box, though ideally we might eventually split that out 

Vault will be in a separate VPC from the hosts we would want it to be managing.  So it would need to jump through the bastion host itself to do what it needs to do.  I took a quick look at the docs page for the SSH backend that's in Git -- I realize it isn't yet published and may not be final, but I didn't see a way this would work.  Is there any such concept in the backend, and if not, any thoughts on what it'd take to add it?  I'm happy to open a ticket up for it, but I wanted to see if it's been considered already first.  There'd need to be some way to associate which bastion host (and which key for the bastion host if we want them to be different) goes with a given server.  

Our security folks would be much happier if we could stick to the rule that ssh traffic is only allowed from the bastion, but I can discuss the idea of allowing a direct path from Vault's IPs if you don't think this is practical.  I do realize there are possible side effects of going through the bastion host, such as if the bastion host gets shut off before an SSH key's lease expires.  But it's still a model I'd be interested in seeing supported despite that.

[Vishal Nayak]

Hi Brian,

If the only job of bastion host is to proxy SSH connections, then Vault with SSH backend can "be that bastion", providing more control over the "real" boxes. Instead of turning-off the bastion host to stop SSH traffic, Vault can selectively turn off access to sets of machines through their respective 'roles'.

Regards,

Vishal

[Brian Rodgers]

Interesting thought.  However, I'm realizing now that there's another important purpose of the bastion host that I wasn't thinking about when I posted yesterday.  The bastion host lives in the public subnet, and most of the EC2 instances we'd want to SSH into live in the private subnet.  So it is also serving as a proxy to get to the private subnet.  I actually can't make those servers available to vault directly without putting some kind of transparent SSH proxy in place in the public subnet to get to them.  I'd rather not do that (if it's even possible to transparently proxy SSH) and stick with the bastion concept.  So I do think I'd need vault to have some bastion functionality.

[Jeff Mitchell]

Hi Brian,

My suggestion is to use haproxy to do TCP proxying from the public
side to the private side. Since it's simply proxying the TCP
connection, the actual authentication happens on the host on the
private subnet; the server could be configured to use the OTP
functionality of the SSH backend, reaching out to Vault to verify the
given OTP (and auditing along the way). I think this should solve the
problem you have nicely -- you don't need to deal with SSH
ProxyCommand stanzas and its set of capabilities, and since the OTP
functionality doesn't require Vault to get onto the box to set the
allowed OTPs, Vault doesn't need to be able to get into the
systems...the hosts on the private subnet just need to be able to get
to Vault.

If you were thinking about the dynamic key capability of the backend,
please read the docs
(https://github.com/hashicorp/vault/blob/master/website/source/docs/secrets/ssh/index.html.md)
-- it exists for those that need it but is highly discouraged.

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/216e3e85-9ba2-4f04-97e3-b822af6fa096%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Brian Rodgers]

This isn't out of the question, though I'd need to do some work to get my security guys on board.  They're pretty well sold on the idea of a bastion host proxying SSH.

Beyond that though, I'm wondering about the logistics of managing that.  Right now, the one bastion host -- with one public IP -- can proxy to any EC2 instance in the VPC by that instances IP.  And we can query AWS for the instances in the VPC and their IPs and automatically generate ProxyCommands within an SSH config file for that VPC.  In fact we have that fully automated, so if an autoscaling event happens we just re-run that automation to recreate the SSH config.  Using the tags on the EC2 instance, we generate friendly names in that ssh config as well (service1, service2, etc).

If I were to go with transparent proxying directly to the instances with haproxy or nginx, that automation would need to be replaced with something that updated the proxy config.  That's doable, but I believe I'd either need to allocate an IP on the proxy box for every host I want to proxy to, or assign a port for each machine.  In either case, I'd have some way clients can (again through automation) get that mapping of ports to the machine they're trying to get to.  There are certainly ways I can make that work, but it seems like it's actually quite a bit more complicated than the system I have now.

Are there reasons why you don't think the SSH backend couldn't be modified to support the bastion model?  Or are you just offering some suggestions for alternatives?  The suggestions are appreciated, but I think the bastion model I'm describing is common, and it'd be useful to support it.  

Also, why do you strongly discourage the dynamic key option?  Is it just the audit reason that is discussed in the doc, or something else?

[Jeff Mitchell]

I'm going to answer your questions a little out of order...

> Also, why do you strongly discourage the dynamic key option? Is it just the
> audit reason that is discussed in the doc, or something else?

It's one of two major reasons.The thing to keep in mind about auditing
with the dynamic keys backend is that unless you can configure SSH to
not only log who logged in when but which key was used (which I don't
think you can do, but I'm happy to be proven wrong), you have no way
to correlate tokens with actions. It's an ordering problem: if three
users grab keys for a host, and then log into the machine in the
reverse order of which the keys were grabbed, you have no way of
knowing that this happened, as opposed to them logging in in-order, or
any other ordering.

The other major reason is that generating SSH keys is expensive, in an
entropy sense. Vault could very quickly be a bottleneck if the machine
can't generate entropy fast enough.

> This isn't out of the question, though I'd need to do some work to get my
> security guys on board. They're pretty well sold on the idea of a bastion
> host proxying SSH.

I used to work at a company that did the bastion model. It was okay,
except for the various SSH features that were unsupported or dog-slow.

> Beyond that though, I'm wondering about the logistics of managing that.
> Right now, the one bastion host -- with one public IP

You'll definitely want more than one, so that that machine going down
doesn't erase the availability of SSH to all of your machines. So then
you have to figure out how to make the fact that there are multiple
bastion hosts transparent. At the previous company I mentioned, there
was a command that you used in place of SSH that would figure out
available bastion hosts and connect to one.

> If I were to go with transparent proxying directly to the instances with
> haproxy or nginx, that automation would need to be replaced with something
> that updated the proxy config. That's doable, but I believe I'd either need
> to allocate an IP on the proxy box for every host I want to proxy to, or
> assign a port for each machine. In either case, I'd have some way clients
> can (again through automation) get that mapping of ports to the machine
> they're trying to get to. There are certainly ways I can make that work,
> but it seems like it's actually quite a bit more complicated than the system
> I have now.

I did this in the past using port numbers, and I exposed a consul
agent's DNS ports to my other subnet so that they could query for the
hosts's SSH service. However, there is another potential solution;
haproxy can inspect a connection to know whether it is an SSH
connection request, and it can also look at SNI. So I believe you
could accomplish this with haproxy by using the method here:
http://blog.chmd.fr/ssh-over-ssl-episode-4-a-haproxy-based-configuration.html
-- but by adding `-servername %h` to the openssl s_client command to
have it send an SNI header.

On haproxy, you look at the SNI header to determine which host to send
the connection to. You could also, if you wished, inspect the data to
ensure that it is an SSH connection, although if the only thing
haproxy would be doing is proxying to SSH then this would be
unnecessary.

Note that I haven't tried this specific scenario out, but I have both
(separately) routed based on SNI header within haproxy and proxied SSH
by haproxy (and it was quite performant).

> Are there reasons why you don't think the SSH backend couldn't be modified
> to support the bastion model? Or are you just offering some suggestions for
> alternatives? The suggestions are appreciated, but I think the bastion
> model I'm describing is common, and it'd be useful to support it.

I'm offering suggestions for alternatives a) because the usage of the
bastion hosts indicates that you are planning on using the dynamic key
model, which is very strongly discouraged; b) I think the bastion host
concept is fine, but multiple SSH jumps is quite slow,
performance-wise, and better solutions exist today; and c) because I
don't know when there will be time to add that support in...there's a
very full plate in the near term.

--Jeff