RAPI extension to the serial console
Lance Albertson
console on a instance via Ganeti Web Manager recently [1]. While we
could do something hackish and just connect to the ganeti command line
interface, we decided it probably would be better if we could just get
support for this included in Ganeti itself. I wasn't sure if we should
bring this up on this list or the devel list so feel free to redirect me.
The idea we had would be to add a feature to the Ganeti RAPI which would
open an INET socket from the console (i.e. for KVM its the UNIX socket).
There probably should be some kind of authentication that happens on the
socket so that its not wide open to the world.
Do you have any thoughts on this idea? Does it sound viable or do you
have any other solutions to this problem? We'll probably work on
creating a patch for this and submitting it for review once we got it
working.
Thanks-
[1] http://code.osuosl.org/issues/2217
--
Lance Albertson
Systems Administrator / Architect Open Source Lab
Network Services Oregon State University
Michael Hanselmann
Am 26. Januar 2011 18:23 schrieb Lance Albertson <la...@osuosl.org>:
> We've been discussing adding support for connecting to the serial
> console on a instance via Ganeti Web Manager recently [1]. While we
> could do something hackish and just connect to the ganeti command line
> interface, […]
Ganeti 2.4 will have an improved and more verbose result for
OpInstanceConsole, see commits 25ce3ec49d and 55cc0a44d5.
> The idea we had would be to add a feature to the Ganeti RAPI which would
> open an INET socket from the console (i.e. for KVM its the UNIX socket).
> There probably should be some kind of authentication that happens on the
> socket so that its not wide open to the world.
I'll send a patch to export the console information via RAPI. I don't
think the RAPI daemon is the right place for providing direct access
to the console. For one, doing so will involve long-running processes
(what if someone leaves the console open for a week?), whereas the
current HTTP server is designed with short-running child processes in
mind.
Then you have the issues of authentication. How to manage it? You'll
have to figure out. SSL client certificates are certainly a
possibility. Passwords not so much as you might run into issues when
doing authentication and console over the same connection.
All in all I think console access should be provided by a separate
process, if it's necessary at all. With the verbose console
information you can also just SSH into the cluster (using a restricted
user) and connect directly.
> Do you have any thoughts on this idea? Does it sound viable or do you
> have any other solutions to this problem? We'll probably work on
> creating a patch for this and submitting it for review once we got it
> working.
Please write at least a small design doc before writing code for such
a change. There are many potential issues with authentication.
Michael
Lance Albertson
> Hi Lance
>
> Am 26. Januar 2011 18:23 schrieb Lance Albertson <la...@osuosl.org>:
>> We've been discussing adding support for connecting to the serial
>> console on a instance via Ganeti Web Manager recently [1]. While we
>> could do something hackish and just connect to the ganeti command line
>> interface, […]
>
> Ganeti 2.4 will have an improved and more verbose result for
> OpInstanceConsole, see commits 25ce3ec49d and 55cc0a44d5.
Was this feature intended to be useful for having external applications
running the commands more directly?
>> The idea we had would be to add a feature to the Ganeti RAPI which would
>> open an INET socket from the console (i.e. for KVM its the UNIX socket).
>> There probably should be some kind of authentication that happens on the
>> socket so that its not wide open to the world.
>
> I'll send a patch to export the console information via RAPI. I don't
> think the RAPI daemon is the right place for providing direct access
> to the console. For one, doing so will involve long-running processes
> (what if someone leaves the console open for a week?), whereas the
> current HTTP server is designed with short-running child processes in
> mind.
I think I didn't explain this clearly. We were wanting the RAPI to only
initiate a backend process that would then create this INET socket.
Ganeti itself could probably then implement some sort of sane timeout.
We weren't intending to serve console data over the RAPI itself.
Regardless, I think an RAPI patch for providing this information will be
useful!
> Then you have the issues of authentication. How to manage it? You'll
> have to figure out. SSL client certificates are certainly a
> possibility. Passwords not so much as you might run into issues when
> doing authentication and console over the same connection.
Right, that is an issue with this feature.
> All in all I think console access should be provided by a separate
> process, if it's necessary at all. With the verbose console
> information you can also just SSH into the cluster (using a restricted
> user) and connect directly.
I talked this over with Peter (our lead developer on the GWM project)
and I think we have a better solution. I suggested connect directly to
the cluster via ssh from a proxy daemon we're currently writing for GWM.
We'd have a restricted key on the cluster for either the root user or a
regular user with sudo access to execute the commands needed.
That way we can use our own authentication layer based on GWM
credentials in the proxy and only use the ssh key as the auth mechanism
to the cluster itself. From a security POV this is pretty sound assuming
that ganeti doesn't have any security holes via the command line. My
main concern is the possibility of a hole that allows the user to gain a
shell on the node.
Do you think that's an acceptable solution?
>> Do you have any thoughts on this idea? Does it sound viable or do you
>> have any other solutions to this problem? We'll probably work on
>> creating a patch for this and submitting it for review once we got it
>> working.
>
> Please write at least a small design doc before writing code for such
> a change. There are many potential issues with authentication.
Will do if we move forward with needing the INET socket route. I think
the ssh route may be a better solution.
Thanks!
Iustin Pop
> I talked this over with Peter (our lead developer on the GWM project)
> and I think we have a better solution. I suggested connect directly to
> the cluster via ssh from a proxy daemon we're currently writing for GWM.
> We'd have a restricted key on the cluster for either the root user or a
> regular user with sudo access to execute the commands needed.
>
> That way we can use our own authentication layer based on GWM
> credentials in the proxy and only use the ssh key as the auth mechanism
> to the cluster itself. From a security POV this is pretty sound assuming
> that ganeti doesn't have any security holes via the command line. My
> main concern is the possibility of a hole that allows the user to gain a
> shell on the node.
>
> Do you think that's an acceptable solution?
We've been using exactly this solution internally for a couple of years.
We've never opensourced it as it is very tied into the internal access
controls, etc., but we can confirm this works well.
regards,
iustin
Lance Albertson
That's great to hear. At some point I would love to hear some
comparisons of what you use internally vs. Ganeti Web Manager although
you probably can't. Would be interesting to see if we took a similar
route or if there are things we could leverage from how you implemented it.
Thanks for the feedback!
Iustin Pop
Ah, in this case it wasn't an Web console. Simply an SSH bastion that
can login to clusters, via an unpriviledged user that has sudo rights,
and which understands a simple protocol:
- start/stop
- reinstall
- console
This particular solution has worked well for its intended purpose, so I
believe that a Web interface that does the same (except that it's done
via a Web UI) would work well.
regards,
iustin
Lance Albertson
We were actually talking about implementing something like that anyways
but ya this is intended to be for the web interface originally. Its good
to know we're headed down the right track :-)