Python exception: 'out of pty devices'

[risk...@gmail.com]

I was just playing around today trying to get Gate One running inside Sandstorm and it's encountering an exception trying to create PTYs:

[E 140804 12:14:42 server:4450] Exception was: ('out of pty devices',)

Gate One needs pty devices to work (really no way around that requirement). I'm just wondering if there's any way I can make sure the sandbox includes /dev/pts and /dev/ptmx when it gets created in order to solve that problem.

Perhaps there's a way to configure such things in the sandstorm-pkgdev.capnp file (or similar)?

Thanks.

[Kenton Varda]

Hi Dan,

Thanks for trying out the porting process!

The reason the TTY devices aren't there is because we have tried to minimize the kernel interfaces that are exposed inside the sandbox in order to reduce the attack surface for potential sandbox breakouts. New kernel vulnerabilities seem to be discovered monthly, but most don't affect Sandstorm because of our efforts to keep the interface narrow.

We could have a discussion about either enabling TTY devices or implementing some userspace emulation of them (there's no theoretical reason they need to be in the kernel, after all, since we're not dealing with actual hardware).

But I think the first question is: Is Gate One useful in an environment with no network? Currently, Sandstorm apps are isolated; it's not possible to create arbitrary outbound connections. So, e.g., you wouldn't be able to connect to some other server over SSH; you could only run commands locally.

We may expand that in the future, but it's much easier to start secure and then poke holes than it is to start wide open and then try to lock things down later. :)

-Kenton

--

Sandstorm.io is crowdfunding! http://igg.me/at/sandstorm

--
You received this message because you are subscribed to the Google Groups "Sandstorm Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan McDougall]

On Monday, August 4, 2014 9:28:55 PM UTC-4, Kenton Varda wrote:

Hi Dan,

Thanks for trying out the porting process!

The reason the TTY devices aren't there is because we have tried to minimize the kernel interfaces that are exposed inside the sandbox in order to reduce the attack surface for potential sandbox breakouts. New kernel vulnerabilities seem to be discovered monthly, but most don't affect Sandstorm because of our efforts to keep the interface narrow.

We could have a discussion about either enabling TTY devices or implementing some userspace emulation of them (there's no theoretical reason they need to be in the kernel, after all, since we're not dealing with actual hardware).

But I think the first question is: Is Gate One useful in an environment with no network? Currently, Sandstorm apps are isolated; it's not possible to create arbitrary outbound connections. So, e.g., you wouldn't be able to connect to some other server over SSH; you could only run commands locally.

We may expand that in the future, but it's much easier to start secure and then poke holes than it is to start wide open and then try to lock things down later. :)

I don't think you can get away with making TTYs in user space since the Linux kernel itself doesn't support that (yet).  Both LXC and Docker duplicate the PTYs and ptmx to their respective sandboxes.  The reason is that anything that needs to run inside of or spawn subprocesses via a "proper" shell will need /dev/pts and /dev/ptmx (which can just be a symlink to /dev/pts/ptmx).

I'd be happy to try all sorts of alternatives if any are available.  Gate One works by using Python's pty module which internally calls os.open('/dev/pty/<name>', os.O_RDWR).  I could use os.fork() instead but then things like shells and sudo wouldn't work (assuming we're just running a command locally--it would work fine if connected via SSH).  As an example of what breaks without PTYs:  Bash.  If you run bash without a pty device available it will only run in non-interactive mode which breaks all sorts of stuff.

Also consider that a number of web-based applications use things like expect, pexpect, etc to control background processes.  None of that will work if /dev/ptx/* isn't available.

I honestly don't know what the security implications of having /dev/pts devices available inside the sandbox would be other than having the ability to spawn things like shells.  I don't think you couldn't use it to break out of the sandbox.

BTW:  Yes, Gate One can work just fine without outbound networking.  You can configure it to run any terminal program you want (e.g. nethack).  That terminal program doesn't have to be SSH.  It will soon also be able to do things like run X11 applications.

[Kenton Varda]

Hi Dan,

Actually, it is quite possible to implement terminals in userspace. You just need a way to intercept system calls. It sounds crazy, but it actually isn't hard at all, and there are even several ways to do it:

- Edit glibc's code to redirect ioctl() and other ops that apply to terminals. Since you can package your own glibc with your app, this actually works.

- Use LD_PRELOAD to load a library that overrides ioctl() et al. This accomplishes the same as above without editing glibc, but it a bit hackier.

- Use seccomp-bpf to make these system calls raise a signal, and implement them in the signal handler. You probably still need an LD_PRELOAD library to implement the signal handler.

Sandstorm itself is likely to implement the third approach at some point in the future, transparently to the app, but probably not for several months at least. Until then the first or second approach is probably the easiest to use in-app.

That said, I understand that you probably aren't particularly interested in working through this, especially if Sandstorm will do it transparently eventually anyway.

We could consider exposing the "real" pty devices temporarily as a way for apps to move forward, but I would need to do some more research to make sure I understand how they work and whether there are any security implications here. One thing I'm worried about is that an app would be able to open pts devices corresponding to some other app's ptm. All Sandstorm containers on a host actually run under the same UID since we instead rely on namespaces for separation, but AFAICT /dev/pts is a namespace that I cannot currently unshare, so there would be nothing preventing you from opening someone else's pts. That said, I would think Docker has solved this somehow, so maybe I'm missing something.

-Kenton

--

Sandstorm.io is crowdfunding! http://igg.me/at/sandstorm

--

[Adam Bliss]

Was there ever more progress on the question of PTY devices? I think it would be cool to port something like https://github.com/tsl0922/ttyd to sandstorm. It could be useful as a dev tool, to quickly prototype ideas inside the sandbox. It could also be pretty useful for pair-programming sessions or phone interviews. I could even imagine it serving as a useful platform for quick-and-dirty sandstorm ports of apps that have good command-line interfaces.

--Adam

[Adam Bliss]

I did a quick bit of research and it looks like a nonshared devpts filesystem can be mounted inside a user namespace since Linux 3.9. I tried doing it myself from inside a grain, but I don't seem to be able to create a new user namespace from within the grain namespace. So perhaps Sandstorm can/should/must do this itself when it sets up the grain?

[Ian Denhardt]

A prereq for sandstorm doing this is doing some research on the security
implications of exposing the API at all. I have vague memories of the
tty layer actually having a really lousy security track record, and am
nervous about opening that up to apps without clear evidence that it's
not a problem (anymore).

The LD_PRELOAD solution is probably good enough to do what we need,
honestly; it will just take a bit of hacking. And has the advantage of
not exposing any extra attack surface to apps. Esp. given relatively few
apps even need this...

Quoting Adam Bliss (2020-05-13 18:03:19)

> I did a quick bit of research and it looks like a nonshared devpts
> filesystem can be mounted inside a user namespace since Linux 3.9. I
> tried doing it myself from inside a grain, but I don't seem to be able
> to create a new user namespace from within the grain namespace. So
> perhaps Sandstorm can/should/must do this itself when it sets up the
> grain?
>

> On Wed, May 13, 2020, 17:06 Adam Bliss <[1]abl...@gmail.com> wrote:
>
> Was there ever more progress on the question of PTY devices? I think it
> would be cool to port something

> like� [2]https://github.com/tsl0922/ttyd to sandstorm. It could be

> useful as a dev tool, to quickly prototype ideas inside the sandbox. It
> could also be pretty useful for pair-programming sessions or phone
> interviews. I could even imagine it serving as a useful platform for
> quick-and-dirty sandstorm ports of apps that have good command-line
> interfaces.
> --Adam
>

> Sandstorm.io is crowdfunding! [4]http://igg.me/at/sandstorm
> On Tue, Aug 5, 2014 at 11:59 AM, Dan McDougall <[5]risk...@gmail.com>

> an email to [6]sandstorm-de...@googlegroups.com.
> For more options, visit [7]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Sandstorm Development" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to [8]sandstorm-de...@googlegroups.com.
> For more options, visit [9]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Sandstorm Development" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [10]sandstorm-de...@googlegroups.com.
> To view this discussion on the web visit
> [11]https://groups.google.com/d/msgid/sandstorm-dev/CAEAeJWy2Xc%2B92rRN
> pCZDsSVcJBuZtsOupgVDBmnq%2Bj9DovKWwg%40mail.gmail.com.
>
> Verweise
>
> 1. mailto:abl...@gmail.com
> 2. https://github.com/tsl0922/ttyd
> 3. mailto:ken...@sandstorm.io
> 4. http://igg.me/at/sandstorm
> 5. mailto:risk...@gmail.com
> 6. mailto:sandstorm-de...@googlegroups.com
> 7. https://groups.google.com/d/optout
> 8. mailto:sandstorm-de...@googlegroups.com
> 9. https://groups.google.com/d/optout
> 10. mailto:sandstorm-de...@googlegroups.com
> 11. https://groups.google.com/d/msgid/sandstorm-dev/CAEAeJWy2Xc%2B92rRNpCZDsSVcJBuZtsOupgVDBmnq%2Bj9DovKWwg%40mail.gmail.com?utm_medium=email&utm_source=footer

[Adam Bliss]

Yeah, doing it purely in userspace definitely seems preferable. I've been searching around for a userspace PTY emulation library, and haven't found anything yet (though "kmscon" claims to be an entire graphical console in userspace, so perhaps it includes what we would need). I'm still very unclear what actually happens underneath the pty interface (something about "line disciplines") or why the kernel should be involved in implementing it in the first place...

[Zach Balleisen]

This is a problem I've been fighting with my Jupyter Lab port as well, so it would awesome if there was a way to get TTY functionality without compromising security.

To unsubscribe from this group and stop receiving emails from it, send an email to sandstorm-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sandstorm-dev/CAEAeJWyfWndon6scobNwURiPT1o71cdOXOgzjopsPCRt8jNeUw%40mail.gmail.com.

[Adam Bliss]

Zach, I've had some promising results from my LD_PRELOAD experiments: https://github.com/abliss/upty . For example, I can host a mostly-functional terminal shell inside a grain with https://github.com/abliss/ttyd . I don't know if it'll be helpful for your Jupyter Lab port. Maybe if you send me some more detail on your problems, I can try to reproduce them locally and see if upty can be made to help?

--Adam