Sage public notebook servers

[William Stein]

Hello,

A security researcher decided to purposely take down sage.math to
demonstrate that it is possible to fork bomb the machine through the
public sage notebook servers. I had always plan to run these comletley
public servers until something like this happened. Therefore,
sagenb.org (and the other public sage notebook servers I host) will be
completely disable until further notice.

I might re-enable them in the future if I set them up from scratch
using a vmware virtual machine and vmware server. Given that I've never
successfully configured vmware server on any Linux box, I don't know
when this will happen. If a Sage developer would like to attempt to do
this instead of me on sage.math please contact me, since this is not
currently my highest priority (especially, because I'm in France
traveling right now).

-- William

[Martin Albrecht]

On Monday 13 October 2008, William Stein wrote:
> Hello,
>
> A security researcher decided to purposely take down sage.math to
> demonstrate that it is possible to fork bomb the machine through the
> public sage notebook servers.

Those 'security researchers' are also known as script kiddies.

Well it had to happen eventually.

Cheers,
Martin

--
name: Martin Albrecht
_pgp: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8EF0DC99
_www: http://www.informatik.uni-bremen.de/~malb
_jab: martinr...@jabber.ccc.de

[Jason Grout]

William Stein wrote:
> Hello,
>
> A security researcher decided to purposely take down sage.math to
> demonstrate that it is possible to fork bomb the machine through the
> public sage notebook servers. I had always plan to run these comletley
> public servers until something like this happened. Therefore,
> sagenb.org (and the other public sage notebook servers I host) will be
> completely disable until further notice.

So the issue is not necessarily the fork-bomb (which could be avoided
using process limitations), but in the malicious use of the notebook?

> I might re-enable them in the future if I set them up from scratch
> using a vmware virtual machine and vmware server. Given that I've never
> successfully configured vmware server on any Linux box, I don't know
> when this will happen. If a Sage developer would like to attempt to do
> this instead of me on sage.math please contact me, since this is not
> currently my highest priority (especially, because I'm in France
> traveling right now).
>

I have a virtualbox image based on ubuntu JeOS that I am using on Linux
right now. Would you consider using VirtualBox?

Thanks,

Jason

[Timothy Clemans]

I had never heard of "fork bomb" until now. According to Wikipedia,
it's somewhat preventable by implementing a limit of the number of
processes per user.

I like the fact that Knoboo makes it easy to run the actual Sage
processes on a completely different machine or at least in a virtual
machine. At some point Knoboo might have a system for dealing with
down kernel servers where one can still access and download notebooks.

Would the entire Sage Notebook be ran in a VMWare image or the
individual Sage per sage unix user processes inside their own? So like
sage0 would have a virtual machine, sage1 would have its own, etc.

[mabshoff]

On Oct 13, 3:05 pm, "Timothy Clemans" <timothy.clem...@gmail.com>
wrote:

Hi Timothy,

> I had never heard of "fork bomb" until now. According to Wikipedia,
> it's somewhat preventable by implementing a limit of the number of
> processes per user.

just read "man ulimit" :)

> I like the fact that Knoboo makes it easy to run the actual Sage
> processes on a completely different machine or at least in a virtual
> machine. At some point Knoboo might have a system for dealing with
> down kernel servers where one can still access and download notebooks.

Nope, once you fork bomb and you do not have a root shell open to the
box it is game over in the vast majority of cases. Any external access
usually requires a fork of some sort and since someone just fork
bombed the box it is a gonner.

> Would the entire Sage Notebook be ran in a VMWare image or the
> individual Sage per sage unix user processes inside their own? So like
> sage0 would have a virtual machine, sage1 would have its own, etc.

Yep, that is pretty much the way to go together with some more tweaks
to the setup. The main issue is that a skilled attacker (not likely
the person who fork bombed the box) can break out or DOS pretty much
any setup, so one has to assume that people interested in using the
Sage notebook are neither idiots or assholes. Back in the day I also
did penetration testing and in the end if you give someone a local
shell account (which is pretty much any notebook account) you have to
trust the person to some extent. Given a shell account it is only a
question of time even for someone semi-skilled to execute an exploit
found on the net before one can patch the box. I guess in the end the
people relying on the public notebook server are the screwed ones here
because even once the server is back up it will be much more locked
down.

Cheers,

Michael

<SNIP>

[Jason Grout]

If this is a final decision, at least for the short term, can we pull
the links from the sage website and put some explanatory text up?

Thanks,

Jason

[William Stein]

Yes, this is a shortterm final decision. I always planned to run the
public servers as is until some script kiddie $%%#$^% etc. I assume
Harald Schilly will fix the website.

Note that as I say above, this "final decision" really is "short term",
i.e., probably 2-3 weeks. vmware is perfect for this application.

William

[Robert Miller]

> A security researcher decided to purposely take down sage.math to
> demonstrate that it is possible to fork bomb the machine through the
> public sage notebook servers.   I had always plan to run these comletley
> public servers until something like this happened.  Therefore,
> sagenb.org (and the other public sage notebook servers I host) will be
> completely disable until further notice.

Tell him where to put his next fork bomb for me.

[William Stein]

On Tue, Oct 14, 2008 at 1:06 AM, Serge Salamanka <sals...@tut.by> wrote:
>
> I have experience with Xen.
> Can set up a virtual machine for Sage.
> It's not that difficult anyway.

What do you need to do this? How secure are they?

By the way, I'm currently copied all the data from sagenb.org
and the other servers to
http://sage.math.washington.edu/home/was/sagenb/
so people can recover any worksheets they have.

William

>>
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[Dorian Raymer]

Hi,
This is a good discussion. It's interesting to see, after all this time, the public notebook being attacked! (exclaimed in the most respectful, positive, excited that now this problem really has to be solved manor :)

Over the last few months I've been thinking about and working on the problems associated with running notebook processes (which is like essentially handing out shell accounts) in a safe, secure and useful way.
I split the problems into orthogonal (as possible) components, and the one I worked on most recently asks 'what is the best way to run a secure python process'.
There are quite a few pieces of work that exist on this problem; probably the main bottleneck in progressing is my inexperience with system administration and knowledge of how Python works.

Rough work and references:
http://trac.knoboo.com/wiki/Security

I've made some headway in directions varying from Sage's security model as it was, but unfortunately I am too busy at the moment to make any serious progress.

I plan to resume working on this in about a month. It would be great to consult and collaborate with Michael Mabshoff and any other experienced gurus of that sort -- sys admins and programmers that really understand security in the os environment.

On Mon, Oct 13, 2008 at 3:31 PM, mabshoff <mabs...@googlemail.com> wrote:

On Oct 13, 3:05 pm, "Timothy Clemans" <timothy.clem...@gmail.com>
wrote:

Hi Timothy,

> I had never heard of "fork bomb" until now. According to Wikipedia,
> it's somewhat preventable by implementing a limit of the number of
> processes per user.

just read "man ulimit" :)

> I like the fact that Knoboo makes it easy to run the actual Sage
> processes on a completely different machine or at least in a virtual
> machine. At some point Knoboo might have a system for dealing with
> down kernel servers where one can still access and download notebooks.

Nope, once you fork bomb and you do not have a root shell open to the
box it is game over in the vast majority of cases. Any external access
usually requires a fork of some sort and since someone just fork
bombed the box it is a gonner.

Actually, the concept Timothy is talking about is true. The framework for running notebook processes in Knoboo is very different from what Sage does to serve notebooks. Indeed, the machine running actual notebook processes (or engine processes as we call them)  is considered history in this situation, however, access to the notebook data would be absolutely un-affected -- an important point -- because users *can* still view/download their notebook data as it is maintained by an entirely separate system running on a physically separated machine. Not to be nit-picky, but this is very attractive from a user perspective; it's not as catastrophic as the entire service disappearing (like what's happening now).


 

> Would the entire Sage Notebook be ran in a VMWare image or the
> individual Sage per sage unix user processes inside their own? So like
> sage0 would have a virtual machine, sage1 would have its own, etc.

Yep, that is pretty much the way to go together with some more tweaks
to the setup. The main issue is that a skilled attacker (not likely
the person who fork bombed the box) can break out or DOS pretty much
any setup, so one has to assume that people interested in using the
Sage notebook are neither idiots or assholes. Back in the day I also
did penetration testing and in the end if you give someone a local
shell account (which is pretty much any notebook account) you have to
trust the person to some extent. Given a shell account it is only a
question of time even for someone semi-skilled to execute an exploit
found on the net before one can patch the box. I guess in the end the
people relying on the public notebook server are the screwed ones here
because even once the server is back up it will be much more locked
down.

This raises the issue of how to balance security (how tight to lock the server down) with feature/functional availability (what functionality is crippled due to imposed security constraints)...


Regards,
-Dorian

 

Cheers,

Michael

<SNIP>

[Jason Grout]

>
> Actually, the concept Timothy is talking about is true. The framework
> for running notebook processes in Knoboo is very different from what
> Sage does to serve notebooks. Indeed, the machine running actual
> notebook processes (or engine processes as we call them) is considered
> history in this situation, however, access to the notebook data would be
> absolutely un-affected -- an important point -- because users *can*
> still view/download their notebook data as it is maintained by an
> entirely separate system running on a physically separated machine. Not
> to be nit-picky, but this is very attractive from a user perspective;
> it's not as catastrophic as the entire service disappearing (like what's
> happening now).

If I understand things correctly, the biggest obstacle to Sage doing
this currently is that the client processes expect to be able to read
and write the notebook files (the files associated with the cells in the
notebook) and the server expects the output to automatically appear in
the correct cell directory. Is that correct?

Jason

[Alex Clemesha]

On Mon, Oct 13, 2008 at 3:31 PM, mabshoff <mabs...@googlemail.com> wrote:
>

> On Oct 13, 3:05 pm, "Timothy Clemans" <timothy.clem...@gmail.com>
> wrote:
>
> Hi Timothy,
>
>> I had never heard of "fork bomb" until now. According to Wikipedia,
>> it's somewhat preventable by implementing a limit of the number of
>> processes per user.
>
> just read "man ulimit" :)
>
>> I like the fact that Knoboo makes it easy to run the actual Sage
>> processes on a completely different machine or at least in a virtual
>> machine. At some point Knoboo might have a system for dealing with
>> down kernel servers where one can still access and download notebooks.
>
> Nope, once you fork bomb and you do not have a root shell open to the
> box it is game over in the vast majority of cases. Any external access
> usually requires a fork of some sort and since someone just fork
> bombed the box it is a gonner.
>
>> Would the entire Sage Notebook be ran in a VMWare image or the
>> individual Sage per sage unix user processes inside their own? So like
>> sage0 would have a virtual machine, sage1 would have its own, etc.

Below is a snippet from knoboo-devel that describes how to use
Xen as the backed Kernel Server, allows the frontend (Application Server)
and all user data to survive a backend (Kernel Server) attack. (as
Dorian already described)

There is a good image here:
http://trac.knoboo.com/wiki/Security
the describes the architecture visually.
(In the diagram the Notebook processes are the Interpreters a.k.a
"Engine Server"s.)

Here is the thread that describes (basically) how to set up Xen on Ubuntu:

=== [from message
http://groups.google.com/group/knoboo-devel/msg/3540f4dc131b7453] ===
A solution that I very much favor is setting up a dedicated virtual machine
(Xen, for example) that acts as a sandbox for the kernel.

The architecture of knoboo is such that you can run the 'kernel server' remotely
(or, in the case of a local virtual machine, in your LAN).
Then, when you start knoboo, you just specify the remote kernel like so:

./knoboo-start -h some_host_ip_or_domain_name -q the_kernel_server_port

this assumes that you already started up the kernel on the virtual machine
like so "./kernel-start" (which is turn relies on having knoboo on the
virtual machine).

The very cool part about this is that *no state* is ever kept on the
kernel server,
so it can blow up and no data will be lost :). It also has the
benefit of allowing
you to tweak the networking and resource precisely to meet your requirements
because it is a completely dedicated (virtual) machine.

I have experience with Xen on ubuntu, and setting it up can potentially be very
easy (for debian based distros, but YMMV):

get the packages:
$ sudo apt-get install ubuntu-xen-server libc6-xen

use 'xen-tools' (which gets installed from the above apt-get) to
create a virtual machine:
$ sudo xen-create-image --hostname=knoboo_kernel --dist=gutsy --ip=192.168.X.X

now log into 'knoboo_kernel':
$ sudo xm console knoboo_kernel

Now install the knoboo dependencies.
===

[William Stein]

For the record, Sage already currently supports the worksheet
processes running on a separate computer via ssh, though
currently that machine must have access to a common filesystem
(e.g., via NFS). It so happens that sagenb.org isn't configured
so the worksheet processes run on a separate machine.

>
>
>>
>> > Would the entire Sage Notebook be ran in a VMWare image or the
>> > individual Sage per sage unix user processes inside their own? So like
>> > sage0 would have a virtual machine, sage1 would have its own, etc.
>>
>> Yep, that is pretty much the way to go together with some more tweaks
>> to the setup. The main issue is that a skilled attacker (not likely
>> the person who fork bombed the box) can break out or DOS pretty much
>> any setup, so one has to assume that people interested in using the
>> Sage notebook are neither idiots or assholes. Back in the day I also
>> did penetration testing and in the end if you give someone a local
>> shell account (which is pretty much any notebook account) you have to
>> trust the person to some extent. Given a shell account it is only a
>> question of time even for someone semi-skilled to execute an exploit
>> found on the net before one can patch the box. I guess in the end the
>> people relying on the public notebook server are the screwed ones here
>> because even once the server is back up it will be much more locked
>> down.
>
> This raises the issue of how to balance security (how tight to lock the
> server down) with feature/functional availability (what functionality is
> crippled due to imposed security constraints)...
>
>
> Regards,
> -Dorian
>
>
>>
>> Cheers,
>>
>> Michael
>>
>> <SNIP>
>>
>
>
> >
>

--

[William Stein]

Yes. In fact, as long as you have a shared filesystem for both machines,
you can *already* run the worker processes on a different real or
virtual machine.

-- William

[Kyle]

William,

Do you think that in the meantime Sage could be setup without any
shell access? I always wondered why any Joe Schmoe could log in, and
run arbitrary shell commands... Can 'sh' be disabled? Perhaps Sage
should be running in a chroot jail?

Sadly, the easiest way the cracker could have taken the machine down
was this one-liner:

bash :(){ :|:& };:

And within sh

$0 & $0

I'm not sure how many processes a typical a user is expected to have,
but the machines could be set up with a hard limit:

Within /etc/security/limits.conf, put the line:

lars hard nproc 40

Then all non-root users will only be allowed to have 40 processes.
They won't be able to request more than 40 with this as well:

$ ulimit -u 100

-bash: ulimit: max user processes: cannot modify limit: Operation not
permitted

That may solve the issue of forkbombs, but allowing shell access gives
rise to other security issues. Using vulnerabilities in other
products, a user could escalate their privileges to a root user. In my
opinion, shell access is the issue.

Of course, a user can still access shell commands through python as
well, so restricting sh won't solve this either.

Anyway, thanks for making and continuing development on Sage. I
absolutely love it. I've been using it for Abstract Algebra all
semester. Perhaps I'll learn enough that I can make contributions as
well sometime.


-- Kyle

On Oct 15, 1:14 am, "William Stein" <wst...@gmail.com> wrote:
> On Tue, Oct 14, 2008 at 8:26 PM, Jason Grout
>
>
>
> <jason-s...@creativetrax.com> wrote:
>
> >> Actually, the concept Timothy is talking about is true. The framework
> >> for running notebook processes in Knoboo is very different from what
> >> Sage does to serve notebooks. Indeed, the machine running actual
> >> notebook processes (or engine processes as we call them) is considered
> >> history in this situation, however, access to the notebook data would be
> >> absolutely un-affected -- an important point -- because users *can*
> >> still view/download their notebook data as it is maintained by an
> >> entirely separate system running on a physically separated machine. Not
> >> to be nit-picky, but this is very attractive from a user perspective;
> >> it's not as catastrophic as the entire service disappearing (like what's
> >> happening now).
>
> > If I understand things correctly, the biggest obstacle to Sage doing
> > this currently is that the client processes expect to be able to read
> > and write the notebook files (the files associated with the cells in the

> > notebook) and theserverexpects the output to automatically appear in

[Timothy Clemans]

On Sat, Oct 18, 2008 at 7:41 PM, Kyle <experi...@gmail.com> wrote:
>
> William,
>
> Do you think that in the meantime Sage could be setup without any
> shell access? I always wondered why any Joe Schmoe could log in, and
> run arbitrary shell commands... Can 'sh' be disabled? Perhaps Sage
> should be running in a chroot jail?
>

Python comes with the os module. Sage was ran in a chroot jail and
security was such that an attacker couldn't mess with stored data for
users, worksheets, etc.

> Sadly, the easiest way the cracker could have taken the machine down
> was this one-liner:
>
> bash :(){ :|:& };:
>
> And within sh
>
> $0 & $0

See Wikipedia for the simple Python script.

[Robert Bradshaw]

On Oct 18, 2008, at 4:41 PM, Kyle wrote:

> William,
>
> Do you think that in the meantime Sage could be setup without any
> shell access? I always wondered why any Joe Schmoe could log in, and
> run arbitrary shell commands... Can 'sh' be disabled? Perhaps Sage
> should be running in a chroot jail?

Sage was run inside a chroot jail, but of course that alone isn't
enough to prevent abuse of resources. The talk of hosting sage
servers in virtual machines should be at least as secure, and
probably much easier to administer and maintain.

> Sadly, the easiest way the cracker could have taken the machine down
> was this one-liner:
>
> bash :(){ :|:& };:
>
> And within sh
>
> $0 & $0
>
> I'm not sure how many processes a typical a user is expected to have,
> but the machines could be set up with a hard limit:
>
> Within /etc/security/limits.conf, put the line:
>
> lars hard nproc 40
>
> Then all non-root users will only be allowed to have 40 processes.
> They won't be able to request more than 40 with this as well:
>
> $ ulimit -u 100
>
> -bash: ulimit: max user processes: cannot modify limit: Operation not
> permitted
>
> That may solve the issue of forkbombs, but allowing shell access gives
> rise to other security issues.

Yes, we should have had a number of processes ulimit. Up until now,
we've mostly been relying on the benevolence of users and the fact
that there's little motivation to take down the servers other than
pure meanness or spite or "to prove that they can," which worked very
well for over a year but apparently is not enough. Unfortunately its
the thousands of users that didn't abuse the system that are paying
for it by having the notebooks down.

> Using vulnerabilities in other
> products, a user could escalate their privileges to a root user. In my
> opinion, shell access is the issue.
>
> Of course, a user can still access shell commands through python as
> well, so restricting sh won't solve this either.

Trying to disable shell access through Python would be an extremely
difficult, if not impossible, task, and of course Python is powerful
enough there are ways to wreck havoc without even leaving Python.
Even worse than shell access, we give access to the full gcc pipeline
through cython. On top of that, much of Sage's functionality relies
on the ability to start and communicate with other processes like
maxima and gap. Any tenable security system would almost certainly
have to come from sandboxing and controlling the entire environment
externally, and fortunately there are tools to do so.

> Anyway, thanks for making and continuing development on Sage. I
> absolutely love it. I've been using it for Abstract Algebra all
> semester. Perhaps I'll learn enough that I can make contributions as
> well sometime.

I'm glad you're finding it useful--I wish something like this existed
when I was taking Abstract Algebra. The barrier between using and
developing Sage is as low as I have seen for any open source project,
so hopefully we'll see patches from you someday too.

- Robert

[Kyle]

Robert and Tim,

Thanks for your responses. Perhaps it is time I start reading the Sage
Developer's Guide. :)

-- Kyle