[PSA] disabling shell script usage on noexec mounts redux

[Mike Frysinger]

background thread here:

https://groups.google.com/a/chromium.org/d/topic/chromium-os-dev/T99xHSpw27E/discussion

to address the concerns of breaking scripts before people have a chance to fix things, i'm taking a more incremental approach.  first, we'll focus on dash and leave bash alone for now.  second, instead of crashing and aborting execution, all code paths issue a warning to stderr, and will create crash reports in the background.

so the new output might look something like:

# printf '#!/bin/sh\necho hi\n' > /var/test.sh

# chmod a+rx /var/test.sh

# dash /var/test.sh

dash: 0: /var/test.sh: warning: script from noexec source

hi

and we have crash reports generated in /var/spool/crash/ for dash.

this should allow us to roll it out now and start gathering data in canary channel.  if crash rates go up, we can see which script is causing problems by checking the command line.

we'll probably stage this out over a few releases to be extra cautious.

-mike

[David Schneider]

Thanks for CCing me :)

crouton uses /bin/sh everywhere (which of course is symlinked to dash), so I think you'll be seeing crash reports from crouton users (about 4% of canary users).  Is that what you were intending?

From https://goo.gl/#analytics/goo.gl/fd3zc/all_time (which will no longer be accessible after the 30th, it seems), the top sources that link to the crouton downloader shorturl are:

31.8% - github.com (the README; phew)

29.4% - unknown (probably people curling it)

12.2% - others (thanks, Google)

10.3% - lifehacker.com

6% - howtogeek.com

4.4% - linux.com

3.1% - davebennett.tech/davebennett.tv [these seem dead]

1.1% - google.com (wat)

0.9% - webupd8.org

0.8% - tutorials.ubuntu.com

(out of ~2.3M)

So it looks like we need to do the following (please provide your thoughts):

* the installer/crouton indirect downloader (everyone's entryway into the installer) will need to either remount exec /tmp or bind-mount the cache subdirectory and execify it before executing the installer bundle.

* the README needs to be updated with a big warning about needing to install crouton to /usr/local before running.  The question is what the canonical commands should be that everyone inevitably copy-pastes?  /usr/local/bin doesn't exist by default, so creating it is an additional step.  We could also just instruct to dump it in /usr/local, but then you'd have to specify the full path when running it.  Alternatively, we could tell people to remount their profile exec, but that also seems like a bad idea (especially since you'll have to do it again when updating crouton later).   I'm assuming you're going to try to break curl xyz | sh so not suggesting that as a potential fix :)

* the README needs to be updated to discuss running crouton on external media, which will be pretty broken.  I think the best option is just tell people to execify the entire mount, which is a bit of a shame, since crouton was designed to avoid doing precisely that.  The point of this approach is to allow people to launch their chroot off of a USB stick without touching the stateful partition at all.

* Once we decide on the canonical commands, we need to notify the source of 25.5% of direct links (listed above) to update their tutorials.

You may also want to reach out to https://github.com/reynhout/chrx, since they're going to have the same issues with out-of-date READMEs and tutorials.

[Mike Frysinger]

On Mon, Mar 11, 2019 at 3:33 PM David Schneider <dnsc...@chromium.org> wrote:

crouton uses /bin/sh everywhere (which of course is symlinked to dash), so I think you'll be seeing crash reports from crouton users (about 4% of canary users).  Is that what you were intending?

i think there might be some confusion here.  if you have a file with a shebang of #!/bin/sh, and that file is marked +x, and it is executed directly (e.g. ./foo.sh), then nothing is changing.  in order to execute a script like that, it must already be on a exec mount point, and we have no plans on changing that.

we're talking about cases where you save the file to a noexec mount (e.g. the Downloads directory), and then manually run `sh foo.sh` on that file.  the user will see a warning that it's no longer supported, and it'll generate a crash report.  getting details on who/what all exactly is affected is intended, even if it means we gather crashes from use cases that we don't intend on "fixing".

So it looks like we need to do the following (please provide your thoughts):

* the installer/crouton indirect downloader (everyone's entryway into the installer) will need to either remount exec /tmp or bind-mount the cache subdirectory and execify it before executing the installer bundle.

would strongly recommend not changing /tmp.  making that more insecure by default will affect the rest of the system.

i'm not sure cache subdirectory you're referring to.  i assume that's some crouton thing as CrOS doesn't have that.

* the README needs to be updated with a big warning about needing to install crouton to /usr/local before running.  The question is what the canonical commands should be that everyone inevitably copy-pastes?  /usr/local/bin doesn't exist by default, so creating it is an additional step.  We could also just instruct to dump it in /usr/local, but then you'd have to specify the full path when running it.

i assume you have a "one line" that people will be blindly copying & pasting right ?  so it doesn't need to be a single command.

  cd /usr/local && curl ... -o ... && sh ... --install

based on the --install flag, you could have it relocate $0 to bin/ and chmod as makes sense.

 

Alternatively, we could tell people to remount their profile exec, but that also seems like a bad idea (especially since you'll have to do it again when updating crouton later).

right, like /tmp, we don't want to encourage things that impact the security of the overall system.

I'm assuming you're going to try to break curl xyz | sh so not suggesting that as a potential fix :)

i don't think it breaks or warns right now, but we do want to break that eventually.  it'll probably be harder to workout of the system though as upstart relies on it currently.

* the README needs to be updated to discuss running crouton on external media, which will be pretty broken.  I think the best option is just tell people to execify the entire mount, which is a bit of a shame, since crouton was designed to avoid doing precisely that.  The point of this approach is to allow people to launch their chroot off of a USB stick without touching the stateful partition at all.

i think that about covers it.  people can either manually remount -o exec, or they can copy the launcher to /usr/local/bin/.  i get that it's kind of nice to have a usb stick that is completely independent and doesn't rely on /usr/local content, but it seems like the number of users of this esoteric flow doesn't justify keeping the security sacrifice.  i'm not coming up with a better answer atm.

* Once we decide on the canonical commands, we need to notify the source of 25.5% of direct links (listed above) to update their tutorials.

You may also want to reach out to https://github.com/reynhout/chrx, since they're going to have the same issues with out-of-date READMEs and tutorials.

considering the usage numbers (no updates in months, 4 contributors, <200 total commits), seems like the existing notification flow should be sufficient.  either they'll get the message and update, or they won't because no one uses it anymore ;).

we're only talking about making it warn for now, and it'll be staged over a release or two, so it most likely will be O(months) between the warnings showing up and then the calls being fatal.

-mike

[David Schneider]

On Mon, Mar 11, 2019 at 4:05 PM Mike Frysinger <vap...@chromium.org> wrote:

On Mon, Mar 11, 2019 at 3:33 PM David Schneider <dnsc...@chromium.org> wrote:

crouton uses /bin/sh everywhere (which of course is symlinked to dash), so I think you'll be seeing crash reports from crouton users (about 4% of canary users).  Is that what you were intending?

i think there might be some confusion here.  if you have a file with a shebang of #!/bin/sh, and that file is marked +x, and it is executed directly (e.g. ./foo.sh), then nothing is changing.  in order to execute a script like that, it must already be on a exec mount point, and we have no plans on changing that.

we're talking about cases where you save the file to a noexec mount (e.g. the Downloads directory), and then manually run `sh foo.sh` on that file.  the user will see a warning that it's no longer supported, and it'll generate a crash report.  getting details on who/what all exactly is affected is intended, even if it means we gather crashes from use cases that we don't intend on "fixing".

Well, we're designed around dash, and all the instructions say to "sh ~/Downloads/crouton" (since of course Downloads by default is noexec).  So no confusion here; you'll definitely get crash reports and that's OK.

 

So it looks like we need to do the following (please provide your thoughts):

* the installer/crouton indirect downloader (everyone's entryway into the installer) will need to either remount exec /tmp or bind-mount the cache subdirectory and execify it before executing the installer bundle.

would strongly recommend not changing /tmp.  making that more insecure by default will affect the rest of the system.

i'm not sure cache subdirectory you're referring to.  i assume that's some crouton thing as CrOS doesn't have that.

Correct.  The curl'd or web-downloaded installer subsequently curls the actual installer bundle and stores it in /tmp/crouton-cache (or whatever).  This is to deal with the fact that people were downloading the crouton installer once, holding onto it, and using it to "update" their chroots months later.

So now we would make the crouton-cache directory, bind-mount it over itself, then mark that exec.

 

* the README needs to be updated with a big warning about needing to install crouton to /usr/local before running.  The question is what the canonical commands should be that everyone inevitably copy-pastes?  /usr/local/bin doesn't exist by default, so creating it is an additional step.  We could also just instruct to dump it in /usr/local, but then you'd have to specify the full path when running it.

i assume you have a "one line" that people will be blindly copying & pasting right ?  so it doesn't need to be a single command.

  cd /usr/local && curl ... -o ... && sh ... --install

based on the --install flag, you could have it relocate $0 to bin/ and chmod as makes sense.

Actually, we've avoided multi-part one-liners.  The instructions (feel free to read them) say to download it by clicking a link and run the result with sudo sh.  Yes, we could instead say to cd /usr/local && sudo curl ... -o ... && sudo sh etc etc etc but we were trying to make the tool approachable so people could avoid blindly copy-pasting instructions and actually understand what they're running.  This saves us somewhat, because tutorials end up following the same approach -- instead of magic one-liners (like the evil curl ... | sudo sh) which will get super broken, they're also saying to download the file by clicking the link and execute it with sh.

I'm leaning towards telling people to

sudo install -Dt /usr/local/bin -m 755 ~/Downloads/crouton

and then just sudo crouton from then on.

 

 

Alternatively, we could tell people to remount their profile exec, but that also seems like a bad idea (especially since you'll have to do it again when updating crouton later).

right, like /tmp, we don't want to encourage things that impact the security of the overall system.

I'm assuming you're going to try to break curl xyz | sh so not suggesting that as a potential fix :)

i don't think it breaks or warns right now, but we do want to break that eventually.  it'll probably be harder to workout of the system though as upstart relies on it currently. 

* the README needs to be updated to discuss running crouton on external media, which will be pretty broken.  I think the best option is just tell people to execify the entire mount, which is a bit of a shame, since crouton was designed to avoid doing precisely that.  The point of this approach is to allow people to launch their chroot off of a USB stick without touching the stateful partition at all.

i think that about covers it.  people can either manually remount -o exec, or they can copy the launcher to /usr/local/bin/.  i get that it's kind of nice to have a usb stick that is completely independent and doesn't rely on /usr/local content, but it seems like the number of users of this esoteric flow doesn't justify keeping the security sacrifice.  i'm not coming up with a better answer atm.

Sounds like in the medium term, people who really want to leave no trace on stateful could do curl -fL https://goo.gl/fd3zc | sudo sh -s -- options for crouton installer

I might bury that in the README somewhere with a disclaimer (may fail in the future) in the hopes that most people won't use that.

* Once we decide on the canonical commands, we need to notify the source of 25.5% of direct links (listed above) to update their tutorials.

You may also want to reach out to https://github.com/reynhout/chrx, since they're going to have the same issues with out-of-date READMEs and tutorials.

considering the usage numbers (no updates in months, 4 contributors, <200 total commits), seems like the existing notification flow should be sufficient.  either they'll get the message and update, or they won't because no one uses it anymore ;).

I disagree with your analysis of those metrics (note that they only need to update chrx when ChromeOS makes significant changes or they want to add HWIDs, and most of the major work is in GalliumOS), and I doubt anyone using chrx is on canary, or really even beta.  I'll file an issue linking to this conversation if you can't be bothered.

 

we're only talking about making it warn for now, and it'll be staged over a release or two, so it most likely will be O(months) between the warnings showing up and then the calls being fatal.

I do appreciate the slow roll-out; hopefully we can get the tutorials updated ahead of the enforcement.

-David

[David Riley]

Provisioning relies (relied?) on executing bash scripts out of /tmp via something like: curl >/tmp/quick-provision && bash /tmp/quick-provision.

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-dev

[Mike Frysinger]

test images and their (mis)use of /tmp is being tracked separately.  test images currently remount /tmp as exec so that won't be impacted by this change.

-mike

[Mike Frysinger]

the CQ has accepted the dash changes now.  thank you Mr. CQ.

i would not be surprised if crash rates start going up for M75, but they will be "advisory" crashes in that they shouldn't actually break the end user's system.  we shall see!

if someone needs to revert this for some reason, here's the two CLs:

  https://crrev.com/c/749524

  https://crrev.com/c/749318

-mike

[Mike Frysinger]

i've written up some docs too:

  https://chromium.googlesource.com/chromiumos/docs/+/master/security/noexec_shell_scripts.md

i'll update the dash code so that when it emits a warning or error, it'll include that link

-mike

On Fri, Mar 8, 2019 at 11:48 PM Mike Frysinger <vap...@chromium.org> wrote:

[Mike Frysinger]

monitoring crash rates for dash showed that it was all coming from users running custom scripts.  so that's good?

i've landed changes to dash for R82 so that the noexec check is now a hard failure under dash.  people will still see the error message pointing to the docs, but the script will stop executing at that point.

-mike

[Miriam Zimmerman]

This seems to have broken cros flash; I get the following error:

sh: 0: Refusing to exec /mnt/stateful_partition/cros-flash/tmp.7qAk5ltRCU/stateful_update from noexec mount; see https://chromium.googlesource.com/chromiumos/docs/+/master/security/noexec_shell_scripts.md

Full command that I'm running is cros flash --clobber-stateful --disable-rootfs-verification ssh://$IP latest --debug 

Full command output attached.

[Mike Frysinger]

https://crrev.com/c/2036452 is in the CQ to fix it.  cros flash has been generating crashes all along, but i guess people didn't notice :).

-mike

[Arthur Bowers]

Hi All,

I'm not sure if this is the best/correct place to post this, but I'm pretty sure someone here can help me...

I recently got a Chromebook 4 and I want to set it up for development, mostly CLI and vim. I would like it running locally, using alacritty as the terminal.

I am running into issues getting anything installed. When I go shell and then go to run the command for xcfe I get the error about noexec scripts running from unmounted location.

Same when trying to download alacritty.

Can't find apt, pkg...

I know enough about linux to know that it is not letting me run a script from an unmounted source for safety reasons, but I don't know enough about noexec on how to fix this. I've tried reading into the documentation and following tutorials on how to get a chromebook running for development but no one else seems to get stuck at this step...I do have it in developer mode by the way.

If anyone knows of, or could ELI5 that would be greatly appriciated. Was looking forward to having a lean light dev machine, but now I realise it's all tied to the web, the cli is not so fast...

Thanks again all, and interesting read nonetheless.

[Mike Frysinger]

i'm assuming you're using crouton.  please see the last question in the FAQ:
https://chromium.googlesource.com/chromiumos/docs/+/master/security/noexec_shell_scripts.md#How-to-run-crouton

-mike

--