Help needed with debugging a "[Errno 13] Permission denied" issue

[Yuke Liao]

Hi OS experts,

One of our Tast tests (lacros.Basic) is flaky, and the way it works (code search link) is that it invokes a Python script, which then launches a binary, but for some reason, launching the binary flakily fails due to "[Errno 13] Permission denied" error.

This file contains an example of the detailed error log:

Traceback (most recent call last):
  File "/usr/local/bin/mojo_connection_lacros_launcher.py", line 149, in <module>
    sys.exit(Main())
  File "/usr/local/bin/mojo_connection_lacros_launcher.py", line 143, in Main
    proc = subprocess.Popen(cmd, pass_fds=pass_fds)
  File "/usr/local/lib64/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib64/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/mnt/stateful_partition/lacros_test_artifacts/lacros_binary/chrome'

At first, we thought it might be due to that the permission on the file is not set correctly, so we also dumped the results of a `ls -l` command to double check:

total 212240
drwx------.  2 root   root       4096 May 20 11:19 792198682
drwx------. 25 root   root       4096 May 20 11:19 979199975
drwxr-xr-x.  3 311298 89939      4096 Apr 28 20:27 WidevineCdm
-rwxr-xr-x.  1 311298 89939 172938568 Apr 28 21:22 chrome
-rw-r--r--.  1 311298 89939    849663 Apr 28 21:22 chrome_100_percent.pak
-rw-r--r--.  1 311298 89939   1487299 Apr 28 21:22 chrome_200_percent.pak
-rwxr-xr-x.  1 311298 89939   4649688 Apr 28 21:22 crashpad_handler
-rw-r--r--.  1 311298 89939   4825605 Apr 28 21:22 headless_lib.pak
-rw-r--r--.  1 311298 89939  10474512 Apr 28 21:22 icudtl.dat
drwxr-xr-x.  2 311298 89939      4096 Apr 28 20:37 locales
-rw-r--r--.  1 311298 89939        77 Apr 28 21:22 metadata.json
-rwxr-xr-x.  1 311298 89939  11103040 Apr 28 21:22 nacl_helper
-rwxr-xr-x.  1 311298 89939   3868208 Apr 28 21:22 nacl_irt_x86_64.nexe
-rw-r--r--.  1 311298 89939   7034097 Apr 28 21:22 resources.pak
-rw-r--r--.  1 311298 89939     48908 Apr 28 21:22 snapshot_blob.bin
drwxr-xr-x.  2 311298 89939      4096 Apr 28 20:27 swiftshader

Which shows that the binary can be executed by anyone.

At this point, we don't know how to move forward, does anyone have any suggestions on how to debug this type of issue?

Thanks,

Yuke

[Dmitry Torokhov]

I am sure the stateful partition is mounted as 'noexec' as part of hardening efforts, so no executables from it can be started.

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
https://groups.google.com/a/chromium.org/group/chromium-os-dev

[Yuke Liao]

On Fri, May 21, 2021 at 10:34 AM Dmitry Torokhov <dt...@chromium.org> wrote:

I am sure the stateful partition is mounted as 'noexec' as part of hardening efforts, so no executables from it can be started.

Thanks Dmitry, I didn't think of that because it's a flaky failure, if there is a 'noexec' hardening effort, I'd expect the behavior to be consistent?

[Yuke Liao]

On Fri, May 21, 2021 at 10:38 AM Yuke Liao <liao...@chromium.org> wrote:

On Fri, May 21, 2021 at 10:34 AM Dmitry Torokhov <dt...@chromium.org> wrote:

I am sure the stateful partition is mounted as 'noexec' as part of hardening efforts, so no executables from it can be started.

Thanks Dmitry, I didn't think of that because it's a flaky failure, if there is a 'noexec' hardening effort, I'd expect the behavior to be consistent?

 

Actually, we also dumped the results of `mount` to here, it seems that there is no 'noexec' bit on it:

/dev/nvme0n1p1 on /mnt/stateful_partition type ext4 (rw,nosuid,nodev,noatime,seclabel,resgid=20119,commit=600)

[Mike Frysinger]

you must not exec anything out of that dir. only /usr/local may be executed out of.  you shouldn't be using /mnt/stateful_partition in general either as it's not for random things to scribble in like this.

so while it might be an interesting puzzle to solve, what you're doing is not supported and needs changing.

-mike

[Dmitry Torokhov]

It looks like src/chromiumos/tast/local/lacros/launcher/util.go remounts stateful removing "noexec" flag, I wonder if that is failing or happens at the wrong time. In any case, as Mike said, we should not be altering mount flags on stateful.

[Mike Frysinger]

thanks for pointing it out.  that code must be deleted.  we never want random tests remounting paths like this because it completely invalidates any tests that run on the system afterwards.

-mike

[Yuke Liao]

Got it, thank you for the pointers Dmitry and Mike! We'll re-write the code using /usr/local.

[Hidehiko Abe]

As for we can move the lacros to wherever appropriately.

Just to be safe, IIUC, /usr/local is the directory several binaries are installed onto on build time.

Is it ok to modify on runtime?

Stepping back, as /mnt/stateful_partition is remounted with exec on runtime, and actually the test log showed it worked,

it does not explain why EPERM is observed. So, I think, even if we move the directory, the permission issue would be still there?

Any thoughts on that?

- hidehiko

[Dmitry Torokhov]

Did it really work? I see in https://storage.cloud.google.com/chromeos-autotest-results/swarming-53b336d5972c7d10/autoserv_test/tast/results/tests/lacros.Basic/mount.txt:

/dev/mmcblk0p1 on /mnt/stateful_partition type ext4 (rw,nosuid,nodev,noexec,noatime,seclabel,resgid=20119,commit=600)

Or was this snapshot taken before running the test?

Thanks,

Dmitry

[Mike Frysinger]

/usr/local is writable & executable and meant for use for testing like this.  it shouldn't run into EPERM issues.

any test that changes mount flags on standard mount points (like noexec on /mnt/stateful_partition) is corrupting the DUT.  we'd deploy a policy that'd block this completely at runtime if it were possible.

-mike

[Hidehiko Abe]

Dmitry, thanks for checking, and I think it's a different case from what I looked at.
Could you share the link to log viewer page (like what Yuke shared)?

BTW, at least, yuke's log says /mnt/stateful_partition does not have noexec.

On Mon, May 24, 2021 at 2:32 PM Mike Frysinger <vap...@chromium.org> wrote:

/usr/local is writable & executable and meant for use for testing like this.  it shouldn't run into EPERM issues.

any test that changes mount flags on standard mount points (like noexec on /mnt/stateful_partition) is corrupting the DUT.  we'd deploy a policy that'd block this completely at runtime if it were possible.

Thanks for the info.

FYI: policies need to be careful, IIUC. deploy_chrome copies files under stateful partition, then bind mount to /opt/google/chrome specifically if the size is huge.

The feature (supporting large size chrome) is important, and should be kept working.

THanks,

- hidehiko

[Mike Frysinger]

thanks for pointing that out.  deploy_chrome is broken too.  filed https://crbug.com/1212649 to fix it.

-mike

[Dmitry Torokhov]

Hidehiko,

The build link is from the log viewer that Yuke shared. The entry is

R93-13988.0.0

celes

chromeos4-row12-rack4-host10

Failed to launch lacros-chrome: failed to wait for lacros-chrome window to be visible: context deadline exceeded; last error follows: no window satisfies the condition

and was the very top of the failures last night. Logs are here: https://stainless.corp.google.com/browse/chromeos-autotest-results/swarming-53b336d5972c7d10/autoserv_test/tast/results/tests/lacros.Basic/

Thanks,

Dmitry