Issue 526340 in chromium: Some autotest results directories end up owned by root, break gs_offloader

[chro...@googlecode.com]

Status: Available
Owner: ----
Labels: Type-Bug Pri-1 OS-Chrome Hardware-Lab

New issue 526340 by jrbarne...@chromium.org: Some autotest results
directories end up owned by root, break gs_offloader
https://code.google.com/p/chromium/issues/detail?id=526340

Looking at some drones/shards, I see that the gs_offloader upstart
job (for offloading regular test jobs) seems to be restarting rather
frequently.

Looking in (for instance) /var/log/gs_offloader.1, I see errors like this:
Traceback (most recent call last):
File "/usr/local/autotest/site-packages/chromite/lib/parallel.py", line
601, in TaskRunner
task(*x, **task_kwargs)
File "/usr/local/autotest/site-packages/statsd/timer.py", line 95, in
_decorator
return function(*args, **kwargs)
File "/usr/local/autotest/site_utils/gs_offloader.py", line 271, in
offload_dir
shutil.rmtree(dir_entry)
File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/usr/lib/python2.7/shutil.py", line 247, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "/usr/lib/python2.7/shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission
denied: '39617385-chromeos-test/chromeos4-row3-rack3-host5/telemetry_Benchmarks.smoothness.tough_canvas_cases/sysinfo/added_packages'

Checking the directory in question, you can see that it's owned by root.
I assume that this is a side-effect of SSP.

Regardless of the root cause we need these things (possibly entailing
more bugs):
1) Fix gs_offloader not to die when shutil.rmtree() fails. Instead, the
error should be queued to generate e-mail notification.
2) Whatever is creating files owned by root should be beaten about
the head and shoulders until it agrees that the files will be owned
by chromeos-test instead.

We also need to do a sweep of all the drones and shards to clean up
the current messes.


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[chro...@googlecode.com]

Updates:
Cc: dgar...@chromium.org d...@chromium.org

Comment #1 on issue 526340 by jrbarne...@chromium.org: Some autotest

results directories end up owned by root, break gs_offloader
https://code.google.com/p/chromium/issues/detail?id=526340

(No comment was entered for this change.)

[chro...@googlecode.com]

Comment #2 on issue 526340 by d...@chromium.org: Some autotest results

directories end up owned by root, break gs_offloader
https://code.google.com/p/chromium/issues/detail?id=526340

That particular test failed with an odd error:

http://cautotest/afe/#tab_id=view_job&object_id=39617385
08/20 12:13:21.736 ERROR| traceback:0013| * Command:
08/20 12:13:21.736 ERROR| traceback:0013| sudo -n chown -R
99158 "/usr/local/autotest/results/39617385-chromeos-
08/20 12:13:21.736 ERROR| traceback:0013|
test/chromeos4-row3-rack3-host5"
08/20 12:13:21.737 ERROR| traceback:0013| Exit status: -11

Not sure why chown command can fail, and no idea what exit status -11
means. Any help?

[chro...@googlecode.com]

Comment #3 on issue 526340 by jrbarne...@chromium.org: Some autotest

results directories end up owned by root, break gs_offloader
https://code.google.com/p/chromium/issues/detail?id=526340

Exit status -11 means the process died from SIGSEGV...

[chro...@googlecode.com]

Comment #4 on issue 526340 by d...@chromium.org: Some autotest results

directories end up owned by root, break gs_offloader
https://code.google.com/p/chromium/issues/detail?id=526340

invalid memory access (segmentation fault), the container was destroyed
before that call was made, so any mounted directory should have been
unmounted. I might need to do some logging in that method to track the
error.