permission denied to file (afs problem??)

[nbecker]

Several jobs died with 'Permission denied' after having been running for hours.  These were caused by

 cloud.files.putf

Traceback (most recent call last):

  File "/root/.local/lib/python2.7/site-packages/cloudserver/workers/employee/child.py", line 583, in run

  File "./run-cloud-11121401.py", line 23, in run_test

  File "test_8psk_cancel_lms.py", line 919, in run_line

  File "test_8psk_cancel_lms.py", line 689, in run

  File "/afs/picloud.com/user/000/001/1440/restricted/ap_repos/repo_1478_nbecker6.hughes.com/86d492bc020299ec6b9710fa61bd5b6b8c1bacca/logger.py", line 16, in reopen

OSError: [Errno 13] Permission denied

[Aaron Staley]

Hello,

Our apologies.  We use openafs as our distributed file system.  Security is controlled by a "ticket system".  We issue you a ticket to your job right before it begins.  Unfortunately, the tickets expire in 10 hours.  After your job passed the 10 hour mark, it no longer had permission to read the file system - which resulted in the error you saw.

The underlying issue will be fixed in our next system update; until then, please be sure to catch and ignore any OSError exceptions.  We also recommend keeping job times to no more than a few hours if possible.

Finally, we are crediting the errored jobs' computation time to you.  Please file a support ticket if you wish for us to restart the job.

Regards,

Aaron Staley

[Neal Becker]

I can ignore OSError, but is this just a transient problem, or will a long running process continue to get OSError after 10 hours?

[Aaron Staley]

We won't be able to deploy our system update for at least a week, possibly longer.  We will notify you when the issue is resolved.