|
Summary:
========
When Jenkins results are stored on a NAS location, we are seeing very frequent errors (100x/day) when log rotation runs after a job completes. These appear to be a very small (4-second or less) race condition when using NAS storage.
The net effect is that log rotation does not appear to work properly, at least from the Jenkins perspective, resulting in increasing build records over time.
The typical symptom, as seen in the Jenkins console, is:
Mar 15, 2014 10:38:32 PM hudson.model.Run execute
SEVERE: Failed to rotate log
java.io.IOException: Unable to delete /zzz/jenkinsarchive/SomeJob/builds/.2014-03-15_20-32-56/.nfs000000000118864100003187
at hudson.Util.deleteFile(Util.java:254)
at hudson.Util.deleteRecursive(Util.java:301)
at hudson.Util.deleteContentsRecursive(Util.java:203)
at hudson.Util.deleteRecursive(Util.java:300)
at hudson.model.Run.delete(Run.java:1452)
at hudson.tasks.LogRotator.perform(LogRotator.java:124)
at hudson.model.Job.logRotate(Job.java:440)
at hudson.model.Run.execute(Run.java:1739)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
More Details:
==============
We have a good-sized Jenkins install, with several hundred jobs, and due to the archiving size requirements and large # of jobs, we are archiving results to a NAS location.
The Jenkins Build Record Root Directory option is configured to /zzz/jenkinsarchive/${ITEM_FULLNAME}/builds
Where /zzz/jenkinsarchive is a symlink to a mount-point location, such as /mnt/foo.com/export/foo/bar/jenkinsarchive
So records are written to /zzz/jenkinsarchive/SomeJob/builds/... which really evaluates to /mnt/foo.com/export/foo/bar/jenkinsarchive/SomeJob/builds/...
My understanding of .nfs lock files is that they appear on the NAS when a NAS file is unlinked but is still open by a NFS client. (presumably in this case, Jenkins).
Using lsof -n -P -N -u someUser -a I can see that normally we have 2 files in-use on the NAS – one at
jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/log
and one at
jenkinsarchive/SomeJob/builds/2014-03-15_19-19-58/timestamper/timestamps
but for very short periods of time, when log rotation runs, I see a files at a path like
jenkinsarchive/SomeJob/builds/.2014-03-15_19-19-58/.nfs000000000009e0cc000003c4
exist. It appears to only exist for a couple seconds.
So it seems that something in Jenkins is keeping access open to these files, which then (per NFS semantics) is preventing deletion at the time of the stack trace above.
|
Major