[JIRA] (JENKINS-12753) Instant update of plugins doesn't work in nfs directory
alexlehm@gmail.com (JIRA)
--------------------------------------
Summary: Instant update of plugins doesn't work in nfs directory
Key: JENKINS-12753
URL: https://issues.jenkins-ci.org/browse/JENKINS-12753
Project: Jenkins
Issue Type: Bug
Components: plugin
Environment: linux, nfs, jenkins 1.450, any plugin
Reporter: Alex Lehmann
Priority: Minor
Got the question from the irc channel, this was initially discussed on the fosdem jenkins booth.
Updating plugins doesn't work when the jenkins home directory is located on nfs, the failure looks like this
Caused by: java.io.IOException: Unable to delete /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/.nfs00000000000e897d00000096
The .nfs files appear when deleting a file on nfs that is still open, in this case the jar files in the lib directory are still open and cannot be deleted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
kk@kohsuke.org (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158980#comment-158980 ]
Kohsuke Kawaguchi commented on JENKINS-12753:
---------------------------------------------
This appears to indicate that we are trying to overwrite a plugin currently in use, and that won't work regardless, NFS or not.
> Instant update of plugins doesn't work in nfs directory
> -------------------------------------------------------
jenkins-ci.org@michael-prokop.at (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159257#comment-159257 ]
Michael Prokop commented on JENKINS-12753:
------------------------------------------
Being the one who raised this issue on IRC (thanks to Alex for reporting it here): this seems to be a locking issue caused by NFS. I can't reproduce the very same issue without NFS and the open .nfs000000... file handle indicates that there's something going wrong with file deletion.
> Instant update of plugins doesn't work in nfs directory
> -------------------------------------------------------
alexlehm@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159258#comment-159258 ]
Alex Lehmann commented on JENKINS-12753:
----------------------------------------
the way open files are handled is different on local filesystems and nfs, the issue may be nfs specific, however if Kohsuke is right, it doesn't work on local storage either but it doesn't give an obvious error message in that case.
if the plugin is in use, the different jar files for the plugin are all loaded into the jvm and are open (this is visible with lsof), if the plugin is updated, the files are still open even though they are already delete (this is perfectly ok on local filesystems), e.g. like this (this is the output of lsof|grep java.*xunit.*guice as one example file):
before:
{code}
java 1430 lehmann mem REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
java 1430 lehmann 237r REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
{code}
after:
{code}
java 1430 lehmann DEL REG 8,1 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
java 1430 lehmann 237r REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar (deleted)
{code}
the jars will not be reloaded even if they were written to the directory in a new version.
on nfs, open deleted files are not directly possible, so instead each file is changed to a .nfs file which causes the exception when trying to delete the directory, but reloading wouldn't work either if the files could be deleted (or e.g. are moved to another dir and delete later).
When updating a plugin when the directory is on a local filesystem, the update page still says "xunit plugin is already installed. Jenkins needs to be restarted for the update to take effect" with a yellow ball, so the update process notices that it cannot update the files and forces suggests to restart. With nfs he never gets to the stage where the message appears, but the problem is the same in both cases.
> Instant update of plugins doesn't work in nfs directory
> -------------------------------------------------------
alexlehm@gmail.com (JIRA)
[ https://issues.jenkins-ci.org/browse/JENKINS-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=159258#comment-159258 ]
Alex Lehmann edited comment on JENKINS-12753 at 2/19/12 3:59 PM:
-----------------------------------------------------------------
the way open files are handled is different on local filesystems and nfs, the issue may be nfs specific, however if Kohsuke is right, it doesn't work on local storage either but it doesn't give an obvious error message in that case.
if the plugin is in use, the different jar files for the plugin are all loaded into the jvm and are open (this is visible with lsof), if the plugin is updated, the files are still open even though they are already delete (this is perfectly ok on local filesystems), e.g. like this (this is the output of lsof|grep java.*xunit.*guice as one example file):
(removed some columns from the lsof output to fit into the page width)
before:
{code}
mem REG 8,1 807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r REG 8,1 807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
{code}
after:
{code}
DEL REG 8,1 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r REG 8,1 807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar (deleted)
{code}
the jars will not be reloaded even if they were written to the directory in a new version.
on nfs, open deleted files are not directly possible, so instead each file is changed to a .nfs file which causes the exception when trying to delete the directory, but reloading wouldn't work either if the files could be deleted (or e.g. are moved to another dir and delete later).
When updating a plugin when the directory is on a local filesystem, the update page still says "xunit plugin is already installed. Jenkins needs to be restarted for the update to take effect" with a yellow ball, so the update process notices that it cannot update the files and forces suggests to restart. With nfs he never gets to the stage where the message appears, but the problem is the same in both cases.
was (Author: alexlehm):
the way open files are handled is different on local filesystems and nfs, the issue may be nfs specific, however if Kohsuke is right, it doesn't work on local storage either but it doesn't give an obvious error message in that case.
if the plugin is in use, the different jar files for the plugin are all loaded into the jvm and are open (this is visible with lsof), if the plugin is updated, the files are still open even though they are already delete (this is perfectly ok on local filesystems), e.g. like this (this is the output of lsof|grep java.*xunit.*guice as one example file):
before:
{code}
java 1430 lehmann mem REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
java 1430 lehmann 237r REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
{code}
after:
{code}
java 1430 lehmann DEL REG 8,1 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
java 1430 lehmann 237r REG 8,1 807021 4067923 /home/lehmann/.jenkins/plugins/xunit/WEB-INF/lib/guice-2.0.1.jar (deleted)
{code}
the jars will not be reloaded even if they were written to the directory in a new version.
on nfs, open deleted files are not directly possible, so instead each file is changed to a .nfs file which causes the exception when trying to delete the directory, but reloading wouldn't work either if the files could be deleted (or e.g. are moved to another dir and delete later).
When updating a plugin when the directory is on a local filesystem, the update page still says "xunit plugin is already installed. Jenkins needs to be restarted for the update to take effect" with a yellow ball, so the update process notices that it cannot update the files and forces suggests to restart. With nfs he never gets to the stage where the message appears, but the problem is the same in both cases.
> Instant update of plugins doesn't work in nfs directory
> -------------------------------------------------------
jglick@cloudbees.com (JIRA)
|
||||||||||||||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
jglick@cloudbees.com (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
jglick@cloudbees.com (JIRA)
|
|
|||||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
jglick@cloudbees.com (JIRA)
|
|
|
||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
jglick@cloudbees.com (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
scm_issue_link@java.net (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
scm_issue_link@java.net (JIRA)
|
|
|||||||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
daniel@beckweb.net (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
dogfood@java.net (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |
jglick@cloudbees.com (JIRA)
|
|
|||||||
|
This message is automatically generated by JIRA. |
| If you think it was sent incorrectly, please contact your JIRA administrators. |