Jenkins stash failed after npm install

[murty.de...@8x8.com]

During my build process, I run NPM install among other tools, before stashing the contents of the working directory for later stages. However, we've noticed that when NPM install is used, the stash at the end of this stage throws a Java I/O exception, saying that the archive contains unclosed entries. Disabling the NPM install stops this issue from occurring. Would appreciate any inputs on this as we are stuck with this over the last one week trying various options.  The I/O exception is shown below: 

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to  (sir-r9rg4htg)
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1089)
		at hudson.FilePath.act(FilePath.java:1078)
		at hudson.FilePath.archive(FilePath.java:481)
		at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:107)
		at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:113)
		at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:101)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
		at hudson.security.ACL.impersonate(ACL.java:290)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.io.IOException: This archives contains unclosed entries.
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225)
	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241)
	at hudson.util.io.TarArchiver.close(TarArchiver.java:127)
	at hudson.FilePath$Archive.invoke(FilePath.java:498)
	at hudson.FilePath$Archive.invoke(FilePath.java:483)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3082)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

[murty.de...@8x8.com]

Can someone please help me with this issue?  Completely blocked on this with no clues at this point.

[Slide]

Are you on Windows?

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6f2615e7-1662-4a64-8ecc-34ecdddf7e86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[murty.de...@8x8.com]

No.  We are using CentOS 7

[Slide]

You could add an lsof or similar before the stash to see if there are still open files in that directory. This would mean that NPM is staying resident somehow I would think.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/394b50b9-cb57-4358-89d9-9d7472c2bb19%40googlegroups.com.

[murty.de...@8x8.com]

Thank you.  We already tried that option and we couldn't extract any clues out of that.  Here is the lsof output in the jenkins console, where everything is pointing to the Jenkins scrips rather than the npm install:

After npm install
+ cd ..
+ tee processids.txt
++ pwd
+ lsof +D /home/ec2-user/workspace/XXXX/source
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
sh        305 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
sh        311 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
script.sh 312 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
sleep     410 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
lsof      411 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
tee       412 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
lsof      414 root  cwd    DIR  259,3     4096 3551849 /home/ec2-user/workspace/XXXX/source
+ sh printprocessnames.sh processids.txt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
PID
UID        PID  PPID  C STIME TTY          TIME CMD
root       420   415  0 20:32 ?        00:00:00 grep PID
sh 305 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
305
root       305     0  0 20:31 ?        00:00:00 sh -c { while [ -d '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917' -a \! -f '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt' ]; do touch '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt'; sleep 3; done } & jsc=durable-bed9bdd6353a4a7353f5c0d5c795cf28; JENKINS_SERVER_COOKIE=$jsc '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/script.sh' > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt' 2>&1; echo $? > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp'; mv '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp' '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt'; wait
root       311   305  0 20:31 ?        00:00:00 sh -c { while [ -d '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917' -a \! -f '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt' ]; do touch '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt'; sleep 3; done } & jsc=durable-bed9bdd6353a4a7353f5c0d5c795cf28; JENKINS_SERVER_COOKIE=$jsc '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/script.sh' > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt' 2>&1; echo $? > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp'; mv '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp' '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt'; wait
root       312   305  0 20:31 ?        00:00:00 /bin/sh -xe /home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/script.sh
root       425   415  0 20:32 ?        00:00:00 grep 305
sh 311 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
311
root       311   305  0 20:31 ?        00:00:00 sh -c { while [ -d '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917' -a \! -f '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt' ]; do touch '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt'; sleep 3; done } & jsc=durable-bed9bdd6353a4a7353f5c0d5c795cf28; JENKINS_SERVER_COOKIE=$jsc '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/script.sh' > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-log.txt' 2>&1; echo $? > '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp'; mv '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt.tmp' '/home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/jenkins-result.txt'; wait
root       410   311  0 20:32 ?        00:00:00 sleep 3
root       430   415  0 20:32 ?        00:00:00 grep 311
script.sh 312 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
312
root       312   305  0 20:31 ?        00:00:00 /bin/sh -xe /home/ec2-user/workspace/XXXX/source@tmp/durable-95723917/script.sh
root       415   312  0 20:32 ?        00:00:00 sh printprocessnames.sh processids.txt
root       435   415  0 20:32 ?        00:00:00 grep 312
sleep 410 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
410
root       410   311  0 20:32 ?        00:00:00 sleep 3
root       440   415  0 20:32 ?        00:00:00 grep 410
lsof 411 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
411
root       445   415  0 20:32 ?        00:00:00 grep 411
tee 412 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
412
root       450   415  0 20:32 ?        00:00:00 grep 412
lsof 414 root cwd DIR 259,3 4096 3551849 /home/ec2-user/workspace/XXXX/source
414
root       455   415  0 20:32 ?        00:00:00 grep 414
[Pipeline] }
[Pipeline] // dir
[Pipeline] stash
Warning: overwriting stash ‘workspace’
Post stage
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] updateGitlabCommitStatus
[Pipeline] }
$ docker stop --time=1 344055f88b684753195b4d796a74250c5df37b48baec6e37dceead15e1464f71
$ docker rm -f 344055f88b684753195b4d796a74250c5df37b48baec6e37dceead15e1464f71
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Security)
Stage 'Security' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Publish)
Stage 'Publish' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Acceptance)
Stage 'Acceptance' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Staging)
Stage 'Staging' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Canary)
Stage 'Canary' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Production)
Stage 'Production' skipped due to earlier failure(s)

My guess from the above is, we were not able to catch some process in the lsof, that's holding the resources as part of the npm install.  Any clues???

Regards,

Murty.