Build failed in Jenkins: osv-build #1063

[jen...@scylladb.com]

See <https://jenkins.scylladb.com/job/osv-build/1063/display/redirect>

Changes:


------------------------------------------
Started by an SCM change
Running as SYSTEM
[EnvInject] - Loading node environment variables.
[EnvInject] - Preparing an environment for the build.
[EnvInject] - Keeping Jenkins system variables.
[EnvInject] - Keeping Jenkins build variables.
[EnvInject] - Injecting as environment variables the properties content
http_proxy=10.0.0.4:3128
PATH=/usr/local/bin:/usr/bin:/usr/sbin

[EnvInject] - Variables injected successfully.
[EnvInject] - Injecting contributions.
Building remotely on muninn (pack-no-dg packager-amd64 artifact-testers packager pack-no-sif) in workspace <https://jenkins.scylladb.com/job/osv-build/ws/>
[EnvInject] - Unset unresolved 'BASH_FUNC_which%%' variable.
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to muninn
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.mkdirs(FilePath.java:1374)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1209)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
at hudson.model.Run.execute(Run.java:1896)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
java.nio.file.FileSystemException: <https://jenkins.scylladb.com/job/osv-build/ws/>: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
at java.nio.file.Files.createDirectory(Files.java:690)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
at java.nio.file.Files.createDirectories(Files.java:783)
at hudson.FilePath.mkdirs(FilePath.java:3609)
at hudson.FilePath.access$1100(FilePath.java:212)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1384)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1379)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3487)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
Retrying after 10 seconds
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to muninn
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.mkdirs(FilePath.java:1374)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1209)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
at hudson.model.Run.execute(Run.java:1896)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
java.nio.file.FileSystemException: <https://jenkins.scylladb.com/job/osv-build/ws/>: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
at java.nio.file.Files.createDirectory(Files.java:690)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
at java.nio.file.Files.createDirectories(Files.java:783)
at hudson.FilePath.mkdirs(FilePath.java:3609)
at hudson.FilePath.access$1100(FilePath.java:212)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1384)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1379)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3487)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
Retrying after 10 seconds
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to muninn
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.mkdirs(FilePath.java:1374)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1209)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
at hudson.model.Run.execute(Run.java:1896)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
java.nio.file.FileSystemException: <https://jenkins.scylladb.com/job/osv-build/ws/>: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
at java.nio.file.Files.createDirectory(Files.java:690)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
at java.nio.file.Files.createDirectories(Files.java:783)
at hudson.FilePath.mkdirs(FilePath.java:3609)
at hudson.FilePath.access$1100(FilePath.java:212)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1384)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1379)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3487)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
Retrying after 10 seconds
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to muninn
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.act(FilePath.java:1183)
at hudson.FilePath.mkdirs(FilePath.java:1374)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1209)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
at hudson.model.Run.execute(Run.java:1896)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
java.nio.file.FileSystemException: <https://jenkins.scylladb.com/job/osv-build/ws/>: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
at java.nio.file.Files.createDirectory(Files.java:690)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
at java.nio.file.Files.createDirectories(Files.java:783)
at hudson.FilePath.mkdirs(FilePath.java:3609)
at hudson.FilePath.access$1100(FilePath.java:212)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1384)
at hudson.FilePath$Mkdirs.invoke(FilePath.java:1379)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3487)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
IRC notifier plugin: Sending notification to: #osv

[jen...@scylladb.com]

See <https://jenkins.scylladb.com/job/osv-build/1064/display/redirect>

[jen...@scylladb.com]

See <https://jenkins.scylladb.com/job/osv-build/1065/display/redirect>

[Nadav Har'El]

The OSv tests have been failing because of the test machine (muninn) ran out of disk space. I'll see if I can find something conspicuous I can delete to reclaim space.

--
Nadav Har'El

n...@scylladb.com

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/1733004883.7314.1648942420927%40jenkins.cloudius-systems.com.

[Nadav Har'El]

The local disk on muninn, which just has half a terabyte, is mostly filled by:

1. Almost a quarter terabyte of /jenkins/.local/share/containers/storage/overlay.  This appears to be a stash of ancient docker containers that probably needs to be cleaned up

2. Another similarly sized directory /var/lib/docker

Those two things account for almost all of the disk space used on that machine.

I hesitate to clean up things that I don't fully understand, so please let me know if one of the others people on this thread (Beni? Avi?) can do this, or you want me to do this myself.

--
Nadav Har'El

n...@scylladb.com

[Nadav Har'El]

On Sun, Apr 3, 2022 at 10:47 AM Nadav Har'El <n...@scylladb.com> wrote:

The local disk on muninn, which just has half a terabyte, is mostly filled by:

1. Almost a quarter terabyte of /jenkins/.local/share/containers/storage/overlay.  This appears to be a stash of ancient docker containers that probably needs to be cleaned up

2. Another similarly sized directory /var/lib/docker

Those two things account for almost all of the disk space used on that machine.

I hesitate to clean up things that I don't fully understand, so please let me know if one of the others people on this thread (Beni? Avi?) can do this, or you want me to do this myself.

I did "docker image prune" which is (as far as I know) the least intrusive or risky form for docker cleanup, and cleared 55 GB.

Now the Jenkins job on munininn (including all OSv jobs, but not only!) will be able to run.

I'm sure a lot more can be reclaimed with more aggressive docker cleanup commands, but I didn't do it. Let me know if you want me to do it.

[jen...@scylladb.com]

See <https://jenkins.scylladb.com/job/osv-build/1066/display/redirect?page=changes>

[Nadav Har'El]

On Sun, Apr 3, 2022 at 12:54 PM Nadav Har'El <n...@scylladb.com> wrote:

On Sun, Apr 3, 2022 at 10:47 AM Nadav Har'El <n...@scylladb.com> wrote:

The local disk on muninn, which just has half a terabyte, is mostly filled by:

1. Almost a quarter terabyte of /jenkins/.local/share/containers/storage/overlay.  This appears to be a stash of ancient docker containers that probably needs to be cleaned up

2. Another similarly sized directory /var/lib/docker

Those two things account for almost all of the disk space used on that machine.

I hesitate to clean up things that I don't fully understand, so please let me know if one of the others people on this thread (Beni? Avi?) can do this, or you want me to do this myself.

I did "docker image prune" which is (as far as I know) the least intrusive or risky form for docker cleanup, and cleared 55 GB.

Now the Jenkins job on munininn (including all OSv jobs, but not only!) will be able to run.

All the OSv jobs ran, and passed, but unfortunately, in the last couple of hours  the amount of free space on the disk was reduced by a further 5 GB.

At this pace, we'll run out disk space again very soon.

So not only should we clear a lot more docker junk (should be easy, but I didn't do it), I think we'll have to do it periodically - or find a way to convince docker not to save all the old stuff when a test completes.