[JIRA] [core] (JENKINS-34742) FilePath#installIfNecessaryFrom consistently fails to download certain files

63 views
Skip to first unread message

chris@orr.me.uk (JIRA)

unread,
May 11, 2016, 2:14:01 PM5/11/16
to jenkinsc...@googlegroups.com
Christopher Orr created an issue
 
Jenkins / Bug JENKINS-34742
FilePath#installIfNecessaryFrom consistently fails to download certain files
Issue Type: Bug Bug
Assignee: Oleg Nenashev
Components: core
Created: 2016/May/11 6:13 PM
Environment: Jenkins 1.532 and 2.0; running on OS X 10.11.4 and Ubuntu 14.04.3
Priority: Major Major
Reporter: Christopher Orr

When testing the Go Plugin, which is a ToolInstaller and so makes use of this FilePath API, I've seen that it consistently fails to download certain files — regardless of which OS Jenkins is running on, and from which web server the files are being downloaded from (and whether HTTP or HTTPS).

An typical stacktrace:

java.io.IOException: Failed to install https://storage.googleapis.com/golang/go1.6.2.darwin-amd64.tar.gz to /tmp/jenkins/tools/Go_1.6.2
        at hudson.FilePath.installIfNecessaryFrom(FilePath.java:832)
        at Repro.main(Repro.java:31)
Caused by: java.io.IOException: Failed to unpack https://storage.googleapis.com/golang/go1.6.2.darwin-amd64.tar.gz (158817 bytes read of total 84750242)
        at hudson.FilePath.installIfNecessaryFrom(FilePath.java:826)
        ... 1 more
Caused by: java.io.IOException: Failed to extract input stream
        at hudson.FilePath.readFromTar(FilePath.java:2300)
        at hudson.FilePath.access$400(FilePath.java:190)
        at hudson.FilePath$10.invoke(FilePath.java:720)
        at hudson.FilePath$10.invoke(FilePath.java:718)
        at hudson.FilePath.act(FilePath.java:990)
        at hudson.FilePath.act(FilePath.java:968)
        at hudson.FilePath.untarFrom(FilePath.java:718)
        at hudson.FilePath.installIfNecessaryFrom(FilePath.java:824)
        ... 1 more
Caused by: java.io.IOException: Truncated TAR archive
        at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:618)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
        at hudson.util.IOUtils.copy(IOUtils.java:40)
        at hudson.FilePath.readFromTar(FilePath.java:2290)
        ... 8 more

The issue can be reproduced like this:

FilePath filePath = new FilePath(new File("..."));
filePath.installIfNecessaryFrom(new URL(url), null, null);

This Git repository contains this in a ready-to-run form:
https://github.com/orrc/jenkins-filepath-untar
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

cohawk@yahoo.com (JIRA)

unread,
Nov 8, 2016, 12:23:03 PM11/8/16
to jenkinsc...@googlegroups.com
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

o.v.nenashev@gmail.com (JIRA)

unread,
Nov 8, 2016, 12:25:03 PM11/8/16
to jenkinsc...@googlegroups.com
Oleg Nenashev assigned an issue to Unassigned
 
Change By: Oleg Nenashev
Assignee: Oleg Nenashev

neovortex@gmail.com (JIRA)

unread,
Feb 13, 2017, 12:57:02 AM2/13/17
to jenkinsc...@googlegroups.com
Tiernan Messmer commented on Bug JENKINS-34742
 
Re: FilePath#installIfNecessaryFrom consistently fails to download certain files

On further investigation, this is caused by the same issue as JENKINS-39515 - The gz files on the golang site have an incorrect size in the gzip footer field.

https://github.com/golang/go/issues/19052 has more information.

neovortex@gmail.com (JIRA)

unread,
Feb 13, 2017, 1:02:01 AM2/13/17
to jenkinsc...@googlegroups.com

I noticed gzip handling is being done by JZlib not apache commons compress (I had mistakenly assumed commons compress was handling both gzip and tar). The fact that it is relying on the size in the footer indicates it would likely also fail on >4GB (uncompressed) gz files. Changing to commons compress (which I believe handles >4GB) in FilePath would likely resolve this as a side effect to allow it to handle malformed gz files in addition to being able to handle >4GB gz files.

neovortex@gmail.com (JIRA)

unread,
Feb 27, 2017, 9:08:02 AM2/27/17
to jenkinsc...@googlegroups.com

An update from https://github.com/golang/go/issues/19052#issuecomment-279311635 - it appears the root cause of this issue is JZlib does not correctly handle chunked/multistream gzip files.

While the golang distributions are no longer using chunked gzip files to increase compatibility, this may yet appear in other situations, so the long term solution would be to switch JZlib to a better library that correctly handles this part of the gzip RFC, or getting the issue fixed in JZlib
Reply all
Reply to author
Forward
0 new messages