Possible LTS regression in commons-compress

[James Nord]

Hi all,

just a quick note, I think I have discovered a regression in 2.204 upcoming LTS due to commons-compress library bump.

One of our plugins was using TarInputStream for 2.190 but this is now restricted.
so without bumping core we moved the code over to use the commons-compress version (and all is happy).

However when bumping the Jenkins version to 2.204 which picks up the newer compress we have lots of unit test failures due to what seems like (at very first glance as I just narrowed it down and its late!) empty tar files (when they should not be empty)

as a (quick and dirty) test I made the commons-compress version configurable and it does seem like this is the issue

mvn test -Dtest=TheTest -Djenkins.version=2.204 -DcompressVersion=1.19 -Denforcer.skip fails
mvn test -Dtest=TheTest -Djenkins.version=2.204 -DcompressVersion=1.10 -Denforcer.skip passes
mvn test -Dtest=TheTest -Djenkins.version=2.190.3 -DcompressVersion=1.10 -Denforcer.skip passes
mvn test -Dtest=TheTest -Djenkins.version=2.190.3 -DcompressVersion=1.19 -Denforcer.skip fails

doing some more bisection

mvn test -Dtest=TheTest -Djenkins.version=2.190.3 -DcompressVersion=1.15 -Denforcer.skip fails
mvn test -Dtest=TheTest -Djenkins.version=2.190.3 -DcompressVersion=1.13 -Denforcer.skip passes
mvn test -Dtest=TheTest -Djenkins.version=2.190.3 -DcompressVersion=1.14 -Denforcer.skip passes


I don't yet know what is causing this (but it does appear to be something in commons-compress 1.15 and I will investigate that.  (changelog)

But this seems very scary to me (and I am surprised there has not been some reports in the weeklies of archives etc not working correctly).  Also very weired as 1.15 has been released for a good amount of time.

Will continue the investigation tomorrow but if this rings a bell with anyone in any reported Jiras it would be good to correlate.

/James

[Björn Pedersen]

Hi,

I have 2 changes that seem candiates:

1) 1.15 enforces blocklen=512 which is different from the old default and raises an error if anything else is specified

 (https://commons.apache.org/proper/commons-compress/javadocs/api-1.19/index.html)

2) 1.16 renamed the preserveAbsolutePath to TarArchiveEntry(String name, boolean preserveAbsolutePath)

(less likely that is the cause).


Björn

[Baptiste Mathus]

Sounds to me we should git bisect commons-compress between 1.14 and 1.15 usage, to nail and possibly confirm Björn's hypothesis?

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/8b3e8989-4f34-4b1d-a285-036837dc6bd1%40googlegroups.com.

[James Nord]

I have tracked the cause of the issue to an optimisation we had for the old Hudson code to not flush all the time to prevent lots of small chunks being flushed (as the old implementation flushed for each block (512 bytes)) 

If I disable that then the code works with 1.15 and above.  I have not looked into the code to see if commons compress also has issues with flushing, but I doubt any other plugins will be doing any similar optimisation.

/James

To unsubscribe from this group and stop receiving emails from it, send an email to jenkin...@googlegroups.com.

[Oliver Gondža]

James, thanks for the updates. The way I understand that is, problem is
easily fixable, affects proprietary component and is a fairly unlikely
thing to do to begin with, so we do not consider this to be a blocker
for LTS.

Unless someone tells me I am wrong, I am targeting the RC for tomorrow.

On 27/11/2019 12.50, James Nord wrote:
> I have tracked the cause of the issue to an optimisation we had for the
> old Hudson code to not flush all the time to prevent lots of small
> chunks being flushed (as the old implementation flushed for each block
> (512 bytes))
>
> If I disable that then the code works with 1.15 and above.  I have not
> looked into the code to see if commons compress also has issues with
> flushing, but I doubt any other plugins will be doing any similar
> optimisation.
>
> /James
>
>
>
>
> On Wednesday, November 27, 2019 at 8:23:47 AM UTC, Baptiste Mathus wrote:
>
> Sounds to me we should git bisect commons-compress between 1.14 and
> 1.15 usage, to nail and possibly confirm Björn's hypothesis?
>
> On Wed, Nov 27, 2019 at 8:36 AM 'Björn Pedersen' via Jenkins
> Developers <jenkin...@googlegroups.com <javascript:>> wrote:
>
> Hi,
>
> I have 2 changes that seem candiates:
> 1) 1.15 enforces blocklen=512 which is different from the old
> default and raises an error if anything else is specified

>  (https://commons.apache.org/proper/commons-compress/javadocs/api-1.19/index.html <https://commons.apache.org/proper/commons-compress/javadocs/api-1.19/index.html>)

> 2) 1.16 renamed the preserveAbsolutePath to |TarArchiveEntry

> <https://commons.apache.org/proper/commons-compress/javadocs/api-1.19/org/apache/commons/compress/archivers/tar/TarArchiveEntry.html#TarArchiveEntry-java.lang.String-boolean->(String
> <https://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true> name,

> boolean preserveAbsolutePath)|
> |(less likely that is the cause).
> |
>
> Björn
>
> Am Mittwoch, 27. November 2019 01:03:40 UTC+1 schrieb James Nord:
>
> Hi all,
>

> just a quick note, I /think /I have discovered a regression

> in 2.204 upcoming LTS due to commons-compress library bump.
>
> One of our plugins was using TarInputStream

> <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/org/apache/tools/tar/TarInputStream.java#L47>for

> 2.190 but this is now restricted.
> so without bumping core we moved the code over to use the
> commons-compress version (and all is happy).
>
> However when bumping the Jenkins version to 2.204 which
> picks up the newer compress we have lots of unit test
> failures due to what seems like (at very first glance as I
> just narrowed it down and its late!) empty tar files (when
> they should not be empty)
>
> as a (quick and dirty) test I made the commons-compress
> version configurable and it does seem like this is the issue
>
> mvn test -Dtest=TheTest -Djenkins.version=2.204

> -DcompressVersion=1.19 -Denforcer.skip *fails*

> mvn test -Dtest=TheTest -Djenkins.version=2.204
> -DcompressVersion=1.10 -Denforcer.skip passes
> mvn test -Dtest=TheTest -Djenkins.version=2.190.3
> -DcompressVersion=1.10 -Denforcer.skip passes
> mvn test -Dtest=TheTest -Djenkins.version=2.190.3

> -DcompressVersion=1.19 -Denforcer.skip *fails
> *

> doing some more bisection
>
> mvn test -Dtest=TheTest -Djenkins.version=2.190.3

> -DcompressVersion=1.15 -Denforcer.skip *fails
> *mvn test -Dtest=TheTest -Djenkins.version=2.190.3

> -DcompressVersion=1.13 -Denforcer.skip passes
> mvn test -Dtest=TheTest -Djenkins.version=2.190.3
> -DcompressVersion=1.14 -Denforcer.skip passes
>
>
> I don't yet know what is causing this (but it does appear to
> be something in commons-compress 1.15 and I will investigate
> that.  (changelog

> <https://commons.apache.org/proper/commons-compress/changes-report.html#a1.15>)

>
> But this seems very scary to me (and I am surprised there
> has not been some reports in the weeklies of archives etc
> not working correctly).  Also very weired as 1.15 has been
> released for a good amount of time.
>
> Will continue the investigation tomorrow but if this rings a
> bell with anyone in any reported Jiras it would be good to
> correlate.
>
> /James
>
> --
> You received this message because you are subscribed to the
> Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to jenkin...@googlegroups.com <javascript:>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/8b3e8989-4f34-4b1d-a285-036837dc6bd1%40googlegroups.com

> <https://groups.google.com/d/msgid/jenkinsci-dev/8b3e8989-4f34-4b1d-a285-036837dc6bd1%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to jenkinsci-de...@googlegroups.com
> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-dev/6259ccf6-f5de-469e-a7db-18ff7dd52651%40googlegroups.com
> <https://groups.google.com/d/msgid/jenkinsci-dev/6259ccf6-f5de-469e-a7db-18ff7dd52651%40googlegroups.com?utm_medium=email&utm_source=footer>.


--
oliver