Bug#1039607: libjansi-java: causes maven to always output escape character

[Luís Picciochi Oliveira]

Package: libjansi-java
Version: 2.4.0-1
Severity: normal
X-Debbugs-Cc: Pitx...@Gmail.com

Dear maintainer,

I'm running commands that use the `mvn` and capture the output coming from it.

More specifically, I'm catching the version of a Maven project with a command
like the following:

mvn --batch-mode --quiet -DforceStdout help:evaluate
-Dexpression="project.version"

Up until Debian Bullseye, the output from the command above was a single string
with only the version, as intended. For example: "1.0.0-SNAPSHOT".

After upgrading to Debian Bookworm, the `mvn` command now outputs escape
characters, regardless of the flags passed to it.
The example above has now become: "ESC[0m1.0.0-SNAPSHOTESC[0m".

I tested with the upstream Maven package of the same version, and could narrow
down the problem as coming from the libjansi-java jar.

If I downgrade it back to the Debian-packaged version 2.4.0-1, the old
behaviour is restored.

This change seems to have been due to this commit:
https://salsa.debian.org/java-
team/jansi/-/commit/7f186cd4fc22308d7769db8eeeca26b560b81b1a

You can probably reproduce this very simply with the following:
(running `mvn` anywhere should work - it doesn't have to have a Maven project
and the error message is irrelevant to reproduce what we need)

mvn --batch-mode --quiet | less

* With libjansi-java 2.4.0-2, escape characters are seen in the output, like
so:

ESC[0m[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
(...)
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
ESC[0m
(END)

* With libjansi-java 2.4.0-1 (downloaded from snapshot.debian.org), escape
characters are not seen in the output:

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
(...)
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/NoGoalSpecifiedException
(END)

Thank you and best regards,
Luís Picciochi Oliveira

-- System Information:
Debian Release: 12.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 6.4.0-0-amd64 (SMP w/12 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not
set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

[tony mancill]

On Tue, Jun 27, 2023 at 07:49:19PM +0100, Luís Picciochi Oliveira wrote:

> mvn --batch-mode --quiet | less
>
>
> * With libjansi-java 2.4.0-2, escape characters are seen in the output, like
> so:
>
> ESC[0m[INFO] Scanning for projects...

Hi Luís,

Thank you for the bug report. I don't know the full story of why the
change in behavior between 2.4.0-1 and 2.4.0-2, but I
agree that escape sequences shouldn't be sent when maven is running
in batch mode and that we should address the question of colorized
output referenced in that commit [1] another way.

Emmanuel, do you recall what prompted the change? I didn't see a bug,
but can propose adding a new switch to generate colorized output even
when a TTY isn't detected. That will maintain the expected behavior
with --batch-mode, and we can adjust our build tooling to pass the new
switch if we really think we need it.

Thank you,
tony

[1] https://salsa.debian.org/java-team/jansi/-/commit/7f186cd4fc22308d7769db8eeeca26b560b81b1a

[Emmanuel Bourg]

On 08/07/2023 20:22, tony mancill wrote:

> Emmanuel, do you recall what prompted the change?

I think the issue is that when Maven runs in pbuilder, the TTY isn't
detected and colors get disabled (batch mode isn't used for Debian
builds though). If anything is changed please ensure that building with
pdebuild preserves the colors.

Emmanuel Bourg

[tony mancill]

Looking at the sources (finally), jansi already supports a mode to
"force" [1] escape sequences, even when the output is not a terminal. I
propose that we revert the patch and update the Java build tooling to
set the property as desired for our builds.

Once I get this tested and the property configured, I'll upload to
experimental.

Thanks,
tony

[1] https://salsa.debian.org/java-team/jansi/-/blob/738159f052f027fc0816c95ecc38a424f8aa3637/src/main/java/org/fusesource/jansi/AnsiConsole.java#L83-86

[tony mancill]

On Sat, Dec 23, 2023 at 03:54:45PM -0800, tony mancill wrote:
> On Sat, Jul 15, 2023 at 03:56:01PM +0200, Emmanuel Bourg wrote:
> > On 08/07/2023 20:22, tony mancill wrote:
> >
> > > Emmanuel, do you recall what prompted the change?
> >
> > I think the issue is that when Maven runs in pbuilder, the TTY isn't
> > detected and colors get disabled (batch mode isn't used for Debian builds
> > though). If anything is changed please ensure that building with pdebuild
> > preserves the colors.
>
> Looking at the sources (finally), jansi already supports a mode to
> "force" [1] escape sequences, even when the output is not a terminal. I
> propose that we revert the patch and update the Java build tooling to
> set the property as desired for our builds.
>
> Once I get this tested and the property configured, I'll upload to
> experimental.

The following are now available in experimental:

libjansi-java_2.4.0-3_all.deb
maven-debian-helper_2.6.5~exp1_all.deb

The maven-debian-helper change is here [1]. The two packages have been
tested in conjunction to ensure that:

- Maven output is colorized by default when invoked during Debian
package builds that depend on maven-debian-helper.

- Direct `mvn` Maven output is not colorized by default.

- MAVEN_OPTS, or any other mechanism to set the JVM property jansi.mode,
can be used to specify the mode and override the default behavior.

I believe this covers the (large) majority of Debian build cases and
restores the upstream behavior, but please let me know if I've missed
something. The version of gradle in Debian depends on libjansi1-java,
and so wasn't impacted by the patch to jansi (version 2.x).

Assuming there aren't any concerns, I plan to upload maven-debian-helper
and then the jansi package to unstable on January 7th.

Thanks,
tony

[1] https://salsa.debian.org/java-team/maven-debian-helper/-/commit/082b5b639880e8b3b1c1e98f5ae4f649e3b83b67

[Emmanuel Bourg]

On 28/12/2023 00:48, tony mancill wrote:

> - Maven output is colorized by default when invoked during Debian
> package builds that depend on maven-debian-helper.

Good

> - Direct `mvn` Maven output is not colorized by default.

That's an issue, mvn output should be colorized on a terminal by
default, that's the upstream behaviour.

Emmanuel Bourg

[tony mancill]

Ah, that's right. And it's a little odd, because at this point jansi
is no longer patched and our maven packaging doesn't patch for this
behavior.

Thank you for bringing this up. I will figure out why this is
occurring. It's likely what led to the jansi patch in the first
place.

[tony mancill]

After looking a bit closer, it's not so odd after all. Debian's jansi
(2.x) JAR differs from the upstream version bundled with Maven in that
it doesn't include the jansi-native libraries and doesn't depend on
anything that could provide them.

The Debian jansi-native package is for 1.x and doesn't actually provide
the arch-dependent bits anyway (see [1]), so the default TTY detection
(jansi.mode=default) functionality is broken because all of the native
calls are broken. Perhaps it's more surprising that jansi.mode=force
or the patch work as well as they do.

This will take a bit to sort out, since we either need to build the
arch-specific packages, which will need to go through NEW, or come up
with a pure-Java implementation that behaves like upstream. (Not
directly related, but it would be nice to migrate to upstream 2.4.1 [2],
which uses moditect [3], which would need to be packaged.)

My intuition is that it will be better to update the Debian jansi
package to create the necessary arch-any packages so our version more
closely resembles upstream, and we are better positioned if/when jansi
merges into jline, as mentioned here [4].

Thanks,
tony

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921994
[2] https://github.com/fusesource/jansi/releases/tag/jansi-2.4.1
[3] https://github.com/moditect/moditect
[4] https://github.com/fusesource/jansi/issues/277

[Emmanuel Bourg]

Le 01/01/2024 à 00:27, tony mancill a écrit :

> (Not directly related, but it would be nice to migrate to upstream 2.4.1,

> which uses moditect [3], which would need to be packaged.)

moditect can be ignored, it's mostly used to build JPMS module info
files with Java 8. We can add the missing module-info.java files
directly if that's really needed.

Emmanuel Bourg

[tony mancill]

On Sun, Dec 31, 2023 at 03:27:53PM -0800, tony mancill wrote:
> On Fri, Dec 29, 2023 at 07:23:35AM -0800, tony mancill wrote:
> > On Fri, Dec 29, 2023 at 12:27:07PM +0100, Emmanuel Bourg wrote:
> > > On 28/12/2023 00:48, tony mancill wrote:
> > >
> > > > - Maven output is colorized by default when invoked during Debian
> > > > package builds that depend on maven-debian-helper.
> > >
> > > Good
> > >
> > > > - Direct `mvn` Maven output is not colorized by default.
> > >
> > > That's an issue, mvn output should be colorized on a terminal by default,
> > > that's the upstream behaviour.
>

> My intuition is that it will be better to update the Debian jansi
> package to create the necessary arch-any packages so our version more
> closely resembles upstream, and we are better positioned if/when jansi
> merges into jline, as mentioned here [4].

After giving this some more thought, I don't think we want a jansi
source package that builds arch-any packages. jansi (already) has a
circular build dependency on itself via maven, but there's no
bootstrapping problem because the packages are currently arch-all.

Since the native library is desired but optional, we could create a
separate source package and add a Recommends dependency to
libjansi-java. It may even be possible leverage the existing
jansi-native, but I haven't looked into the question of compatibility
yet.

However, for the short-term, I believe we can achieve the desired
behavior and fix the issue for most use cases by

1. Patching the mvn wrapper script in our maven package to set
jansi.mode=force (and thus colorize output) unless the --batch-mode
option is provided.

2. Moving forward with the current jansi package in experimental.

That restores the desired --batch-mode behavior but maintains
colorized output by default and during Debian package builds.

Would that be acceptable?

[tony mancill]

On Thu, Jan 04, 2024 at 10:22:02PM -0800, tony mancill wrote:
> However, for the short-term, I believe we can achieve the desired
> behavior and fix the issue for most use cases by
>
> 1. Patching the mvn wrapper script in our maven package to set
> jansi.mode=force (and thus colorize output) unless the --batch-mode
> option is provided.
>
> 2. Moving forward with the current jansi package in experimental.
>
> That restores the desired --batch-mode behavior but maintains
> colorized output by default and during Debian package builds.

I am preparing an upload of maven to experimental (version 3.8.7-2~exp1)
that implements the changes described above. I will push the packaging
changes to the debian/experimental branch in the Salsa repo. The patch
for mvn can be found here:

https://salsa.debian.org/java-team/maven/-/blob/4501966b3678135deacca45bc1a918380f6dac47/debian/patches/03_jansi_behavior.patch

When used in conjunction with the jansi package in experimental, the
behavior is:

**output is colorized**
mvn

**output is non-colorized**
mvn -B
mvn --batch-mode

The patched mvn script also checks for MAVEN_JANSI_PROPERTY in the
environment, which allows users to override the behavior if desired.
I don't expect this to be needed very often, but it may be useful if
we modify the behavior of jansi in the future, and I wanted there to
be an escape valve if the patched behavior is problematic for some
unforeseen use case.

The invocations below demonstrate the behavior of Debian's jansi that
led to the patch in 2.4.0-2, i.e., that without the native jansi
libraries or -Djansi.mode=force, the default TTY detection fails and
results in non-colorized output.

MAVEN_JANSI_PROPERTY="" mvn
MAVEN_JANSI_PROPERTY= mvn

For completeness:

MAVEN_JANSI_PROPERTY="-Djansi.mode=default" mvn --> non-colorized
MAVEN_JANSI_PROPERTY="-Djansi.mode=strip" mvn --> non-colorized
MAVEN_JANSI_PROPERTY="-Djansi.mode=force" mvn --> colorized

With these changes, we shouldn't need the patched maven-debian-helper I
proposed previously, since the output of Maven invoked during builds
will be colorized.

I believe this will address the issues reported in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039607 and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059147

Feedback welcome. We can coordinate here before any migrations from
experimental to unstable.

Thanks,
tony

[Clemens Ballarin]

Am 08.01.2024 um 07:03 schrieb tony mancill:

> I am preparing an upload of maven to experimental (version 3.8.7-2~exp1)
> that implements the changes described above. I will push the packaging
> changes to the debian/experimental branch in the Salsa repo. The patch
> for mvn can be found here:
>

> [...]

>
> I believe this will address the issues reported in
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039607 and
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059147
>
> Feedback welcome. We can coordinate here before any migrations from
> experimental to unstable.

Thank you for the fix. To test we created a chroot environment with the
following upgrades:

apt install libjansi-java/experimental
apt install maven/experimental libmaven3-core-java/experimental libcommons-cli-java/unstable libcommons-lang3-java/unstable

The issue reported in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1059147 as well as our
original issue are resolved.

[tony mancill]

Thank you for your testing.

Are there any concerns before I upload maven and jansi to unstable? We
can always revisit the changes if we discover any issues.