Bug#1040226: tomcat10: deployment-time Java EE to Jakarta EE migration fails

[Tamás J.Tóth]

Package: tomcat10
Version: 10.1.10-1
Severity: important
X-Debbugs-Cc: debrep...@jnet.hu

Dear Maintainer,

The deployment-time Java EE to Jakarta EE migration, as documented in
the Tomcat 10 migration guide
<https://tomcat.apache.org/migration-10.html#Specification_APIs>, fails
with `NoClassDefFoundError: org/apache/tomcat/jakartaee/Migration`. This
affects only those who use the deployment-time Java EE to Jakarta EE
migration, but makes this migration completely unusable.

Steps to reproduce:

1. Install Tomcat 10.
2. Put a web app in the `appBase` directory (/var/lib/tomcat10/webapps).
It can be very simple, e.g. a directory (say, test/) containing a
single index.html.
3. Start Tomcat.
4. Try to open the web app. Assuming you named the directory test/, it
should be at http://localhost:8080/test/.
5. Verify that it loads.
6. Move the web app to the `legacyAppBase` directory
(/var/lib/tomcat10/webapps-javaee) and restart Tomcat.
7. Try to open the web app again.

Actual result:

The web app doesn't load. The Tomcat log contains the following:

WARNING [main] org.apache.catalina.startup.HostConfig.migrateLegacyApp Migration failure
java.lang.NoClassDefFoundError: org/apache/tomcat/jakartaee/Migration
at org.apache.catalina.startup.HostConfig.migrateLegacyApp(HostConfig.java:1243)
at org.apache.catalina.startup.HostConfig$MigrateApp.run(HostConfig.java:1996)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
at org.apache.catalina.startup.HostConfig.migrateLegacyApps(HostConfig.java:1208)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:419)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1656)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:893)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:846)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1328)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1318)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:866)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:241)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:428)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:919)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Catalina.start(Catalina.java:795)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:347)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.tomcat.jakartaee.Migration
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 35 more

Expected result:

The web app still loads.

-- System Information:
Debian Release: 12.0
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-9-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=C.UTF-8 (charmap=locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: unable to detect

Versions of packages tomcat10 depends on:
ii systemd [systemd-tmpfiles] 253.5-1
ii sysvinit-utils [lsb-base] 3.06-4
ii tomcat10-common 10.1.10-1
ii ucf 3.0043+nmu1

Versions of packages tomcat10 recommends:
pn libtcnative-1 <none>

Versions of packages tomcat10 suggests:
ii tomcat10-admin 10.1.10-1
pn tomcat10-docs <none>
pn tomcat10-examples <none>
pn tomcat10-user <none>

-- debconf information excluded

[Markus Koschany]

Am Montag, dem 03.07.2023 um 18:28 +0200 schrieb Tamás J.Tóth:
>
>
> The web app doesn't load. The Tomcat log contains the following:
>
> WARNING [main] org.apache.catalina.startup.HostConfig.migrateLegacyApp
> Migration failure
>         java.lang.NoClassDefFoundError: org/apache/tomcat/jakartaee/Migration

Thanks for the report.

You have to install the tomcat-jakartaee-migration package. [1] I guess we
could either suggest or recommend this package to make it more obvious.

https://tracker.debian.org/pkg/tomcat-jakartaee-migration

Regards,

Markus

[Tamás J.Tóth]

Hi,

I’ve installed the tomcat-jakartaee-migration package, but it still
throws the same exception. Are you sure the JAR file it contains is on
the classpath of Tomcat?

Also, thinking about what I would’ve done if this package was suggested
or even recommended – I probably wouldn’t have installed it. It’s
obvious that it’s suggested to use it if one *doesn’t* want to use the
deployment-time migration, so I would’ve thought that the suggestion is
because of that, not because it’s necessary for Tomcat to work properly.
If you decide to make it only a recommendation or suggestion, please
indicate in the package description of tomcat10 that it’s necessary for
the deployment-time migration.

I realized that I missed an important detail in my report: I ran Tomcat
manually as root using tomcat-start.sh (in a Docker container) rather
than as a systemd service. I don’t have Sid with systemd set up, but
testing on Bookworm with systemd throws a different exception: instead
of a NoClassDefFoundError at HostConfig.java:1243, it already throws an
IOException at HostConfig.java:1232 – it tries to write the
webapps-javaee directory, but only the webapps directory is read/write
enabled in the systemd unit file. After editing the unit file to enable
read/write access for the webapps-javaee directory, the systemd service
also throws the NoClassDefFoundError. Testing on Bookworm by manually
running tomcat-start.sh as root also throws the NoClassDefFoundError.

Regards,
Tamás

[Markus Koschany]

Hello,

Am Montag, dem 07.08.2023 um 20:22 +0200 schrieb J. Tóth Tamás:
> Hi,
>
> Did you notice my reply sent on 4 July?

Yes, I did. Please keep the bug report always in CC.

> We’d like to gradually upgrade
> to Bookworm, but I don’t want to make sysops’ lives more complicated by
> giving them one WAR file to install on Bookworm and another one to
> install on Bullseye.

I haven't had the time to investigate this problem yet. You don't have to wait
for a solution of this bug though. You can just manually run the tomcat-
jakartaee-migration tool on your existing war files. All it mainly does is to
replace the old occurrences of javax.* with jakarta.*. You could also send a
patch with your proposed changes to Debian's tomcat10 package and I take a look
at it.

Markus

[J. Tóth Tamás]

Hi,

> Please keep the bug report always in CC.

I thought my 8 August mail contains no new information, so it makes no
sense to spam the BTS with it. But okay, next time (and this time) I’ll
use Reply All regardless of the message content.

>> We’d like to gradually upgrade
>> to Bookworm, but I don’t want to make sysops’ lives more complicated by
>> giving them one WAR file to install on Bookworm and another one to
>> install on Bullseye.
>

> […] You can just manually run the tomcat-jakartaee-migration tool on

> your existing war files. All it mainly does is to replace the old
> occurrences of javax.* with jakarta.*.

That’s exactly what I meant by “making sysops’ lives more complicated by

giving them one WAR file to install on Bookworm and another one to

install on Bullseye”, so it’s out of question for me in production.

> You could also send a patch with your proposed changes to Debian's tomcat10 > package and I take a look at it.

I created https://salsa.debian.org/java-team/tomcat10/-/merge_requests/1
– it’s quite likely wrong in its current form, but I hope it can be fixed.

Tamás

[Markus Koschany]

Hi,

Am Dienstag, dem 15.08.2023 um 14:52 +0200 schrieb J. Tóth Tamás:
> Hi,
>
> > Please keep the bug report always in CC.
>
> I thought my 8 August mail contains no new information, so it makes no
> sense to spam the BTS with it. But okay, next time (and this time) I’ll
> use Reply All regardless of the message content.

I'm not the only Debian developer who could help you. There may be other users
too who are interested in this topic and who could provide valuable insight or
suggestions. Hence why it's so important to have a public bug tracker with the
record of our conversation.

>
> > > We’d like to gradually upgrade
> > > to Bookworm, but I don’t want to make sysops’ lives more complicated by
> > > giving them one WAR file to install on Bookworm and another one to
> > > install on Bullseye.
> >
> > […] You can just manually run the tomcat-jakartaee-migration tool on
> > your existing war files. All it mainly does is to replace the old
> > occurrences of javax.* with jakarta.*.
>
> That’s exactly what I meant by “making sysops’ lives more complicated by
> giving them one WAR file to install on Bookworm and another one to
> install on Bullseye”, so it’s out of question for me in production.

Ok, I'm not familiar with your workflow. Though I would never trust a tool to
do the right thing on the fly. I would instruct my development team to port the
application to Tomcat 10 and test it thoroughly and then my production team
only would take care of the administration and deployment task as usual.

>
> > You could also send a patch with your proposed changes to Debian's tomcat10
> > > package and I take a look at it.
>
> I created https://salsa.debian.org/java-team/tomcat10/-/merge_requests/1 
> – it’s quite likely wrong in its current form, but I hope it can be fixed.

Thank you! I have merged your request and also pushed a new upstream version to
our Git repository but I didn't upload the package yet. Patches should be added
to the debian/patches/series file as well, otherwise they won't be applied. The
rest looked good to me. However I still get a migration warning when I follow
your initial steps with the basic "test" app. There is a
DirectoryNotEmptyException. Initially tomcat migrates the test folder in
webapps-javaee to a newxxxxx.tmp folder and then tries to remove the latter,
which fails. What can we do to avoid this warning?

Regards,

Markus