jOOQ 3.11, Java 8, JAXB, OOM Compressed class space

[Thorsten Schöning]

Hi all,

I'm running a web-app in Tomcat using jOOQ to access a PostgreSQL
database and some months ago I upgraded jOOQ to version 3.11.7, a bit
later OpenJDK to version 8 and Tomcat as well on my production
servers. At first, things seemed to work fine, but end of March this
year I encountered the OOM "Compressed class space" for the first
time, which happened two additional times since then, once every
month. But I'm somewhat sure that this is really only by accident and
simply depends on how often Tomcat got restarted.

Using JMX, I collected data for "GC.class_stats" the last few days and
recognized that the number of loaded classes is constantly growing,
reducing the meta space of the JVM at the same time. Diffing the files
showed especially some jOOQ-related classes increasing:

> org.jooq.conf.MappedSchema$JaxbAccessorF_input
> org.jooq.conf.MappedTable$JaxbAccessorF_input
> org.jooq.conf.RenderFormatting$JaxbAccessorF_indentation
> org.jooq.conf.RenderFormatting$JaxbAccessorF_printMargin
> org.jooq.conf.RenderMapping$JaxbAccessorF_schemata
[...]

> Searching for: org\.jooq\.conf\..+\$JaxbAccessorF_.+
> [...]\2019-05\29\18-53\JMC 04 Diag Cmds GC.class_stats.txt: 242
> [...]\2019-05\29\19-10\JMC 04 Diag Cmds GC.class_stats.txt: 530
> [...]\2019-05\30\19-15\JMC 04 Diag Cmds GC.class_stats.txt: 134842
> [...]\2019-05\31\19-10\JMC 04 Diag Cmds GC.class_stats.txt: 275290
> [...]\2019-06\02\12-45\JMC 04 Diag Cmds GC.class_stats.txt: 425626

Searching for "Compressed class space" before already, I came across
problems for the combination of Java 8 and JAXB:

https://stackoverflow.com/questions/40346869/how-to-analyze-memory-leaks-in-java-8-compressed-class-space?rq=1
https://stackoverflow.com/questions/3259291/do-i-have-a-jaxb-classloader-leak
https://stackoverflow.com/questions/33255578/old-jaxb-and-jdk8-metaspace-outofmemory-issue

So these jOOQ-classes got my attention and it seems that I'm suffering
from the same problem like mentioned on SO. The only problem is I
didn't find anyone else having the same problem with jOOQ and Java 8,
which makes me wonder.

So, in your opinion, does my investigation make sense so far? Did
anyone else ran into the same problem already and I just missed it? If
so, how did you resolve the problem?

I've read that jOOQ gets rid of JAXB in version 3.12 and am going to
give this a try next most likely:

> In hindsight, I deeply regret having ever used JAXB. jOOQ 3.12 will
> ship with its own "mini JAXB" implementation to get rid of this mess
> for most people, depending only on the API (annotations).

https://github.com/jOOQ/jOOQ/issues/6879#issuecomment-440213498

If my investigation is correct and others simply did not run into
this pretty often yet, jOOQ might want to consider backporting one of
the SO-fixes mentioned to older versions as well. Because fomr my
understanding of the problem, everyone should run into this in
persistent environment with long-running JVMs sooner or later.

Mit freundlichen Grüßen,

Thorsten Schöning

--
Thorsten Schöning E-Mail: Thorsten....@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/

Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04

AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow

[Lukas Eder]

Hi Thorsten,

Thank you very much for your thorough analysis. As if we needed another reason for getting rid of JAXB :) jOOQ 3.12, which finally officially supports Java 11+ will have replaced the JAXB implementation by our own, internally, to get rid mostly of the numerous issues caused by the removal of JAXB from the JDK. Great to hear that your issue will likely also be solved by jOOQ 3.12 - at least if you're going to run it on Java 11+

If there's anything you think we should fix on our side in the existing JAXB integration, please let us know. This seems to be difficult to reproduce, so if you can offer a test case / MCVE that would be great, of course: https://github.com/jOOQ/jOOQ-mcve.

For example, this particular fix seems to be easy to implement: https://stackoverflow.com/a/3285931/521799. If you could patch your version and try this fix to see if this OOM still happens, we'd love to apply the patch to 3.11 and 3.10.

On Sun, Jun 2, 2019 at 3:05 PM Thorsten Schöning <tscho...@am-soft.de> wrote:

So these jOOQ-classes got my attention and it seems that I'm suffering
from the same problem like mentioned on SO. The only problem is I
didn't find anyone else having the same problem with jOOQ and Java 8,
which makes me wonder.

Indeed, I'm not aware of any such issue yet

 

If my investigation is correct and others simply did not run into
this pretty often yet, jOOQ might want to consider backporting one of
the SO-fixes mentioned to older versions as well.

I'm afraid that is unlikely going to happen. The path towards this MiniJAXB replacement was rocky, and even in jOOQ 3.12, when JAXB is on the classpath, that will be preferred over MiniJAXB, the latter being a replacement only in cases where the tedious transitive dependency has not already been added.

jOOQ 3.11 has had its 11th patch release, so I would not want to introduce anything as big as an entire dependency replacement now this late in the patch release "chain" given that the expectation for an upgrade to a high patch release number is to be increasingly riskless.

Working around the problem directly using a workaround like the above linked one seems more desireable for 3.11

 

Because fomr my
understanding of the problem, everyone should run into this in
persistent environment with long-running JVMs sooner or later.

I'm quite convinced that a lot of people use jOOQ in long-running JVMs. Given that this particular issue has not popped up on this list, stack overflow, or github yet, I think this might just not be such a frequent issue. What jOOQ/JAXB feature are you using at runtime? The main reason for the JAXB dependency is for the code generator to load XML. There are more edge-casey reasons around using the InformationSchema types, or Settings...

[Thorsten Schöning]

Guten Tag Lukas Eder,
am Sonntag, 2. Juni 2019 um 20:03 schrieben Sie:

> Thank you very much for your thorough analysis. As if we needed another
> reason for getting rid of JAXB :)

I have been too early to blame jOOQ for my problems, sorry for that:

After reading your answer, I had another look at how I'm configuring
jOOQ and that's the place where I use JAXB on my own. I still had in
mind that jOOQ is configuring itself automatically, but I changed that
in the past to place the "jooq-settings.xml" into some other directory
not in the classpath. I'm doing something along the lines of the
official docs:

> Settings settings = JAXB.unmarshal(new File("/path/to/settings.xml"), Settings.class);

https://www.jooq.org/doc/3.11/manual/sql-building/dsl-context/custom-settings/

Only slightly different to be able to add a better error handler and
some logging and stuff. In the end, I'm calling the following myself:

> JAXBContext.newInstance(String contextPath,
> ClassLoader classLoader)

https://docs.oracle.com/javaee/7/api/javax/xml/bind/JAXBContext.html#newInstance-java.lang.String-java.lang.ClassLoader-

Providing a class loader is important because of a special
Axis2-environment. So it seems I'm introducing the problem on my own
and should be able to implement one of the SO-workarounds for the same
reason.

I had a look at your #7734 introducing MiniJAXB and you write the
following there:

> Settings (hardly used as XML)

https://github.com/jOOQ/jOOQ/issues/7734

How do I need to understand that sentence? What are the alternatives
to using JAXB to load the settings from a file? Are other formats
supported as well and I missed those in the docs? Or are file-based
settings at all simply not very common with jOOQ?

I'm asking because with my usage of JAXB I'm going to run into
problems with JDK 9 and 11 like you did already anyway.

Am I supposed to use MiniJAXB to load the settings with jOOQ 3.12? If
I did use that for loading settings while JAXB is still available,
that might result in the same problems I have right now.

> Working around the problem directly using a workaround like the
> above linked one seems more desireable for 3.11

That's what I had in mind only anyway.

> I'm quite convinced that a lot of people use jOOQ in long-running
> JVMs. Given that this particular issue has not popped up on this
> list, stack overflow, or github yet, I think this might just not be
> such a frequent issue. What jOOQ/JAXB feature are you using at
> runtime?

I'm somewhat sure to only load settings from the XML-file and leave
everything else to jOOQ. Can't see any usage of "InformationSchema" as
XML like mentioned in #7734. The code generator uses XML-config as
well, but that isn't a long running process anyway, so can be ignored.
My "jooq-settings.xml" is pretty small as well:

> <?xml version="1.0" encoding="UTF-8"?>
> <settings xmlns="http://www.jooq.org/xsd/jooq-runtime-3.11.2.xsd">
> <executeLogging>true</executeLogging>
> <renderFormatted>true</renderFormatted>
> <renderKeywordStyle>UPPER</renderKeywordStyle>
> <renderSchema>false</renderSchema>
> </settings>

I had situations in the past where it made debugging easier being able
to change those settings, but can't remember the details anymore.
Maybe I should considering hard-coding those things and get rid of
JAXB this way.

[Lukas Eder]

Hi Thorsten,

I'm glad to hear you've been able to further track down the problem. Thanks for documenting these things here.

On Mon, Jun 3, 2019 at 10:34 AM Thorsten Schöning <tscho...@am-soft.de> wrote:

> Settings (hardly used as XML)

https://github.com/jOOQ/jOOQ/issues/7734

How do I need to understand that sentence? What are the alternatives
to using JAXB to load the settings from a file? Are other formats
supported as well and I missed those in the docs? Or are file-based
settings at all simply not very common with jOOQ?

The last, from what I can tell. If you're using file based settings, then yes, JAXB is a good option, and we don't support any other means out of the box. But throughout the manual, almost only programmatic construction of Settings is documented, so I doubt a lot of people are aware of the JAXB based option.

 

I'm asking because with my usage of JAXB I'm going to run into
problems with JDK 9 and 11 like you did already anyway.

Am I supposed to use MiniJAXB to load the settings with jOOQ 3.12? If
I did use that for loading settings while JAXB is still available,
that might result in the same problems I have right now.

You can continue using JAXB, once you've set it up. As I said, MiniJAXB will be used behind the scenes if a JAXB implementation is not on the classpath (which it often isn't, post JDK 11).

 

I had situations in the past where it made debugging easier being able
to change those settings, but can't remember the details anymore.
Maybe I should considering hard-coding those things and get rid of
JAXB this way.

Your life will definitely be much easier that way, given that you also no longer need to care about updating the xmlns version when we do it :)

Cheers,

Lukas