Starting Quarkus 40% faster with CDS

[Leonardo Terrão]

I was doing some tests with Quakus and I found something may help your application start faster.

I'm using class data sharing (CDS) to reduce startup time.

 

I put these three instructions in my docker file.

RUN /usr/bin/java -XX:+UseAppCDS -XX:DumpLoadedClassList=/deployments/classes.lst -jar /deployments/app.jar & sleep 5 && exit
RUN /usr/bin/java -Xshare:dump -XX:+UseAppCDS -XX:SharedClassListFile=/deployments/classes.lst -XX:SharedArchiveFile=/deployments/app-cds.jsa --class-path /deployments/app.jar
ENTRYPOINT /usr/bin/java -Xshare:on -XX:SharedArchiveFile=/deployments/app-cds.jsa -jar "/deployments/app.jar"

In the first line, we start the application and load a set of classes from the jar file into a private internal representation.

In the second line, we dump the internal representation into a shared file.

In the third line, we start our application with the dumped file.

Here is my docker file. It's like the Quarkus default docker file, I removed the last line to put my instructions.

https://github.com/leonardoterrao/pets-app/blob/master/src/main/docker/Dockerfile-cache.jvm#L55

My app is small and starts 40% faster. Has anyone been tried it with a complex Quakus app? How faster is it with a complex app?

[Kevin Viet]

Hello,

I think running AppCDS in a Docker Build might be an off topic Quarkus question, since you can do the same with Spring Boot.

What would be interesting is that Quarkus generates App CDS to throw in the classes archive during build time like it's described here https://github.com/quarkusio/quarkus/issues/6015

Regards

Kevin

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/1f34c136-b41a-4c92-b230-ab8cc04a616a%40googlegroups.com.

[Loïc MATHIEU]

I Leonardo, I tried it in a more complex application (REST Json / JPA / OpenAPI / Metrics / ...) and it gives me a 60% startup improvements (1,2s => 500ms).

Fun you talked about it, I wrote a blog post about it yesterday and planned to publish it today ;)

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CA%2B%2Bd4ot%2BF52u5NH0fxbpAUXF676zLrGt%3DmeLuLxDg-0wXtcv%3DA%40mail.gmail.com.

[Georgios Andrianakis]

This is interesting.

Now that we have Jib support, we could probably have a step that starts the application in order to capture the CDS and then bake them into the generated image, while also modifying the launch command.

Of course something like this would be opt-in and would only work if the base image was Java 13+.

WDYT?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVG6a%3DrLam5gtS-6pBM2eV_1YtT06dS7y8OMxa7nRTc6hg%40mail.gmail.com.

[Loïc MATHIEU]

AppCDS exist since 8 and has been opensourced in 10 (https://openjdk.java.net/jeps/310) and enhanced in 11 (https://openjdk.java.net/jeps/350)

In Java 12, there is a default AppCDS for the JDK (https://openjdk.java.net/jeps/341) apparently it is generated from an empty java command line ...

My 60% improvement is for Java 11.

[Georgios Andrianakis]

I was thinking of https://openjdk.java.net/jeps/350.

What's the difference between Java 11 and 13 with regards to that?

[Loïc MATHIEU]

JEP 350 facilitate the creation of the archive, that's all. No real difference on how it works.

The issue with automating AppCDS is that in order to discover the more classes, you need to be sure that your apps starts (that's why Leonardo add a sleep 5 command), and most classes are discovered by the VM.

As Quarkus do most of it's stuff at build time and not lazily at runtime I assume that starting the apps is OK but in my test I saw a difference in the size of the AppCDS archive between starting the apps and stoping righ after, or starting the apps and calling some rest service on it ...

So, we can explain how to do it somewhere but I don't think we can automate it for all use cases.

[Georgios Andrianakis]

On Fri, May 29, 2020 at 10:46 AM Loïc MATHIEU <loik...@gmail.com> wrote:

JEP 350 facilitate the creation of the archive, that's all. No real difference on how it works.

The issue with automating AppCDS is that in order to discover the more classes, you need to be sure that your apps starts (that's why Leonardo add a sleep 5 command), and most classes are discovered by the VM.

As Quarkus do most of it's stuff at build time and not lazily at runtime I assume that starting the apps is OK but in my test I saw a difference in the size of the AppCDS archive between starting the apps and stoping righ after, or starting the apps and calling some rest service on it ...

So, we can explain how to do it somewhere but I don't think we can automate it for all use cases.

I don't see why not in the case of the container image creation. Since all we have to do is run the application with the proper flags and hit some endpoint, then stop the application and plug the output into the launch command used by the container, I don't see why it wouldn't work.

In any case, please post the link to your blog post when it's done and I'll take a look and see what we can do for Quarkus + Jib to automate things.

[Stuart Douglas]

We could have a special entry point that loads all the classes, so they would all be compiled. Given that a lot of them may not actually be used by the running app though I am not sure how this would affect memory usage and overall startup time.

Stuart

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVH2FxttO5mU4zKCczZQ0P7Fc12L4xYujwvErgmrB_3QVQ%40mail.gmail.com.

[Loïc MATHIEU]

@Stuart Douglas very interesting idea !

If by "all the classes" you mean the one indexed by Jandex this should not be a lot of them anyway.

But yes, this will impact memory usage negatively (class metadata usage exactly) but should impact startup time positively.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czSvJ53979eOj%2B-VubXGHqzjj2wfHp9AYZsY5wA3uLvBA%40mail.gmail.com.

[Emmanuel Bernard]

Make sure you check any potential RSS increase when using AppCDS. This is a very important metric for Quarkus. More than startup time.

https://quarkus.io/guides/performance-measure

[John O'Hara]

This is something that we investigated a few years ago with Wildfly, so things may have improved since then. There was the issue of making sure you classes were loaded, but you also have to bundle the generated class list with you app, which for an app server was 100s MB, I don't know what the file size would be a reasonable sized quarkus app.  IIRC the was a metadata overhead, but can not remember how much it impacted RSS.

There is also the heap serialization work that Christine Flood has done that serializes the started heap (similar to graalVM) to disk, but that only works on Linux afaik.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CANYWk7Nk9Ji68c_K48NTzJXRHVMTee2RMczrd84an4j9aYNPwg%40mail.gmail.com.

--

John O'Hara

Principal Software Engineer, Middleware (Performance)

Red Hat

[Loïc MATHIEU]

My class bundle is 60MB large.

RSS size is not impacted as the JVM loads only the needed class, AppCDS is not eagerly loaded, it can be seen as a cache for class metadata (avoid searching it on the classpath, loading it from disk, verify it's bytecode and preparing the Metadata object).

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CA%2BHD-p37xatMTmoCJ1DuOB0YKKQDrdVp0V77RcGy73RN-6TMLQ%40mail.gmail.com.

[Georgios Andrianakis]

Looking forward to your blog post 😉

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVFPMJuvnZo7Hw__5EfkO%3DzqpvHdPmYViZzyiZJNSjD8zw%40mail.gmail.com.

[Loïc MATHIEU]

@Georgios Andrianakis send on a different mail as I also include test for JLink ;)

@Stuart Douglas I think again at the capability to list all classes, in fact I try a class bundle without accessing any endpoint (as Leonoardo did) and an other after accessing my rest endpoint. Despite the second bundle is 12MO larger, there is no noticeable startup improvement.

I think the main startup improvement is for pre-loading the metadata for the class used during the startup of Quarkus: the JDK classes and the Quarkus core and extension classes.

Application specific classes will be loaded over time, when used. 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-km_08UtU4sPzC7nF6w2dLOC0%3D5CD7JKuou%2BPaBpSqD6w%40mail.gmail.com.

[Georgios Andrianakis]

On Fri, May 29, 2020 at 3:21 PM Loïc MATHIEU <loik...@gmail.com> wrote:

@Georgios Andrianakis send on a different mail as I also include test for JLink ;)

Thanks! I went through it and I don't see anything there that we can't automate for the Jib use case. Maybe I'll have time to do a PR next week.

[Loïc MATHIEU]

Cool, I'll update my article with those info then.

[Georgios Andrianakis]

I did a quick manual test and it indeed worked well with the regular jar format, but it didn't seem to have any effect with the new fast-jar format. I haven't read up on AppCDS requirements yet, but I assume that this might have something to do with the custom ClassLoader we use in the fast-jar case. Perhaps in that case we need to do some extra for the AppCDS to work properly.

[Stuart Douglas]

The main point of manually listing the classes is that it removes the need to start the application as part of the normal build process. If you are going to start it you need to make sure that all the services are present as well (databases etc), and by default it will attempt to use the production database. Generating the class list in an offline manner avoids all this.

Something else to check is time to first request, rather than just startup time, it will likely be slightly improved in the 'all classes' case, unless you actually serve requests after starting you app.

Stuart

[Gunnar Morling]

Hey all,

I've put together a blog post with my experiences on using AppCDS with a Quarkus app:

    https://www.morling.dev/blog/building-class-data-sharing-archives-with-apache-maven/

Time-to-first-response for a DB-backed CRUD app is down by ~30% with OpenJDK 14 (measured as described in the Quarkus docs) and another ~10% with the latest OpenJDK 15 EA build. It's better with 15 as more classes become eligible for CDS (besides some other improvements). It should get even further improved in the next preview build (27) where also Lambda proxy classes will be archived.

Size of the archive file is ~40 MB, i.e. on K8s, a remote download would typically slow down the first start on a given node, before the image is cached and it pays off.

Memory usage is interesting: I see larger RSS initially with CDS, but after a GC it's at the same level with and without CDS. It's a bit beyond my insight, but AFAICS it's largely byte[] data on the heap, which I reckon originates from reading the memory-mapped CDS file. Metaspace is ~30 MB less with CDS. If I start the application with -Xmx64m, RSS is roughly the same with and without CDS, I suppose GC kicks in early on then and discards those byte arrays.

All in all, I think it's a pretty attractive improvement in cases where native images are no option. Not only for Quarkus, but also for things like Kafka, when restarted frequently during dev. The code is all on GitHub (linked in the post), in case you want to do your own comparisons or point out where I've messed up :)

--Gunnar

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2czS1qwDAgumafZZ9Yb9X%3DSf7bKnB8iP6SrWhTW5ucLPcQ%40mail.gmail.com.

[Gunnar Morling]

> I see larger RSS initially with CDS, but after a GC it's at the same level with and without CDS.

> It's a bit beyond my insight, but AFAICS it's largely byte[] data on the heap, which I reckon 

> originates from reading the memory-mapped CDS file.

Thanks to Flight Recorder and Mission Control, I understand what's happening now; it's interesting, as it's rather counter-intuitive:

Without CDS, the app needs to allocate more memory when starting up, and this causes some GCs to happen. Doing so, the G1 GC also uncommits unused heap, so RSS shown when stopping the program after a while is lower. Whereas with CDS, there's no GC happening which would trigger the release of memory back to the OS. I.e. it keeps the full 256M it allocated.

The smart folks working on G1 realized that it's not ideal that heap never gets uncommitted when no GC is happening, e.g. when an app goes idle after some phase of load. JDK 12 addresses this with JEP 346 (https://openjdk.java.net/jeps/346), which returns unused memory also without a GC happening. It's opt-in, and indeed, if I specify -XX:G1PeriodicGCInterval=..., RSS looks much better now with CDS. I should have used Shenandoah, where the same is happening by default (https://shipilev.net/jvm/anatomy-quarks/21-heap-uncommit/).

Mystery solved :) 

--Gunnar

[Gunnar Morling]

So with the latest OpenJDK 15 EA build 27, things look even better: 1s 115ms for time-to-first-response with AppCDS on my laptop, down from 2s 162ms on OpenJDK 14 without CDS. Lambda proxy classes are also archived now, so ~6.600 classes out of ~7.500 classes overall are loaded from the CDS archive now.

--Gunnar