JVM-configuration for custom container images

[Marco Bungart]

Clement and I had a discussion today in zulip about configuring the JVM
within containers [1]. What I learned in the process was that a
run-java.sh exists in the Red Hat OpenJDK runtime image [2] that should
be used to start the java process, something like

    FROM registry.access.redhat.com/ubi8/openjdk-17@sha256:...
    ...
    ENV JAVA_APP_DIR="/path/to/directory/containing/the/jar"
    ENV JAVA_APP_JAR="${APP_DIR}/jarfile-to-run.jar"
    ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]

The script /opt/jboss/container/java/run/run-java.shsets a lot of JVM
properties, optimizing the overall performance. Problem is: the script
is not standalone, it incorporates other scripts.

---

Current state:

If an application uses the aforementioned Red Hat image (correctly), the
optimizations through run-java.sh are applied. If an application uses
jib to build containers, the optimizations are also applied. All other
means of producing an image do not profit from those optimizations.

---

Points I would like to discuss:

- As far as Clement is aware, there is no documentation that the
run-java.sh script should be used if the Red Hat image is used. Should
we update the documentation to highlight the correct usage? And how much
do we need to document here? Just to given you an example:
    - The collection of scripts evaluates a bunch of environment
variables (e.g. GC_MIN_HEAP_FREE_RATIO) and "transforms" them to
corresponding -XX:... parameters. If those env vars are not present,
default values are set by the script (e.g. 10 for GC_MIN_HEAP_FREE_RATIO).
    - For other parameters (e.g. JAVA_[INITIAL|MAX]_MEM_RATIO, which
are mapped to -XX:[Initial|Max]RAMPercentage), there are no explicit
defaults set, so implicit defaults from the JVM take effect.
    - And then there is the case that corresponding parameters (e.g.
JAVA_MIN_MEM_RATIO, which should map to XX:MinRAMPercentage, which is
different from XX:XX:InitialRAMPercentage) is not considered at all.

- I understand that Quarkus is a Red Hat product and integrates best
with other Red Hat products. There are, however, many reasons why one
would or could not use the UBI image, e.g. company policy. Thus, it
would be desirable that a set of scripts, achieving the same
optimizations, is provided that can be used in custom images. Ideally,
those scripts should be hosted in a separate repository, under the
quarkus project. Even more ideally, the repository should produces a
.tar.gz artifact that can be ADDed to a containerfile via its URI.

- Do you see a way to make a (simplified) version of this work with
containers that do not have a shell, e.g. google's distroless images
[3]? The challenge here is that - as far as I am aware - the JVM options
are not bound to environment variables. Since we do not have a shell, we
can not use parameter substitution. Fixating parameters at buildtime is
easy through ARGs. Making them configurable at container start time is a
challenge. The only thing that comes to my mind is defining an ENV
JAVA_DEFAULT_OPTS with sensible default values and let users override it
when needed (they'd have to override the variable "as a whole", since no
parameter substitution is available).

Happy to hear your opinions on the matter :)

[1]:
https://quarkusio.zulipchat.com/#narrow/stream/187030-users/topic/JDK.2021/near/395496426
[2]:
https://catalog.redhat.com/software/containers/ubi8/openjdk-17-runtime/618bdc5f843af1624c4e4ba8
[3]: https://console.cloud.google.com/gcr/images/distroless/GLOBAL/java17

[Georgios Andrianakis]

cc  Jonathan Dowland

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/ac8a00a9-3639-4876-812b-071d2088568f%40googlemail.com.

[Max Rydahl Andersen]

> cc Jonathan Dowland
>
> On Sun, Oct 8, 2023 at 4:52 PM 'Marco Bungart' via Quarkus Development
> mailing list <quark...@googlegroups.com> wrote:

sorry for late reply I missed this at first post.

>> Clement and I had a discussion today in zulip about configuring the JVM
>> within containers [1]. What I learned in the process was that a
>> run-java.sh exists in the Red Hat OpenJDK runtime image [2] that should
>> be used to start the java process, something like
>>
>> FROM registry.access.redhat.com/ubi8/openjdk-17@sha256:...
>> ...
>> ENV JAVA_APP_DIR="/path/to/directory/containing/the/jar"
>> ENV JAVA_APP_JAR="${APP_DIR}/jarfile-to-run.jar"
>> ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]
>>
>> The script /opt/jboss/container/java/run/run-java.shsets a lot of JVM
>> properties, optimizing the overall performance. Problem is: the script
>> is not standalone, it incorporates other scripts.
>>
>> ---
>>
>> Current state:
>>
>> If an application uses the aforementioned Red Hat image (correctly), the
>> optimizations through run-java.sh are applied. If an application uses
>> jib to build containers, the optimizations are also applied. All other
>> means of producing an image do not profit from those optimizations.

"all other" means "you do it manually" or are there some place you would expect these optimisations takes place?

>> ---
>>
>> Points I would like to discuss:
>>
>> - As far as Clement is aware, there is no documentation that the
>> run-java.sh script should be used if the Red Hat image is used.

Just so I have my assumptions right here - by red hat image you mean the UBI image?
These are freely and openly available and not tied to Red Hat per se - this could
be any image

And it is a SHOULD, not a MUST afak - as a way to optimize best to what the UBI openjdk image recommends.
Whether you apply those or not is up to you. Not something we can guarantee.

>> Should
>> we update the documentation to highlight the correct usage?

+1 on updating doc to make it clearer.

>> And how much
>> do we need to document here? Just to given you an example:
>> - The collection of scripts evaluates a bunch of environment
>> variables (e.g. GC_MIN_HEAP_FREE_RATIO) and "transforms" them to
>> corresponding -XX:... parameters. If those env vars are not present,
>> default values are set by the script (e.g. 10 for GC_MIN_HEAP_FREE_RATIO).
>> - For other parameters (e.g. JAVA_[INITIAL|MAX]_MEM_RATIO, which
>> are mapped to -XX:[Initial|Max]RAMPercentage), there are no explicit
>> defaults set, so implicit defaults from the JVM take effect.
>> - And then there is the case that corresponding parameters (e.g.
>> JAVA_MIN_MEM_RATIO, which should map to XX:MinRAMPercentage, which is
>> different from XX:XX:InitialRAMPercentage) is not considered at all.

On first hunch - these looks like things the UBI java image should document.

I don't see that as something we rely on in higher layers of Quarkus - do you ?

>> - I understand that Quarkus is a Red Hat product and integrates best
>> with other Red Hat products.

I'll be pedantic here and state Quarkus is not a Red Hat product, Quarkus is a project.
Red Hat Build of Quarkus is a product and that contains other defaults
that makes sense for Red Hat products.

That said - yes, we use the UBI images because we a) find that since those images
already contain optimizaitons for how to best run java in a container they are easier to use
b) it means we have at least one image we can point to and say "if you do your own images/container setup
look at this if you want to know the various java options"

>> There are, however, many reasons why one
>> would or could not use the UBI image, e.g. company policy. Thus, it
>> would be desirable that a set of scripts, achieving the same
>> optimizations, is provided that can be used in custom images. Ideally,
>> those scripts should be hosted in a separate repository, under the
>> quarkus project. Even more ideally, the repository should produces a
>> .tar.gz artifact that can be ADDed to a containerfile via its URI.

In principle I agree - especially to the latter one. That it would be nice to have
such "library" of "how to start java in a container"...the issue is that
it really gets hairy and depends on java version, OS vendor and java vendor ...
and in some cases Quarkus and GraalVM version too.

Thus not sure how generic we can make it.

>> - Do you see a way to make a (simplified) version of this work with
>> containers that do not have a shell, e.g. google's distroless images
>> [3]? The challenge here is that - as far as I am aware - the JVM options
>> are not bound to environment variables. Since we do not have a shell, we
>> can not use parameter substitution. Fixating parameters at buildtime is
>> easy through ARGs. Making them configurable at container start time is a
>> challenge. The only thing that comes to my mind is defining an ENV
>> JAVA_DEFAULT_OPTS with sensible default values and let users override it
>> when needed (they'd have to override the variable "as a whole", since no
>> parameter substitution is available).

Yeah, so you are nicely outlining the (Im)possibilty of making this generic -
when you have no shell you are more or less toast.

In your case you would need those parameters to be "burned" into the dockerfile or ?

That said - this issue of generation of some config based on your actual project
is not unique to these settings. it applies to dockerfile, tekton script, GitHub actions,
you name it.

I don't have a solution off hand but I do think it feels like parts of this could at least
be considered for the issue related to improving how we (re)generate config based on the project
rather than statically upfront: see https://github.com/quarkusio/quarkus/issues/24502

/max