Native-image support

[Kamil Toman]

Hi all,

I experimented lately with Graal's SubstrateVM to create executable images. One of the goals was a small application that accesses a database using a connection pool.

SubstrateVM is in the very early stage so most things don't work yet. Plus there is a big list of various limitattions that programs compiled into native code must work with.

I tried to adapt HikariCP to fit into SubstrateVM's closed-world assumptions and the good news is that it mostly works.

The biggest pain points are:

1) unsupported features in JDBC driver code,

2) dynamic loading of JDBC drivers,

3) reflective setup that includes dynamic classloading,

4) optional dependencies.

Let's look at those individually:

1) I found out that, unlike many others, PostgresQL JDBC driver mostly does work - the most problematic code only takes care of cleaning up badly handled resources. Limited but workable.

2) The driver code can be explicitly loaded as a dependency and manually registered. OK.

3) The reflective setup in HikariCP is a tough nut for Graal to break. But I figured out that if I just replace the dynamic code with some repetitive (and a bit ugly) static code Graal can compile the code without problems.

   See this commit.  It doesn't end there - the program must prepare and pass the datasource to the pool directly bacause of other limitations. OK.

4) Optional dependencies are another thing that Graal's native-image can't handle well. Even if you don't use those features native-image will try to compile them. For now I just stubbed those. TBD.

You can have a look at the small fork (just a few patches really) that can be compiled into a native binary, here https://github.com/leafclick/HikariCP.

And the small (clojure) program that uses it to run database migrations, here https://github.com/leafclick/pgmig.

What I'd like to see is a single version of HikariCP that would fully work in either traditional JVM or natively compiled code. The question is how to get there. What do you think?

Kamil

[Brett Wooldridge]

Hi Kamil,

First, let me say it is a great goal and I appreciate your investigating it.  I too am interested in Graal, but not for native-image generation.  Still, I understand your goal and motivation.

Having said that ... man is some of that code ugly! :-)  That is not a comment on your coding style, but a result of the necessary "unrolling" of the reflection.

I've made some comments on your code.  Note, this is not a committment to merge this, just my feedback on the code in a purely acedemic context.  Let's keep up the dialog.

Brett

[Kamil Toman]

Hi Brett!

Thanks for looking at those patches. In the meantime I looked into optional dependencies and I found out that if the pool only accesses them by reflection Graal's native-image doesn't go deep into that part of the dependency tree. But if you want them included you can add a configuration feature and it will pick them up.

With the latest Graal's support for incomplete classpath the process is much smoother.

I reworked previous patches and made them less aggressive (taking your comments into account). See https://github.com/leafclick/HikariCP/commits/dev

- "Actually use maven-compiler-plugin to set java source and target to 1.8." - a workaround the force IDEA to import the correct java versions; mvn seems to use the same settings (no change)

- "Replace some introspective code by static code." - manually unrolled config and config loading

- "Remove direct references to classes defined in optional dependencies." - only referencing HikariCP classes that don't import optional dependencies; lazy loading connectors in setters

The good news is that even a native image can be configured via standard configuration options now - preparing a pre-configured datasource is no longer needed if the JDBC driver is configured for reflection access. I updated PGMig accordingly (and it works).

Kamil