Convenience for loading the Libraries from a Jar, for each OS

[Support]

Greetings and compliments.

I am looking for some guidance and ideas about deploying a Fat Multi-platform Jar to Maven, using GitHub and  supporting at least  Linux, Windows and MacOS 64 bit.

Using the Gradle `cpp-library` plugin, I will get my libraries in folder like `fpng/build/lib/main/debug/shared/linux/x86-64/libfpng.so`.

So I have a Gradle `copy` task to copy that to the JNA project:

task copyNativeLib(type: Copy) {
    dependsOn assemble
    from './build/lib/main/debug/shared'
    from './build/lib/main/release/shared'
    into '../fpng-java/build/resources/main/lib'
    include '**/*.dll', '**/*.so', 'build/lib/**/*.dylib'  // Adjust for the native library file extensions on different platforms
}

This works quite well and is ok (although I wonder, why Gradle does not provide any tooling for C/Native into Java project situations).

When building my JNA library, it contains a folder /lib with all the possible libraries for Linux, Windows and MacOS. However, based on Gradle's `cpp-library` that folder is not flat but contains sub-folder for each architecture, e. g. `linux/x86-64/libfpng.so`.

If I understand the situation correct, then I need to find this `lib` folder in the JAR file and copy all the libraries into the filesystem (e.g. `/tmp`). This can be done of course, see below.

But now it gets hairy. It looks like `Native.load()` wants to know exactly, which library to load, e.g. `/tmp/fpng-java/linux/x86-64/libfpng.so`.

In order to achieve this, I will need to know the parts `linux`, `x86-64` and `.so` while I decide about `/tmp/fpng-java` and `libfpng` (only).

I ended up with a quite stupid `if .. then .. else` based on `System.getProperty()` and it felt a lot like re-inventing the wheel.

So I wonder, why JNA does not provide more convenience and tooling around this problem? Why am I supposed to write this kind of boilerplate code every time for myself? (I don't mind writing it, but I do not know about the details of the other platforms Windows, MacOS, let alone BSD, ARM and whatever).

Please advise, thank you in advance.

Andreas

PS: I know about https://github.com/scijava/native-lib-loader but it does not even allow to set the source directories or to react to the Gradle `cpp-library` output, also it seems to be dormant.

// copy a folder with all its content from inside the JAR to a filesystem destinantion (e.g. '/tmp/)
static void extractFilesFromURI(URI uri, String target) throws IOException {
    Path uriPath = Paths.get(uri);
    File targetFolder = new File(target);
    targetFolder.deleteOnExit();

    Files.walkFileTree(uriPath, EnumSet.noneOf(FileVisitOption.class), Integer.MAX_VALUE, new SimpleFileVisitor<Path>() {
        @Override
        public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
            Files.copy(file, targetFolder.toPath().resolve( uriPath.relativize(file) ) , StandardCopyOption.REPLACE_EXISTING);
            return FileVisitResult.CONTINUE;
        }
        @Override
        public FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {
            return FileVisitResult.CONTINUE;
        }
        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
            Files.createDirectories( targetFolder.toPath().resolve( uriPath.relativize(dir)));
            return FileVisitResult.CONTINUE;
        }
    });
}

String osName = System.getProperty("os.name");
String osArch = System.getProperty("os.arch");

String resourcePath = "/lib";
String extension = ".so";
String targetFolder = System.getProperty("java.io.tmpdir") + File.separator + FPNGEncoder.class.getSimpleName() + File.separator;

try {
    extractFilesFromURI(FPNGEncoder.class.getResource("/lib").toURI(), targetFolder);
} catch (IOException e) {
    throw new RuntimeException(e);
} catch (URISyntaxException e) {
    throw new RuntimeException(e);
}

// linux/x86-64
// linux/x86
// windows/x86-64
// macOS/x86-64
if ( osName.equalsIgnoreCase("linux") ) {
    targetFolder += "linux";
} else if ( osName.equalsIgnoreCase("windows") ) {
    targetFolder += "linux";
    extension = ".dll";
} else if ( osName.equalsIgnoreCase("macos") ) {
    targetFolder += "macos";
    extension = ".dylib";
}

if ( osArch.contains("64") ) {
    targetFolder += "/x86-64";
} else {
    targetFolder += "/x86-32";
}

FPNGEncoder ENCODER = (FPNGEncoder) Native.load(targetFolder + "/libfpng" + extension, FPNGEncoder.class);

[Tres Finocchiaro]

So I wonder, why JNA does not provide more convenience and tooling around this problem?

It does! It's normally as easy as copying JNA's directory structure and JNA will automatically extract the native library for you, quoting.

> If bundled in a jar file, the resource will be extracted to jna.tmpdir for loading, and later removed (but only if jna.nounpack is false or not set).

https://java-native-access.github.io/jna/5.10.0/javadoc/index.html

The path pattern:

https://github.com/java-native-access/jna/blob/a4aca64973061cf7d6c9c21e031683340c674e92/src/com/sun/jna/Platform.java#L307

An example project which uses this pattern and relies on the automatic extraction:

https://github.com/gary-rowe/hid4java/tree/develop/src/main/resources

However, based on Gradle's `cpp-library` that folder is not flat but contains sub-folder for each architecture, e. g. `linux/x86-64/libfpng.so`.

You probably simply need to create a simple remapping  to move these into where JNA would expect them, e.g.

• linux/x86-64/libfpng.so -> linux-x86-64/libfpng.so
• windows/x86/libfpng.so -> win32-x86/libfpng.so
• ... etc

[Support]

On Sun, 2023-11-19 at 00:22 -0500, Tres Finocchiaro wrote:

So I wonder, why JNA does not provide more convenience and tooling around this problem?

It does! It's normally as easy as copying JNA's directory structure and JNA will automatically extract the native library for you, quoting.

> If bundled in a jar file, the resource will be extracted to jna.tmpdir for loading, and later removed (but only if jna.nounpack is false or not set).

https://java-native-access.github.io/jna/5.10.0/javadoc/index.html

Thank you Tres,

Linux

libX11.so

X11

unfortunately I missed that "libX11" vs. "X11" part and that's why it always failed to work.

You probably simply need to create a simple remapping  to move these into where JNA would expect them, e.g.

• linux/x86-64/libfpng.so -> linux-x86-64/libfpng.so
• windows/x86/libfpng.so -> win32-x86/libfpng.so
• ... etc

Yes, and this is the ugly part -- especially when building a Multi OS JAR on GitHub.

I know, JNA is not to blame since it was there before Gradle `cpp-lib`. Though it's rather unfortunate that both tools ignore each other.

I still don't want to have a fragile mapping logic inside the Gradle Copy task.

So I ended up in a savage rewrite of `Native.load()` which adheres to the Gradle `cpp-lib` structure: https://github.com/manticore-projects/fpng-java/blob/main/encoder-java/src/main/java/com/manticore/tools/Encoder.java

Lets see what happens on the various platforms (especially what fpng.dll gradle will build on Windows, is it libfpng.dll or fpng.dll).

Thank you for your input and help.

Cheers

Andreas

[Tres Finocchiaro]

I still don't want to have a fragile mapping logic inside the Gradle Copy task. [...] So I ended up in a savage rewrite 

It seems trivial to me, but this second answer has a comment which seems to share your sentiments.

• https://stackoverflow.com/questions/46908376
  
• https://stackoverflow.com/a/51029487/3196753

Perhaps the JNA project would be receptive to a PR which provides a gradle-cpp-compatible pattern as an optional resolver to getNativeLibraryResourcePrefix.  I've worked with other projects that use different patterns and strings too, such as the os-maven-plugin.  I believe most downstream projects just work around these differences.

The other option would be to propose the option to Gradle to offer a JNA-compatible structure (or the ability to customize).

[Support]

Tres and All.

On Sun, 2023-11-19 at 01:46 -0500, Tres Finocchiaro wrote:

Perhaps the JNA project would be receptive to a PR which provides a gradle-cpp-compatible pattern as an optional resolver to getNativeLibraryResourcePrefix.

We could introduce an optional Flavour to Native.load(...), with one Flavour.GRADLE_CPP. I would be most willing to provide such an implementation.

Would the JNA be willing to look at such a PR? Please advise.

  I've worked with other projects that use different patterns and strings too, such as the os-maven-plugin.  I believe most downstream projects just work around these differences.

I believe that too, but I find it a kind of sad when 1000 projects re-invent the wheel especially when Multi Arch is involved and one single person won't have full access to all architectures.

For such a Multi Arch problem, the hive may find the most comprehensive solution.

The other option would be to propose the option to Gradle to offer a JNA-compatible structure (or the ability to customize).

Yes and I have no idea, why they did not consider that.

It appears that Gradle Java and Gradle C/CPP are completely disjoint. There is not even an option or guide for a JNI/JNA build. Instead you have to setup 2 projects and then copy over the Native Library to your Java project (which is kind of easy, as long as you don't need to identify the particular library).

Please fee back if such a PR has a chance and will receive feedback.

Thank you and cheers

Andreas

[Tres Finocchiaro]

> We could introduce an optional Flavour to Native.load(...), with one Flavour.GRADLE_CPP.

I can't speak on behalf of the team, but making it flexible would be my vote, e.g.

    DefaultNativeResourcePrefix extends NativeResourcePrefix {
        @Override
        resolve(int osType, String arch, String name) {
            String osPrefix;
            arch = getCanonicalArchitecture(arch, osType);
            switch(osType) {
                case Platform.ANDROID:
                    if (arch.startsWith("arm")) {
                        arch = "arm";
                    }
                    // ...
        }
    }

... and then Native.setNativeResourceResolver(new DefaultNativeResourcePrefix())

... and then getNativeLibraryResourcePrefix would fallback to default if not provided.

With regards to having GRADLE_CPP support out of the box or having boilerplate 3rd party code would depend on the stability of the 3rd party conventions, otherwise JNA would be maintaining this volatility downstream.

[Matthias Bläsing]

Hi,

Am Sonntag, dem 19.11.2023 um 13:23 -0500 schrieb Tres Finocchiaro:
> > >  We could introduce an optional Flavour to Native.load(...), with one Flavour.GRADLE_CPP.
>
> I can't speak on behalf of the team, but making it flexible would be my vote, e.g.

I don't see a value here. It is trivial to place binaries in the right
path inside a jar so that the current mechanism works. It is a one time
cost to do it while building the jar.

In the worst case you can add a task to the CI/CD pipeline, that
combines the shell tools "unzip", "mv" and "zip" to do that move. But I
though that the whole gradle coolness was about flexibility and I can
imagine doing it in maven, so it should be doable in gradle (remapping
files in a JAR).

From my perspective the trend goes to move things that can be done at
compiletime, to compile time and do at runtime only things that have to
be done then.

Greetings

Matthias

[Tres Finocchiaro]

I don't see a value here. It is trivial to place binaries in the right path inside a jar so that the current mechanism works. It is a one time cost to do it while building the jar.

My experience with projects like Maven is that developers are told to use "convention over configuration", so a proposal to add support for a particular convention is objectively valuable.

- Tres.Fin...@gmail.com

[Support]

Tres and Matthias.

compliments and thank you for responding.

On Sun, 2023-11-19 at 13:23 -0500, Tres Finocchiaro wrote:

With regards to having GRADLE_CPP support out of the box or having boilerplate 3rd party code would depend on the stability of the 3rd party conventions, otherwise JNA would be maintaining this volatility downstream.

Very true unfortunately. In a perfect world JNA would define `NativeResourcePrefix` and `DefaultNativeResourcePrefix` but Gradle `cpp-library` would implement and maintain its flavor.

Guess we will just continue re-inventing the wheel in every mixed Java/C project for while. Apparently not enough demand, no guaranteed stability (Gradle is terrible in this regard) and also Matthias did not sound too enthusiastic about it. 

On Sun, 2023-11-19 at 19:29 +0100, 'Matthias Bläsing' via Java Native Access wrote:

It is trivial to place binaries in the right

path inside a jar so that the current mechanism works. It is a one time

cost to do it while building the jar.

I am sorry, but its not trivial when you want to build a Multi OS Fat Jar. At least you have to investigate a) the output convention of `cpp-library` and then investigate b) the convention of `JNA`.

At least I failed with that (not understanding, that `JNA` expects `fpng` instead of `libfpng`). And I still don't know yet if on Windows `cpp-library` will produce `fpng.dll` or `libfpng.dll`.

Of course I will figure that eventually, but I also will have forgotten that in 6 months from now.

In the worst case you can add a task to the CI/CD pipeline, that

combines the shell tools "unzip", "mv" and "zip" to do that move. But I

though that the whole gradle coolness was about flexibility and I can

imagine doing it in maven, so it should be doable in gradle (remapping

files in a JAR).

It can be done of course and Gradle can do that efficiently compared to Maven (you asked for it 🙂).

However, I really don't like bloated build files. And I really don't want to maintain such a mapping which i will have just forgotten about soon.

From my perspective the trend goes to move things that can be done at

compiletime, to compile time and do at runtime only things that have to

be done then.

I agree: Gradle should provide a `JNA output flavor` as part of their `C/CPP` plugin. But as I wrote above, they don't care about mixing Java with C code.

Maybe I need to write an extra Plugin "cpp-library-jna". Right now my own implementation of `Native.load(...)` reading the `cpp-library` convention will do it (for my 4 architectures).

Thank you for insight and opinion, we can put that to a rest (until I will dive into the "cpp-library-jna" Gradle plugin).

Cheers

Andreas