JPMS (java module system) and JNA - prototype available

[Matthias Bläsing]

Hi,

it was now repeatedly stated, that people want module descriptors in
JNA and this turn JNA artifacts into named modules.

At this point in time JNA offers automatic module names, but that is
not enough to satisfy all users, especially the new packaging options
need named modules to function correctly.

So it is a no-brainer to add a module-info.class and be done with it,
right? It is not that easy. A module-info.class is a java class and
there are classpath scanners out there, that bail out, when they find
classes they don't understand, thus adding a jar with a module
descriptor on the classpath can break existing applications.

But Java 9 added another feature: multi release JARs. MRJARs allow to
provide different classes for different java versions. This is
accomplished by putting the class files into the directory:

META-INF/versions/<major-version>/<package-structure>

As in the past META-INF was not a location class files may be placed,
adding the module-descriptor as:

META-INF/versions/9/module-info.class

and marking the JAR as MRJAR should be save.

Actually no it is not. There are class path scanners out there, that
also scan META-INF and bail out, when finding the module-info.class.

Given that JNA currently is backwards compatible going back to 6
(untested, but at least intentionally), all the ugliness of the non
enforced rules of the java platform can hunt us. So instead of breaking
with the past, lets prepare for the future:

Add new artifacts, that are identical to the original JNA jars and add
a module-class.info at the safest location.

The prototype can be found here:

https://github.com/java-native-access/jna/pull/1237

The artifacts are prepared to be pushed to maven central in addition to
the base artifacts and can be accessed by specifying the qualifier
"jpms".

I'll attach prebuild binaries to the PR shortly, install in a local
maven repository run "ant install" from the jna directory.

Testing would be great.

Matthias

[Daniel B. Widdis]

I'll attach prebuild binaries to the PR shortly, install in a local
maven repository run "ant install" from the jna directory.
Testing would be great.

Works great on oshi (tested on Windows so far, will follow up with macos shortly).  For anyone else trying to do this in maven rather than ant:

1. Unzip the jars from the PR into a /lib folder.

2. Change JNA dependencies to:

<dependency>

<groupId>net.java.dev.jna</groupId>

<artifactId>jna-jpms</artifactId>

<version>${jna.version}</version>

<scope>system</scope>

<systemPath>${basedir}/lib/jna-jpms.jar</systemPath>

</dependency>

<dependency>

<groupId>net.java.dev.jna</groupId>

<artifactId>jna-platform-jpms</artifactId>

<version>${jna.version}</version>

<scope>system</scope>

<systemPath>${basedir}/lib/jna-platform-jpms.jar</systemPath>

</dependency>

3. Then in the module-info:

    requires com.sun.jna;

    requires com.sun.jna.platform;

This is exciting, as I'll be able to do something similar for my downstream project eventually!

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/8efeeec66c8d1e5ea6342b27ac9d4967b6494579.camel%40doppel-helix.eu.

--

Dan Widdis

[Daniel B. Widdis]

Tested on macOS, with JDK14+module.info, JDK14 no modules, JDK8.

No problems on Windows JDK11+module.info, JDK 8.

Struggling with JDK11 with no module.info class, seems the imported module ends up defining a module path including junit, that seems to not play nice with my own test class.  Could be a configuration thing.  Not sure why macOS is different that way.

--

Dan Widdis

[Matthias Bläsing]

Hi,

Am Montag, den 20.07.2020, 14:34 -0700 schrieb Daniel B. Widdis:
> Struggling with JDK11 with no module.info class, seems the imported
> module ends up defining a module path including junit, that seems to
> not play nice with my own test class. Could be a configuration
> thing. Not sure why macOS is different that way.

I'm not sure, that I parse this correctly. mac OS (aka darwin) needs
special treatmeant, because for all other architectures, the name
contains a hyphen. The hyphen invalidates package names and thus the
JPMS does not encapsulate the resources. darwin is the only
architecture affected by JPMS encapsulation for native libraries.

This might affect loading libraries from the classpath as JNA needs to
be able to access the resources.

Can you provide a log for the run? (with jna library load debgging
activated).

Greetings

Matthias

[Daniel B. Widdis]

I'm not sure, that I parse this correctly. 

I'm not sure I explained it correctly.  I had no problems at all on macOS.  I was able to use the -jpms JARs without any problems, either in JDK8 (blissfully unaware of modules), JDK14 without any module-info setup, and JDK14 with a full module-info setup.

My only problem was on Windows with JDK 11, without a module-info file/class, and only in regards to junit classes, and in the context of the Eclipse IDE which refused to compile the test classes.  I suspect some sort of classpath (unnamed module) vs. module path interaction that I probably could have fixed given enough time and/or another IDE, etc., that I did not pursue.

My comment was to wonder why whatever configuration issue I was having on Windows was not a problem on macOS.

Can you provide a log for the run? 

I couldn't even get anything to run (from Eclipse) in that situation, so this doesn't apply. But perhaps with the explanation above, this question is not relevant anymore.

 

 

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/7fc840cd61abbf8ca841321be4cb9ff69cbc96d6.camel%40doppel-helix.eu.

--

Dan Widdis

[Matthias Bläsing]

Hi Daniel,

thank you very much for testing! I expect issues to come up, which is
the reason I only intent to advertise it only as experimental.

Greetings

Matthias

[Daniel B. Widdis]

> thank you very much for testing! I expect issues to come up, which is the reason I only intent to advertise it only as experimental.

I fully agree here.  I've been putting off doing anything on my own project since it didn't matter what I did if my dependencies weren't modularized.  Your experiment got me to start trying out various setups to do similar...

Which I realized are crazily hard to do within my own project while trying to maintain compatibility and the ability to run tests on CI systems, given the relatively easy alternative that other projects have to use Moditect to just alter my JAR.  It seems that, particularly with regards to test-only dependencies, the Java community still hasn't rallied around any sort of a standard.  

While I don't mind playing with this for my own education, I don't think it should in general be the responsibility of project maintainers who want to retain JDK8 compatibility to do more than trivial changes (e.g., automatic module name).  Although happy to accept a PR from any user who wants it that badly..

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/4ad549c50f0f0a9348ca5693d80d8b93ec1378b5.camel%40doppel-helix.eu.

--

Dan Widdis

[Matthias Bläsing]

Hi,

thank you again for testing Daniel. If noone objects, I'll merge the PR
in the next few days. It will not get better and when it is merged,
I'll also push snapshot artifacts to the OSSRH snapshot repository, so
that people can test easier.

Greetings

Matthias

[Daniel Widdis]

So I'm trying a modular build and encountering an exception (Structure doesn't have public no-arg constructure) that I think may have to do with module path.

Was this -jpms artifact supposed to make it onto Maven? Is it still just a snapshot?

I can probably work around it with ModiTect, just was wondering if I missed something about where I'm supposed to find it.

[Matthias Bläsing]

Hi Daniel,

Am Samstag, den 13.02.2021, 18:06 -0800 schrieb Daniel Widdis:
> So I'm trying a modular build and encountering an exception
> (Structure doesn't have public no-arg constructure) that I think may
> have to do with module path.

it would be helpful to see full code (preferably a minimal sample that
reproduces the probelm).

> Was this -jpms artifact supposed to make it onto Maven? Is it still
> just a snapshot?

It got intentionally onto maven central. I created the artifact because
it was demanded and people have the tendency not to test until it is in
the wild. The jna-5.7.0.jar and jna-5.7.0-jpms.jar are identical apart
from:

- jna-5.7.0.jar has an Automatic-Module-Name entry in the MANIFEST
- jna-5.7.0-jpms.jar has a fill module descriptor in
jpms/META-INF/versions/9/module-info.class


The -jpms artifact was created, so that broken classpath scanners don't
get tripped. It was reported, that there are classpath scanners out
there, that scan META-INF and fail hard when finding classes with an
unsupported Java version (the module descriptor is class file version
53 (java 9)).

If your question is how to get the artifact, than the jpms articat is
deployed along side the normal artifact. For a maven build this would
be:

<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>5.7.0</version>
<classifier>jpms</classifier>
<exclusions>
<exclusion>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
</exclusion>
</exclusions>

</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>

<artifactId>jna</artifactId>
<version>5.7.0</version>
<classifier>jpms</classifier>
</dependency>

Two things are relevant here:

The classifier is specified to switch to the JPMS artifact and I
notices, that the dependency resolution of maven creates a problem -
the platform jpms artifact uses the same dependency as the non-jpms
artifact, which is wrong. I worked around that above with an exclusion.
I think we'll need to create separate artifacts from this.

Please note: This is untested, so feel free to ask if you see something
problematic.

Greetings

Matthias

[Daniel Widdis]

Matthias,

Thank you for the detailed explanation and example Maven dependency. It was the use of the "classifier" that I was missing. I was looking for jna-jpms-5.7.0. Your answer gives me the information I need.... now to figure out how to do the same with my own project!

I was able to resolve my other issue; I was unaware that if I had a class extending JNA's Structure class that I needed to export that class (to JNA) in order for it to be instantiated, so adding lines to my module-info of the form "exports oshi.jna.platform.mac to com.sun.jna;" was necessary.

Dan

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/ab02bd351187ca2e0329566804904f189178cfda.camel%40doppel-helix.eu.

[Daniel Widdis]

As a follow-up, it looks like I have most things working. (The things that aren't working aren't JNA-related.)

For the benefit of anyone else wanting to do similar to what I did (which may not be the best approach but worked for me):

1. I created a new branch "java11". (Reasoning: while now it's just the module-info, in the future I may leverage new features to improve code efficiency.)

2. I modified the pom.xml files to rename the parent and core artifacts adding -java11 to them. (Since I left the directory structure intact, I also needed to specify the site.url parameter in distributionManagement in both parent and core maven modules.)

3. I used the Maven dependency code indicated below.

4. I added my own module-info.java with "requires com.sun.jna;" and "requires com.sun.jna.platform;". I needed to also export my packages that included classes with "extends Structure" to com.sun.jna to permit reflection to instantiate them. (There may be a better way to do this.)

I was then able to build and deploy new artifacts to Maven Cenrtral.

Thanks again for the detailed answer.

[Neil C Smith]

On Mon, 15 Feb 2021 at 04:59, Daniel Widdis <wid...@gmail.com> wrote:
> I needed to also export my packages that included classes with "extends Structure" to com.sun.jna to permit reflection to instantiate them. (There may be a better way to do this.)

I would have thought using "opens to" rather than "exports to" might
be the better thing to have as general documentation here for full
compatibility? There are at least some uses of setAccessible in JNA
code base which might behave differently otherwise?

Aside - I appreciate the use of the jpms classifier - been amused over
the last few weeks by various JDK people trying to stop the use of
that acronym - a ship that seems to have well and truly sailed!

Best wishes,

Neil


--
Neil C Smith
Codelerity Ltd.
www.codelerity.com

Codelerity Ltd. is a company registered in England and Wales
Registered company number : 12063669
Registered office address : Office 4 219 Kensington High Street,
Kensington, London, England, W8 6BD

[Daniel Widdis]

>> I needed to also export my packages that included classes with "extends Structure" to com.sun.jna to permit reflection to instantiate them. (There may be a better way to do this.)

> I would have thought using "opens to" rather than "exports to" might be the better thing 

Note I did say there may be a better way! 

I had the same thought regarding "opens to" but reasoned that JNA only ever accesses the public members of a structure, and "exports to" provides a lower level of access.  From a security standpoint of "least privileged access" it seemed the better choice to me.

> ...  to have as general documentation here

Once we settle on the best choice, it'd probably be useful to add some of this how-to (the maven dependencies and this exports/opens to... requirement) to the main project documentation somewhere.

> JDK people trying to stop the use of that acronym  

That makes no sense. It's the only way to describe this particular type of module without confusing it with OSGi modules or Maven modules or <insert any other modules that existed before Jigsaw>.  And Jigsaw was a temporary name.  So either JPMS or the full word spelled out is the only sane choice.

[Daniel Widdis]

Reading more about this, I do see the benefit of “opens” in restricting to runtime-only access (including private members via reflection), while “exports” would allow JNA to create a dependency on my project and write code at compile time, but only allow access to public (and protected via inheritance) members.  Seems either choice gives more access than is needed, but I suppose given the reflective runtime access, “opens” does seem to more logically align with the needed access.

--

You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/b4f4b84a-5167-4f41-a37d-1c05875bd8c8n%40googlegroups.com.

[Neil C Smith]

On Mon, 15 Feb 2021 at 15:47, Daniel Widdis <wid...@gmail.com> wrote:
> > I would have thought using "opens to" rather than "exports to" might be the better thing
>
> Note I did say there may be a better way!

I know, that's actually why I responded ...

> I had the same thought regarding "opens to" but reasoned that JNA only ever accesses the public members of a structure, and "exports to" provides a lower level of access. From a security standpoint of "least privileged access" it seemed the better choice to me.

There are other bits of JNA that might need opens though - I wasn't
thinking specifically about Structure. eg. there are a few uses of
setAccessible, and doesn't the default methods support require private
lookup?

> Once we settle on the best choice, it'd probably be useful to add some of this how-to (the maven dependencies and this exports/opens to... requirement) to the main project documentation somewhere.

I'm not sure there is a best choice, and it's possibly a case where
migrating pre-JPMS code vs writing new JPMS-only code could also
suggest different defaults. Certainly reflection and JPMS sometimes
causes some surprises!

> > JDK people trying to stop the use of that acronym
>
> That makes no sense. It's the only way to describe this particular type of module without confusing it with OSGi modules or Maven modules or <insert any other modules that existed before Jigsaw>. And Jigsaw was a temporary name. So either JPMS or the full word spelled out is the only sane choice.

Tell me about it! But then you see things like
https://twitter.com/pressron/status/1356262219886039048

[Daniel Widdis]

> There are other bits of JNA that might need opens though ... there are a few uses of setAccessible, and doesn't the default methods support require private lookup?

If so, that makes "opens to..." the clear choice. I had meant to write something about "requirements other than Structure" in my previous email but it was pre-coffee.

> I'm not sure there is a best choice, and it's possibly a case where migrating pre-JPMS code vs writing new JPMS-only code could also suggest different defaults.

True. Perhaps just document the options and let the users decide?

In my use case I plan to support JDK8 for the very long term, so "migrating" is appropriate for me, but I still have to choose new artifact vs. MRJAR vs. classifiers vs. other methods and I'm not sure there's a solid consensus yet around any of them as a best choice.

--
You received this message because you are subscribed to the Google Groups "Java Native Access" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jna-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jna-users/CAPxOS5Fya43t8wU6_GWwCb4MHBAUFTiPdGygR9dT6kw5W64Vyg%40mail.gmail.com.

[Daniel Widdis]

Another thought. I wondered why jna-platform Structure subclasses didn't have the same problem. The jna (jpms) artifiact includes "exports com.sun.jna.internal to com.sun.jna.platform" which may or may not be relevant. I can't see any other obvious difference.

I'll throw together a documentation PR tonight that tries to summarize this and ask for your review/edit.

[Daniel Widdis]

Neil, I'd appreciate your review of  Add JPMS artifact details to FAQ by dbwiddis · Pull Request #1315 · java-native-access/jna (github.com)  

[Neil C Smith]

On Tue, 16 Feb 2021 at 18:31, Daniel Widdis <wid...@gmail.com> wrote:
> Neil, I'd appreciate your review of Add JPMS artifact details to FAQ by dbwiddis · Pull Request #1315 · java-native-access/jna (github.com)

Looks good - just added a couple of comments. Not sure I'm
necessarily the most important or knowledgeable person to review
though! Been a while since I've looked that in depth into JNA's
internals.

Incidentally, looking back at the only project I've done that was
modulepath only with JNA (ironically another, partial GStreamer
binding for a client), that did only use `exports`.

Aside from migrating existing projects, probably another time to use
`opens` is with stuff that's designed to be used on either classpath
or modulepath. JPMS is great for exponentially increasing the number
of things you need to test! :-)