Support Java 9 for jpf-core - Discussion

[Gayan Weerakutti]

I'll try keep my queries and progress updates related to the project "Support Java 9 for jpf-core",  in here.

I ran a clean build with:
OpenJDK Runtime Environment (build 9-Debian+0-9b181-4bpo91)
OpenJDK 64-Bit Server VM (build 9-Debian+0-9b181-4bpo91, mixed mode)   

And it resulted in 5 errors, and 91 warnings
Full log: https://pastebin.com/5SSM2MZM

As the initial step, I'm thinking of fixing those errors, so that it compiles with JDK 9. I'll make distinct commits, so one could review if needed. Looking forward to your replies.

[Cyrille Artho]

Thank you very much, we really appreciate these efforts!

For the use of BCEL, it's probably enough to change com.sun.org.apache.bcel.internal.generic to org.apache.bcel.internal.generic.

For the other classes that are no longer available, it will take some digging in the documentation to find out where it is now (or if a replacement has to be used).

[Gayan Weerakutti]

With the new module system things are bit different. When replacing a class, it requires that we explicitly specify the name of the module that it belongs, like:

   javac --patch-module java.base=src -d mypatches/java.base \
        src/java.base/java/util/concurrent/ConcurrentHashMap.java

Or it will show errors:

    [javac] error: package exists in another module: java.base
    [javac] package java.io;

Our classes directory have the following directory structure:

src
+- classes
 +- java
     +- io
     +- lang
     +- util
         +- function
         +- logging
             +- FileHandler.java

But the problem here is that we got sources for multiple modules in the same tree.
So for example the package 

• java.util exists  in java.base
• java.util.logging exists in java.logging
• java.util.function exists in java.base

See: https://docs.oracle.com/javase/9/docs/api/module-overview-frame.html

So I was wondering if we could restructure these directories based on their respective modules, it would be easier for us to compile. And will help in case we want convert them into modules, which I think will be better done later.

The next problem is that javac throwing error:

[javac] javac: invalid flag: -Xlint:-sunapi

I've asked this in StackOverflow as well and yet to receive an answer.

We use -XDenableSunApiLintControl -Xlint:all,-sunapi flags when compiling peers/classes/tests

[Cyrille Artho]

Hi Gayan,
You are welcome to move files to a different directory if it is not possible (or very difficult) otherwise to compile them with Java 9.
Such efforts would of course have to take place on a separate branch in the git repository, e.g., java-9. After that work is finalized, the current branch could become java-8, while java-9 would become the master branch.

I hope you'll get an answer on stackoverflow or elsewhere about how to update the compiler settings for javac version 9.

On Monday, March 5, 2018 at 3:47:51 PM UTC+1, Gayan Weerakutti wrote:

[Gayan Weerakutti]

Hi Cyrille,

You can find my commits in here:
https://github.com/gayanW/jpf-core/commits/support-java-9

Commits
Remove unused import com.sun.org.apache.bcel.internal.generic.Instruc…
Disable Xlint:deprecation
Use StackWalker instread of sun.misc.SharedSecrets
Remove SunApiLintControl

Logs:
Initial log: c81801b.log
current log: 9bf07ad.log

Unlike in previous versions, Java 9 logs deprecation warnings such as:

 warning: [deprecation] Integer(int) in Integer has been deprecated

I temporarily disable deprecation, to eliminate noise in the log, so we could focus on the actual errors.

Also In main:HashedAllocationContext I used StackWalker instread of sun.misc.SharedSecrets. Let me know if any change is required, I will make changes along the way.

You are welcome to move files to a different directory if it is not possible (or very difficult) otherwise to compile them with Java 9.
Such efforts would of course have to take place on a separate branch in the git repository, e.g., java-9. After that work is finalized, the current branch could become java-8, while java-9 would become the master branch.

Thanks I'll look into it, and see what could be done. I will let you know the result.

[Cyrille Artho]

Thank you very much for your changes.
We are still "settling in" on the new github repo for JPF as to speak, so my account is not enabled there yet, and we haven't finalized any pull-request approval process yet. I'll definitely look into this once I have time!
From your changes, it seems that we cannot have the same sources anymore for Java 8 and Java 9, due to sun.misc.secrets being out, but StackWalker being a good (official) replacement, albeit only for Java 9.

[Gayan Weerakutti]

Hi Cyrille,

What to do to those model classes that are being removed/renamed in Java 9, so for example:
sun.misc.SharedSecrets is now jdk.internal.misc.SharedSecrets
sun.reflect.ConstantPool is now jdk.internal.reflect.ConstantPool
 
But we still have model classes for them in jpf-core/src/classes.

Should we delete/replace those model classes to match the new API replacements or.. just make them compile with Java 9?

I've also been looking into patching, multi module setups and compilation. I'll push those changes for review, once finalized.

Thanks.

[Cyrille Artho]

Hi Gayan,
Great job!
I hope someone can review your PR, as my schedule in the next two weeks is very full. Maybe I can review something at an airport somewhere next week...

As for JPF model classes, you don't need to change them to make the code compile and execute against the Java 8 libraries. However, to handle Java 9 at run time, and to run on the Java 9 VM, the model classes (and native peer classes) all have to be renamed to correspond to the Java 9 naming conventions.
On the upside, it looks like some of these internal APIs finally become official after almost 20 years...

[Gayan Weerakutti]

No worries. Review them when possible :)

I didn't create a PR yet and those commits are just for your reference. Feel free to comment on any of those commits. Let me know if you want me to make any changes.

Also, is it okay that I remove src/classes/sun/misc/AtomicLong.java as it is being removed from the JDK and is not referenced by any other classes within jpf-core?

BTW, I also submitted a draft proposal.

---- On Mon, 12 Mar 2018 13:09:46 +0530 Cyrille Artho <cyrill...@gmail.com> wrote ----

--

---

You received this message because you are subscribed to a topic in the Google Groups "Java™ Pathfinder" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/java-pathfinder/nl0FGilSrcg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to java-pathfind...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gayan Weerakutti]

By saying sun/misc/AtomicLong.java as being removed, I actually meant that it is being repackaged. Sorry for the mistake :)

It is however as stated before is not referenced by any other classes within jpf-core.

[Gayan Weerakutti]

I got things to compile. You could see the latest build log at:

https://travis-ci.org/gayanW/jpf-core/builds/361230379

Commits:
https://github.com/gayanW/jpf-core/commits/java-9-9

Current String implementation uses a byte array along with an encoding flag, instead of just UTF-16 char array.

Thus results in errors: https://travis-ci.org/gayanW/jpf-core/builds/361185257#L524-L545

So I made some temporary changes to the String class assuming that compact string feature is disabled and the all chars are in UTF-16.

Commit - Temporary fix: String

Any suggestions are very welcome. I'll be looking into the test cases that fail. Thanks.

[Cyrille Artho]

Hi Gayan,
Thanks for looking into this. Strings can indeed be tricky. On the surface, they're very simple, but once you mix in different encodings and languages, and the fact that Java's string encoding is rather weird, it becomes quite complex.

I've read up on Java 9 strings (and the changes in String) a bit. It seems that compact strings can save up a lot of memory, which would be very useful for JPF.
Therefore, it would be good if your fix supported compact strings as well.

You already make a copy of StringCoding.Result.coder to remember that information, so it's just in a few places where you assume UTF16 encoding, where you may run into problems.
You can try the following approach:

1. Run the host JVM with compact strings disabled; perhaps your temporary change works in this setting.

2. Run the host JVM with compact strings enabled (the default): this will change the behavior of any tests that use ASCII strings. (Probably only few tests use UTF-8 strings that cannot be encoded in ASCII or LATIN1.)

If you're not sure, just add extra tests for compact strings (e.g., "10") or complex ones (e.g., "こんにちは").