Trouble building j2objc from source

[Daniel Dickison]

I used to be able to build j2objc from source but since some time in the past few weeks it has started to fail with many NullPointerExceptions during translation of jre_emul sources. Have the requirements or procedure for building changed?

I'm running the build with these commands:

$ make clean

$ J2OBJC_ARCHS='macosx simulator64' make -j4 dist

The errors look like this:
error: internal error translating "openjdk/src/share/classes/java/math/BigDecimal.java"
com.google.devtools.j2objc.ast.TreeVisitorError: /Users/daniel/bc/j2objc/jre_emul/openjdk/src/share/classes/java/math/BigDecimal.java:2998: NullPointerException
at com.google.devtools.j2objc.util.UnicodeUtils.hasValidCppCharacters(UnicodeUtils.java:112)
at com.google.devtools.j2objc.gen.LiteralGenerator.generateStringLiteral(LiteralGenerator.java:36)

at com.google.devtools.j2objc.gen.StatementGenerator.visit(StatementGenerator.java:753)
at com.google.devtools.j2objc.ast.StringLiteral.acceptInner(StringLiteral.java:75)
[...]

The first few lines from the make output for context:

Locale: en_US.UTF-8
Darwin daniel-mbp-2014.local 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64
Building j2objc annotations
mvn -q generate-resources dependency:sources
Xcode 8.3.2 Build version 8E2002
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
[...]

[Keith Stanger]

No changes to build procedure. And I just tried cloning the github repo and building with your command line. It worked fine for me. I am running an older xcode version (8.2.1) but that shouldn't affect the translation because that's Java.

We did recently update BigDecimal, which may have triggered something, but I have no theories on why my results would differ from yours. Do you think you could do a bit of debugging on your end to find out how a StringLiteral node is getting a null contantValue?

--
You received this message because you are subscribed to the Google Groups "j2objc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to j2objc-discus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Ball]

I also built successfully, though there were two package-info translator tests that failed (I'll investigate). I also have Xcode 8.2.1 installed, FWIW.

[kgal...@gmail.com]

I've been having issues with building from a clean clone since the last source sync (at least), so broken for me for 2-3 weeks, with the same issues mentioned above (NullPointerException). It wasn't just big decimal. Several files had this issue. I was going to give it another source sync to try to debug, but still having that problem.

If you look at the actual source, it happens parsing a ternary operator, I *think* when the second and/or third argument is a string, but didn't do a full survey. It's always been a '?:' operator, though.

I was thinking this was maybe related to JAVAC vs JDT, but the only way that would be possible on a clean build on your end would be if the environment variable J2OBJC_FRONT_END was set, AFAIK. Trying that. If that fails, then I'm guessing it has to be some combo of java version or some other setting.

Unless something is way off, though, the java file parsing fails, so it shouldn't be related to xcode.

Will update with progress, if any. Cross fingers...

OK, update. Source translate got much further, but objc compile failed on sun/misc/Cleaner.m. Hmm. Digging.

BTW, I generally run 'make -j8 frameworks'. I've been playing in a fork for a while, but clean j2objc master builds were working with that command as of several weeks ago, as far as I remember.

[Tom Ball]

Your comment about a possible javac issue got me thinking: the build copies $JAVA_HOME/lib/tools.jar, after setting JAVA_HOME if it's not defined as an environment variable. Would each of you please run the following: "echo "JAVA_HOME=${JAVA_HOME}" && echo -n java_home= && /usr/libexec/java_home"? Keith and I have:

JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home

Note: if your JAVA_HOME is defined and an older version than java_home, then your environment may be out of date. If you don't need it to be separate, you can run "export JAVA_HOME=`/usr/libexec/java_home" in your ~/.bash_profile instead of explicitly setting the version.

[kgal...@gmail.com]

Already running down that path. I updated my java 8. It was 1.8.0_91 (ish). After updating to the latest (8u131), it looks like I'm past the null pointer issue and things are compiling. Will let you know if this completes.

[kgal...@gmail.com]

OK. Way past original failure. While we're waiting for build, here's the output of the home command now:

Kevins-MBP:~ kgalligan$ echo "JAVA_HOME=${JAVA_HOME}" && echo -n java_home= && /usr/libexec/java_home
JAVA_HOME=
java_home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home

Unless the jdk reinstall did something to the env vars, though, I think it was simply updating jdk version.

[Daniel Dickison]

My JAVA_HOME is unset and java_home shows jdk1.8.0_77.jdk, which I guess is a year old. I thought Java auto-updates (I certainly get popups prompting to install updates regularly), and the Java Control Panel claims to be on Java 8 Update 131. I'll try installing JDK directly from the web and see if that helps.

$ echo "JAVA_HOME=${JAVA_HOME}" && echo -n java_home= && /usr/libexec/java_home
JAVA_HOME=
java_home=/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home

[Kevin Galligan]

OK. My clean build went through 100% with updated jdk.

--
Kevin Galligan
https://twitter.com/kpgalligan

[Daniel Dickison]

Me as well, a clean build succeeded after updating my JDK to 1.8.0_131. Thanks for the help. I had forgotten that the JDK is installed and updated independently of the JVM.

[Tom Ball]

That's good news. My guess is this was both introduced (in BigDecimal.java) and fixed (in javac) in OpenJDK. I check what version I pulled the BigDecimal source, and make that the minimum Java version.

[Daniel Dickison]

I don't think I was clear in my original message — I was getting the same NullPointerException across many of the files being translated in jre_emul, not just BigDecimal.java. That one just happened to be the first one listed. So perhaps it was some change or bug fix in the parser or something like that?

[Tom Ball]

Right, we use the whole javac front-end to create an annotated AST, then convert it into j2objc's. The issue looks to be that javac sometimes defined a string literal node with a null value. I'll see if the converter can be enhanced to use the value from the associated element. Those two values should always be the same, but regardless, the element is the source of truth.

[Daniel Dickison]

I'm now running into another issue, and I'm afraid I'm pretty much helpless with this stuff. I appreciate all the help!

While "make dist" completed successful the first time, if I change anything and try to do another "make dist", or a "make test" from the jre_emul directory, the translation now fails with several errors. If I do a "make clean", then I can get it to build again.

This is the output:

building jre_emul.jar

building jre_emul-src.jar

translating jre_emul sources

android/libcore/luni/src/main/java/java/net/URL.java:515: error: method toExternalForm in class java.net.URLStreamHandler cannot be applied to given types;

        return new URI(streamHandler.toExternalForm(this, true));

                                    ^

  required: java.net.URL

  found: java.net.URL,boolean

  reason: actual and formal argument lists differ in length

android/libcore/luni/src/main/java/java/net/URL.java:571: error: cannot find symbol

        return URI.getEffectivePort(protocol, port);

                  ^

  symbol:   method getEffectivePort(java.lang.String,int)

  location: class java.net.URI

android/libcore/luni/src/main/java/java/io/ObjectInputStream.java:945: error: incompatible types: java.lang.String cannot be converted to java.lang.Class<?>

            ObjectStreamField f = new ObjectStreamField(classSig, fieldName);

                                                                  ^

android/libcore/luni/src/main/java/java/io/ObjectInputStream.java:1101: error: cannot find symbol

                Class<?> type = fieldDesc.getTypeInternal();

                                         ^

  symbol:   method getTypeInternal()

  location: variable fieldDesc of type java.io.ObjectStreamField

android/libcore/luni/src/main/java/java/io/ObjectInputStream.java:1151: error: cannot find symbol

                        Class<?> fieldType = localFieldDesc.getTypeInternal();

                                                           ^

  symbol:   method getTypeInternal()

  location: variable localFieldDesc of type java.io.ObjectStreamField

android/libcore/luni/src/main/java/java/io/ObjectInputStream.java:1683: error: cannot find symbol

            element.resolve(loader);

                   ^

  symbol:   method resolve(java.lang.ClassLoader)

  location: variable element of type java.io.ObjectStreamField

android/libcore/luni/src/main/java/java/io/ObjectInputStream.java:1940: error: cannot find symbol

        Object result = input.decodeUTF((int) length);

                             ^

  symbol:   method decodeUTF(int)

  location: variable input of type java.io.DataInputStream

android/libcore/luni/src/main/java/java/io/ObjectStreamClass.java:368: error: offset has private access in java.io.ObjectStreamField

                _fields[i].offset = primOffset;

                          ^

android/libcore/luni/src/main/java/java/io/ObjectStreamClass.java:371: error: offset has private access in java.io.ObjectStreamField

                _fields[i].offset = objectOffset++;

                          ^

android/libcore/luni/src/main/java/java/io/ObjectStreamClass.java:874: error: cannot find symbol

                    loadField.setUnshared(field.isUnshared());

                             ^

  symbol:   method setUnshared(boolean)

  location: variable loadField of type java.io.ObjectStreamField

android/libcore/luni/src/main/java/java/util/regex/Matcher.java:241: error: cannot find symbol

        address = openImpl(pattern.address);

                                  ^

  symbol:   variable address

  location: variable pattern of type java.util.regex.Pattern

Classes/java/lang/System.java:435: error: cannot find symbol

      return Console.getConsole();

                    ^

  symbol:   method getConsole()

  location: class java.io.Console

make[1]: *** [/Users/daniel/bc/j2objc/jre_emul/build_result/Classes/.translate_mark_jre_emul] Error 12

make: *** [jre_emul_dist] Error 2

Any thoughts or places I should start looking?

[Tom Ball]

All of these errors are in classes that aren't in the jre_emul/android/platform/libcore/ojluni root, which may be significant because the "oj" stands for OpenJDK. The libcore team is migrating from a Apache Harmony base to OpenJDK, since OpenJDK is the reference implementation (plus Apache Harmony is dead). Since all the failing files are originally from Apache Harmony, the issue may be that jre_emul.jar is either missing or malformed. 

How do you invoke j2objc? The j2objc script (scripts/j2objc.sh) sets jre_emul.jar as the bootclasspath, which it forwards to javac. If the bootclasspath isn't set or jre_emul.jar is incomplete or corrupted, then javac should fall back to the JRE's rt.jar, which will have some API differences from the Apache Harmony classes.

--

[Daniel Dickison]

These errors are from j2objc's own Makefile calling the translator, i.e. I get those errors when I call "make dist" from the j2objc directory, so I'm not sure exactly what the invocation of j2objc looks like there, though I've tracked down USE_SYSTEM_BOOT_PATH=TRUE seems to be set while translating jre_emul.

To clarify, if I do a "make clean" first, followed by "make dist", the build succeeds. If I then make a trivial change in one of the jre_emul source files (e.g. adding a System.out.println("foo") in IosHttpURLConnection.java) and then call "make dist", I now get the errors from above. (I apologize for the formatting, by the way — the Google Groups editor seems to add a black background and extra line breaks when pasting from Terminal.)

It looks like the second translation of jre_emul is happening using the JRE interface from my rt.jar file at:

/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre/lib/rt.jar

If I extract, say, java/net/URLStreamHandler$class from rt.jar and inspect it with javap, sure enough there is only one overload of toExternalForm taking one parameter instead of two.

So I guess the problem is that incremental "make dist" uses the classes from rt.jar instead of the sources or the existing jre_emul.jar. How is everyone else compiling jre_emul? Surely you're not doing a full clean/rebuild each time? Maybe I should have a distribution of OpenJDK on my machine instead of having java_home pointing to the one from Oracle, but I'm not seeing an obvious way to install OpenJDK on a Mac.

[Keith Stanger]

Daniel, do you happen to have this fix synced? https://github.com/google/j2objc/commit/953286c2faa17ad853738c33548e9494f36e51c7

I'm unable to reproduce these errors using the steps you describe.

The JRE translation does use the system boot path but also puts jre_emul.jar in the classpath. (We may be able to fix this and just put jre_emul.jar as the boot path) When you get these errors, try opening up jre_emul/build_result/jre_emul.jar and see if it's missing the classes for some of the symbols that are breaking.

[Daniel Dickison]

I do have that "set -e" fix. My working copy a clone of the current GitHub HEAD.

I looked at jre_emul/build_result/jre_emul.jar and it does include the classes causing these issues, e.g. java/net/URLStreamHandler$class and URI$class, and those classes do contain the methods that the build fails to find.

I arranged to have the j2objc command run with "set -x" and it looks like jre_emul/build_result/jre_emul.jar is in fact specified in the -classpath. And android/libcore/luni/src/main/java is in the -sourcepath and that version of e.g. URI.java contains the purportedly missing method. I'm stumped. This is the java execution command from "make dist" with "set -x":

+ java -jar ../dist/lib/j2objc.jar '' -classpath ../dist/lib/j2objc_annotations.jar:/Users/daniel/bc/j2objc/jre_emul/build_result/jre_emul.jar -sourcepath apache_harmony/classlib/modules/luni/src/main/java:apache_harmony/classlib/modules/concurrent/src/main/java:apache_harmony/classlib/modules/luni-kernel/src/main/java:android/platform/libcore/dalvik/src/main/java:android/libcore/luni/src/main/java:android/libcore/xml/src/main/java:Classes:apache_harmony/classlib/modules/archive/src/main/java:android/frameworks/base/core/java:android/libcore/json/src/main/java:../annotations/src/main/java:apache_harmony/classlib/modules/beans/src/main/java:android/platform/libcore/luni/src/main/java:android/platform/libcore/luni/src/objc/java:android/platform/libcore/ojluni/src/main/java:android/platform/external/okhttp/okio/okio/src/main/java:android/platform/libcore/ojluni/src/lambda/java:openjdk/src/share/classes:stub_classes -d /Users/daniel/bc/j2objc/jre_emul/build_result/Classes -encoding UTF-8 --doc-comments -Xtranslate-bootclasspath @/Users/daniel/bc/j2objc/jre_emul/build_result/Classes/.translate_list_jre_emul

And:

$ cat /Users/daniel/bc/j2objc/jre_emul/build_result/Classes/.translate_list_jre_emul 

Classes/com/google/j2objc/net/IosHttpURLConnection.java

[Keith Stanger]

My best guess here is that your java is searching the boot path before the classpath. This would only cause trouble on incremental builds because from a clean build, all of the JRE classes are source inputs to the compiler and the inputs will take precedence over the boot or classpath. Since your java version is slightly ahead of mine (1.8.0_131 vs 1.8.0_112) that might explain why I can't reproduce.

Tomorrow I can try using jre_emul.jar as the boot path and see if that works. Otherwise I can try updating my java version to see if I can reproduce the errors.

[Daniel Dickison]

I installed 1.8.0_112 and set JAVA_HOME to its path, and I'm still seeing the same errors after the same sequence (clean; dist; edit IosHttpURLConnection; dist).

I'm starting to worry that this is due to some silly customization of my environment, though I don't recall doing anything particularly unusual especially in terms of java. It would be great to hear if you are able to reproduce the problem there.

[Keith Stanger]

I actually have a vague recollection of seeing similar behavior several weeks ago. I believe the problem occurred only in one of my older clients and was fixed by wiping out and re-creating the client. It was very strange because even a diff against another client without the issue revealed nothing. I never did figure out exactly what was happening, but I think I got around the errors by removing jre_emul.jar from the -classpath for translation (https://github.com/google/j2objc/blob/master/jre_emul/Makefile#L44).

[Daniel Dickison]

It looks like it's well-defined behavior that the bootclasspath (rt.jar) is searched before -classpath (jre_emul.jar) according to:
https://docs.oracle.com/javase/8/docs/technotes/tools/findingclasses.html

I was able to get incremental builds working by commenting out TRANSLATE_USE_SYSTEM_BOOT_PATH = TRUE in jre_emul/Makefile to use the existing jre_emul.jar with -Xbootclasspath.

Perhaps detecting the existence of dist/lib/jre_emul.jar and setting TRANSLATE_USE_SYSTEM_BOOT_PATH accordingly be a general solution to this problem?

[Keith Stanger]

Maybe the problem has more to so with whether the compiler resolved the dependency from the bootpath vs the sourcepath. Anyways, I have a change coming that will always use jre_emul.jar as the boot path.

[Tom Ball]

TRANSLATE_USE_SYSTEM_BOOT_PATH is unrelated to this discussion, so my guess is that your specifying the -Xbootclasspath flag is most likely what fixed this. The TRANSLATE_USE_SYSTEM_BOOT_PATH flag tells the translator to translate classes that are on the bootclasspath, by setting j2objc's -Xtranslate-bootclasspath. This is necessary so that libraries like jsr250, which has annotation classes that were later added to the JRE, don't translate those existing classes and thereby avoid duplicate symbol linker errors (we can't stop developers from using Apple's -ObjC linker hack). j2objc can't just skip translating JRE classes, or it couldn't be used to build jre_emul.

[Keith Stanger]

Daniel, try syncing to head now. I updated how we translate jre_emul sources so that we use jre_emul.jar as the boot path.

https://github.com/google/j2objc/commit/890bed383c8108f5de618ae914203b5e28bcce9c

https://github.com/google/j2objc/commit/55bedf2f439fed4d8af56d3162e7158d5bace51c

[Daniel Dickison]

Keith, with your changes I am now able to do both clean builds and incremental builds successfully. Thanks a ton!

> You received this message because you are subscribed to a topic in the Google Groups "j2objc-discuss" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/j2objc-discuss/iAmaiwXUSXI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to j2objc-discus...@googlegroups.com.