Issue 61 in vogar: Running Android Benchmarks

[vo...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

I couldn't get Vogar to run benchmarks on Android, executing the benchmarks
would fail with "Fatal Error: Unable to find package java.lang in classpath
or bootclasspath" http://pastebin.com/zZcTL3WN

The project is under the `prateek/vogar` branch;
https://github.com/segmentio/analytics-android/tree/prateek/vogar, running
`java -jar vogar.jar --benchmark --mode device --device-dir /sdcard/vogar
--sourcepath core/src/main/java --verbose
core/src/androidTest/java/com/segment/analytics/SampleBenchmark.java`

I tried setting up a similar project on Okio using the UTF8Benchmark I saw
Jesse posting (https://gist.github.com/swankjesse/7b7f12ea5dc06196bedd).
Tried it setting that up here -
https://github.com/f2prateek/okio/commit/c709e3b5a54daecd9dd6eed29a20dd2114258738.
It fails with a similar message "Fatal Error: Unable to find package
java.lang in classpath or bootclasspath"

Am I missing some obvious steps?

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[vo...@googlecode.com]

Comment #1 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

This was with JDK 7, 8 running on Android 5.0

[vo...@googlecode.com]

Updates:
Status: Fixed
Owner: nful...@google.com

Comment #2 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Using a recent version of vogar (the one described here:
https://groups.google.com/forum/#!topic/vogarcode/DamJXHpC-nw) works for me
with a sample benchmark from my AOSP dev tree:

e.g.
vogar --benchmark --mode device --verbose
libcore/benchmarks/src/benchmarks/ArrayCopyBenchmark.java

From the classpath, it looks like the version of vogar you are using dates
from KitKat or before. The core library changed its name when ART was
introduced. Pick up the latest version from the AOSP tree.

If you need to, you should still be able to target older devices that came
with dalvik with --mode device_dalvik

One thing to watch out for with ART is that it needs root access for the
AOT compile:

e.g. call this before running vogar:
$ adb root
$ adb remount

If you don't, it will still work but it will be using the (much slower)
interpreter.

Marking this as fixed, but please feel free to come back if you continue to
have problems.

[vo...@googlecode.com]

Comment #3 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

A small update after talking to somebody who knows about these things:

The need for root I suggested *may* not be needed if you are specifying
--device-dir and pointing at an alternative location.

You should expect the first time you run on a device for it to take a while
as it performs the AOT compile for the boot image. The runtime performance
will be better and more representative of behavior in a real app.

On a related side note, if you are using AOSP top-of-tree:

I'm currently seeing issues using AOSP top-of-tree with the --device-dir
default setting (e.g. when you don't specify it, you get /sdcard/vogar).
The bug mean it's falling back to the interpreter anyway (despite running
as root). It may be related to any use of the /sdcard file system.

I'm going to take that up with the ART guys. In the meantime, if you see
issues you might want to try --device-dir /data/local/tmp
On my tests just now it made the benchmark I was looking at 2x faster,
which would be consistent with interpreter Vs AOT.

It doesn't look like you would need root to write there, but look for files
of non-zero length in /data/local/tmp/dalvik-cache/arm/ to be sure that AOT
is working. Once/if files have been written there as root you should
continue to do so or clear them out.

[vo...@googlecode.com]

Comment #4 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Thanks for the detailed notes! I was still unable to get it working
however, failing with the same error.

Here's what I did:

1. git clone https://android.googlesource.com/platform/external/vogar (SHA
5a8fd4b1d6c046471c7b951680cace3eec723fe7)

2. Build the JAR (I used the /bin/vogar script, only modifying the
location of my Android SDK Path as on my system)
- android_tools_dir=`dirname ${adb_path}`
+ android_tools_dir=~/android-sdk/platform-tools

3. Clone libcore (git clone
https://android.googlesource.com/platform/libcore) (SHA
59c2a0aa48b2e7f6770e29993332fecadb145f21)

4. run `java -jar vogar.jar --benchmark --mode device --verbose
libcore/benchmarks/src/benchmarks/ArrayCopyBenchmark.java`

I also tried the variant that was in the vogar-android script, by running
`java -classpath vogar.jar vogar.Vogar --benchmark --mode device --verbose
libcore/benchmarks/src/benchmarks/ArrayCopyBenchmark.java`

And the error report still looks very similar to the previous one
http://pastebin.com/ut9y0yH0

My device is also not rooted, so I couldn't try running `adb root` before
the benchmarks.

[vo...@googlecode.com]

Comment #5 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Som more notes:

1. `--device-dir /data/local/tmp` failed with the same error

2. Running on a Genymotion Emulator for API 19 (with root) failed with the
same error.
`java -classpath vogar.jar vogar.Vogar --benchmark --mode device_dalvik
--verbose libcore/benchmarks/src/benchmarks/ArrayCopyBenchmark.java`

[vo...@googlecode.com]

Comment #6 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Ok. We've made progress: you have the latest vogar.

I suspect from the change you had to make in 2 that an assumption I made
about your dev environment was incorrect.

vogar has determined that you are not using an Android SDK. I had assumed
the same.

Vogar has two modes based on where the "adb" in your path lives: if it
thinks you are using an SDK it works differently. I can see from the
pastebin that it either picked up "adb" in a directory other than
platform-tools, or it couldn't find it at all.

To get adb, make sure you run the sdk-manager and download platform-tools.

If adb is not in your path (as suggested by your change in 2) it falls back
to assuming you have a full AOSP dev environment.

So, given your path in 2 above, execute this:

export PATH=$PATH:~/android-sdk/platform-tools

and vogar should detect the adb is in the platform-tools directory and
should switch to compiling against the android framework.jar (which
contains java.lang, etc.). vogar requires adb in your path later in the
process as well, so this is needed anyway.

If you do that you can/should also undo your change in 2: the script makes
the same assumption.

--------------------

In the case you _are_ using vogar with a full AOSP tree....

The next step is to make sure the libraries referenced exist. The immediate
complaint is that it cannot find java.lang.

That is supposed to be here:
out/target/common/obj/JAVA_LIBRARIES/core-libart_intermediates/classes.jar

This jar is built as a side effect of various rules on Android. I typically
build / flash my devices so I typically have these libs already built (and
with root access to the device).

To get a full android build (i.e. one that you can push to a device), you
would do:

$ make all

(usually with a -j flag like -j16 to make it threaded, e.g. make -j16 all).

*However*, that takes a long time and some setup
(https://source.android.com/source/initializing.html).

If you just need the libraries vogar wants for compilation you just need:

$ make -j<whatever> core-libart

The other libraries listed in your pastebin are required depending on the
nature of the tests you want to run. core-libart contains the java.*
libraries. Some of the networking code needs conscrypt and bouncycastle for
secure sockets / crypto. The process is the same (take the name before
_intermediates and you have the name of the build target).

[vo...@googlecode.com]

Comment #7 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Another update: please pull the latest code from vogar and it should now
create the AOT / dalvik-cache resource in a location that works for
non-rooted devices. Now, there should be no need to use --device-dir for K
and L devices.

[vo...@googlecode.com]

Comment #8 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

That did it! I had installed the Android SDK and build-tools with homebrew
which put `adb` (and other binaries) in /usr/local/bin, which caused issues
with the assumption that ../../ would give the Android SDK installation
directory.

Most tools (Gradle, Maven, etc.) use the $ANDROID_HOME variable to
determine the location of the Android SDK. Would it be possible to get
vogar to do the same?

Here's my attempt to try to do that -
https://gist.github.com/f2prateek/f9397d52c13904ecf530. I got the vogar.sh
script to work, but not the actual vogar.jar. Hope that helps if you want
to add it to vogar.

[vo...@googlecode.com]

Comment #9 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Created https://code.google.com/p/android/issues/detail?id=82018 to track /
remind me.

I can't be sure when / if I'll get to it: I never have ANDROID_HOME set,
and I could only find one reference to it on the actual
developer.android.com site.

I agree, though: if it is a common pattern it seems reasonable to use it as
a signal if it is present. A quick search for it does suggest it is
becoming a de facto standard.

[vo...@googlecode.com]

Comment #10 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Thanks for opening, I should be able to submit a complete patch sometime
next week after my exams.

[vo...@googlecode.com]

Comment #11 on issue 61 by nful...@google.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Good luck with the exams!

If you are planning to contribute to Android, take a look here:
https://source.android.com/source/contributing.html

If you haven't already, you will need to agree to the contributor license
agreement for Android: https://source.android.com/source/licenses.html

FYI if unaware: to obtain access to the ANDROID_HOME, you should be able to
use System.getenv("ANDROID_HOME").

[vo...@googlecode.com]

Comment #12 on issue 61 by f2prat...@gmail.com: Running Android Benchmarks
https://code.google.com/p/vogar/issues/detail?id=61

Thanks, I've submitted a patch here
https://android-review.googlesource.com/#/c/120030/