Let's start testing Scala on Avian¹

[Simon Ochsenreither]

Hi everyone!

I was able to resolve the remaining issues. Scala successfully bootstraps on Avian. All tests pass.²

Can we create a nightly Jenkins task which builds Scala on Avian and runs the test suite on it?³

Thanks and bye,

Simon



¹ Elevator pitch for those who don't know Avian:

Avian is not just yet-another JVM, but comes with some distinct advantages compared to HotSpot:

• Ability to create native, self-contained, embeddable binaries
• Runs on iPhone, Android and other ARM targets
• AOT and JIT compilation are both fully supported
• Support for tail calls and continuations
• An intelligible code base
• Responsive maintainers
• Open for improvements (value classes, specialization, etc.)

² Excluding the instrumented ones; instrumentation is not supported on Avian.

³ Imho, it is currently not fast enough to be included in the PR validation.

[Grzegorz Kossakowski]

On 30 August 2013 10:02, Simon Ochsenreither <simon.och...@gmail.com> wrote:

Hi everyone!

I was able to resolve the remaining issues. Scala successfully bootstraps on Avian. All tests pass.²

² Excluding the instrumented ones; instrumentation is not supported on Avian.

Out of curiosity: is there particular reason why instrumentation is not supported?

 

³ Imho, it is currently not fast enough to be included in the PR validation.

What's the performance compared to Oracle JVM? How long does it take to run all tests?

--
Grzegorz Kossakowski

Scalac hacker at Typesafe

twitter: @gkossakowski

github: @gkossakowski

[Simon Ochsenreither]

Out of curiosity: is there particular reason why instrumentation is not supported?

As far as I remember, the necessary Java agent stuff is not implemented in Avian.

What's the performance compared to Oracle JVM? How long does it take to run all tests?

Running test/partest --run took 02:20:32, which is by far the most time-consuming part.
Keep in mind that all of this is run inside VirtualBox with limited amount of CPU and RAM.

Bootstrapping Scala itself is a lot faster: For the small modules it's roughly on par with Hotspot while the large ones (compiler, library) are ~4 times slower as far as I remember.

[Vlad Ureche]

2013/8/30 Simon Ochsenreither <simon.och...@gmail.com>

I was able to resolve the remaining issues. Scala successfully bootstraps on Avian. All tests pass.²

Nice work Simon! Avian just got another watcher, I'm really looking forward to seeing value classes and specialization in the VM! :)

Vlad

[Simon Ochsenreither]

Any chance to move this forward?

[Grzegorz Kossakowski]

On 8 September 2013 10:09, Simon Ochsenreither <simon.och...@gmail.com> wrote:

Any chance to move this forward?

Yes, I think so. However, the biggest question is: who is going to do the work and how much work is needed?

How hard is to maintain (install, update, etc.) Avian VM?

I'd suggest that you look at https://github.com/scala/jenkins-scripts/blob/master/job/scala-nightly-main (which is used to run https://scala-webapps.epfl.ch/jenkins/job/scala-nightly-main-master/) and determine if those scripts we use for jenkins could be easily adapted so we can have nightly build on top of Avian.

I think at this point all we could offer is to run nightly on top of Avian. You'd need to build community around supporting Scala on Avian which would be responsible for keeping that job to be green. Once we gain more experience with the whole experiment we might consider tighter integration.

[Simon Ochsenreither]

Hi Grzegorz,

thanks for looking into this!

Yes, I think so. However, the biggest question is: who is going to do the work and how much work is needed?

How hard is to maintain (install, update, etc.) Avian VM?

I'd suggest that you look at https://github.com/scala/jenkins-scripts/blob/master/job/scala-nightly-main (which is used to run https://scala-webapps.epfl.ch/jenkins/job/scala-nightly-main-master/) and determine if those scripts we use for jenkins could be easily adapted so we can have nightly build on top of Avian.

I think at this point all we could offer is to run nightly on top of Avian. You'd need to build community around supporting Scala on Avian which would be responsible for keeping that job to be green. Once we gain more experience with the whole experiment we might consider tighter integration.

Sounds great!

Even if we pick the full program of keeping all involved parts (OpenJDK, Scala, Avian) up-to-date, it is actually quite simple:

• Get and build OpenJDK (or use an existing Java 7 one)
  
• Get and build Avian using the OpenJDK classpath
  
• Get, build and test Scala on Avian

The build script I have been using looks like this (sorry, no bash expert):

### HOST ###

cd
cd Entwicklung/avian-build/jdk7u-dev
# Cloned from http://hg.openjdk.java.net/jdk7u/jdk7u-dev
hg pull default
bash get_source.sh
LANG=C make ALLOW_DOWNLOADS=true ALT_BOOTDIR=/usr/lib/jvm/java-7-openjdk-amd64 clean
unset JAVA_HOME
LANG=C make ALLOW_DOWNLOADS=true ALT_BOOTDIR=/usr/lib/jvm/java-7-openjdk-amd64

cd
cd Entwicklung/avian-build/
rm -rf jdk-avian
cp -r jdk7u-dev/build/linux-amd64/j2sdk-server-image/ jdk-avian

cd
cd Entwicklung/avian-build/avian
# Cloned from https://github.com/ReadyTalk/avian.git
git pull origin
make clean
make tails=true strip=: openjdk=/home/soc/Entwicklung/avian-build/jdk-avian
cp build/linux-x86_64-tails-openjdk/libjvm.so ../jdk-avian/jre/lib/amd64/server

cd
cd Entwicklung/avian-build/scala
# Cloned from https://github.com/scala/scala.git
git pull origin master
ant all.clean

### GUEST ###

cd
cd /home/soc/Entwicklung
sudo mount -t vboxsf -o uid=1000,gid=1000 avian avian-build
export JAVA_HOME="/home/soc/Entwicklung/avian-build/jdk-avian"
export SCALA_BIN="/home/soc/Entwicklung/avian-build/scala/build/pack/bin/"
export PATH=${PATH}:${JAVA_HOME}/bin:${SCALA_BIN}

cd
cd /home/soc/Entwicklung/avian-build/scala
/home/soc/Entwicklung/avian-build/avian/build/linux-x86_64-tails-openjdk/avian-dynamic -Xmx2048m -Xss1m -classpath /home/soc/Entwicklung/avian-build/apache-ant-1.9.0/lib/ant-launcher.jar -Dant.home=/home/soc/Entwicklung/avian-build/apache-ant-1.9.0 -Dant.library.dir=/home/soc/Entwicklung/avian-build/apache-ant-1.9.0/lib org.apache.tools.ant.launch.Launcher test

That's pretty much it.

[Lee_T]

Two footnotes to Simon's recipe above which may help someone
adding avian testing to the scala build, when the proper time comes.
The both are obvious after the fact. I hope to make prospect of avian integration
look more inviting  by removing some stones/gotchas.

1)  One can tell that one is actually using avian in the guest
     two ways:

     a) java -version  shows something like (note Avian on last line):
         java version "1.7.0_25"
        OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.13.04.2)
        Avian (build 0.7, )

      b)  Use the scala REPL (Avian after rightmost left parenthesis) :
           $ scala
           Welcome to Scala version 2.11.0-20130908-003843-e3968a5a40 (Avian, Java 1.7.0_25).

      Otherwise, Avian and OpenJDK JVM look identical, aside from some possible timing issues,
      to me so far.  I found it comforting/essential to be able to tell which JVM I was using.


2)  A testing scenario is likely to follow the "or use an existing Java 7" path of Simon's recipe.

     >      Get and build OpenJDK (or use an existing Java 7 one)

     On Ubuntu (Intel), the OpenJDK requires copying two directory trees:
         /usr/lib/jvm/java-7-openjdk-common
         /usr/lib/jvm/java-7-openjdk-i386

        Some files under the  *-i386 directory are actually links to files under *-common.
      
       On the build machine , the *-i386 directory tree can be copied to the local reference
       jdk-avian in Simon's recipe. I believe the *-common tree is not needed here, but
       i copied it anyway at the same level as jdk-avian, for use by the guest (see below).

       Because of my build setup, I used rsh to clean and rsync to transfer
       files to a running guest.  I ended up with /soc/Entwicklung/avian-build/
       containing: avian-build  java-7-openjdk-common  jdk-avian

       My first implementation pass missed the *-common files. I could get to the
       scala REPL prompt, but even a simple statement (scala> 1<cr>) got complaints
       of a missing MD5.  Tracking down and supplying the *-common files allowed
       scala to operate at least in a simple smoke test. 

       On the guest, I believe only the JRE (but the whole JRE, beware
       links to *-common sub-tree) is necessary. I set my JAVA_HOME
       to (full spec) jdk-avian/jre.  A minimal set of guest JRE files could
       be discovered and files not in that set simply not copied to the guest
       (to save time & complexity).

Simon's recipe describes what to do. I found it invaluable for what it does _not_ say:
it tells me that I need not mess with (no need to mess with editing bash files or
creating links in ~/bin). 

Now that I have a working scala_Avian working in the guest, my next step
is to copy over my working copy of Miguel's GenRefactored99sC and see
if I can break either it or Avian.   Sbt 0.13 could not start the avian JVM,
possibly because Sbt uses scala 2.10.mumble. Sbt looks like the harder
problem, so I'll look at it later.


Lee

 

[Simon Ochsenreither]

Hi Lee!

Thanks for the additional hints! It is incredibly useful to have a pair of fresh eyes go over the instructions and weed out all various undocumented assumptions I have made.

Regarding 1): It is still possible to end up in a situation when boot-strapping Avian where for instance partest itself runs on Avian but the individual test-cases somehow managed to get executed on Hotspot ... don't ask why or how ... but it made me set-up a VM with no Java installed for that reason alone.

I haven't looked into SBT yet, but I think using a version with 2.11 underneath is probably mandatory. I haven't tested Avian with Scala 2.10 recently, so I'm not sure how many issues will arise there (or if they are largely restricted to some exotic tests failing).

Thanks!

Bye,

Simon

[Simon Ochsenreither]

Hi Grzegorz!

I'd suggest that you look at https://github.com/scala/jenkins-scripts/blob/master/job/scala-nightly-main (which is used to run https://scala-webapps.epfl.ch/jenkins/job/scala-nightly-main-master/) and determine if those scripts we use for jenkins could be easily adapted so we can have nightly build on top of Avian.

I have looked into the build scripts Grzegorz linked, but I'm not sure what's the best way to go. (I guess setting up a local Jenkins instance for testing won't really work.)

From an optimistic POV, wouldn't it be enough to build Avian, copy the JDK and export the right paths; then just run the standard nightly build+test job?

Would appreciate any hints here!

Thanks and bye!

Simon

[Grzegorz Kossakowski]

Hi guys,

Thanks for pinging me. I must apologize but I won't be able to help for the next 2 weeks. I'm flying to Strange Loop in a few minutes and next week I'm busy packing and moving back to Europe (from SF) so I won't have much time for coding :-/

[Adriaan Moors]

I think the right way to do this is by creating a VM that runs the Jenkins slave, and then just reuse the existing scripts as-is -- similar to how we test on windows.

Something like vagrant or docker could be used to automate creating the VM.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.