“I would really like to have a meaningful second back-end, that would be great” ~paulp

[Simon Ochsenreither]

After this unobtrusive appeal to authority, let me suggest that we almost reached that point now.
Together with the maintainer of Avian, it was possible to solve the obstacles which prevented the execution of Scala on Avian over the last weeks.

Scala is successfully bootstrapping itself on Avian and passes the complete test suite (except a few “special” tests, which e. g. also fail on newer HotSpot builds or where the test itself is wrong).

From my point of view, these are the remaining steps, which I'm unable to achieve due to various reasons:

• Integration into Jenkins
  Have a build machine which continuously tests Scala on Avian
  
• Integration into the test-suite
  Add a test/files/avian subdir, comparable to test/files/jvm to test Avian-specific functionality (proper tail calls for now)
  
• Integration into the documentation
  Mention that the deployment option exists. (I guess the Android people would be happy about that, too. Just search for “Android” on scala-lang.org.)
  
• Integration into tooling
  Would be nice to have e. g. a SBT plugin which automates the steps to create a native, self-contained executable

Imho, the first point is the most important one. It needs to be done by someone with the necessary machine and Jenkins administration privileges and skills. I can help with setting up/building Avian of course.
The second point is a nice-to-have for now, but it would be good to have the necessary infrastructure set up.
Regarding the third point, I'm currently waiting for the new website and will create PRs as soon as it is public.
Concerning the fourth point: I guess this is more or less a community task/project. I'm currently lacking the necessary SBT plugin skills to contribute.

Thanks and bye,

Simon

[Simon Ochsenreither]

For those, who don't know Avian, this might be a good elevator pitch:

Avian is not just yet-another JVM, but comes with some distinct advantages compared to HotSpot:

• Ability to create native, self-contained, embeddable binaries
• Runs on iPhone, Android and other ARM targets
• AOT and JIT compilation are both fully supported
• Support for tail calls and continuations
• An intelligible code base
• Responsive maintainers
• Open for improvements (value classes, specialization, etc.)
  

[Alex Wilson]

Apologies if this question is a duplicate: but what's the performance (runtime) of avian like compared to hotspot? The latter has been under dev. for so long that I'd almost be disappointed to hear that Avian could keep up after a relatively short lifetime.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Simon Ochsenreither]

Apologies if this question is a duplicate: but what's the performance (runtime) of avian like compared to hotspot? The latter has been under dev. for so long that I'd almost be disappointed to hear that Avian could keep up after a relatively short lifetime.

These are measurement from my machine:

2013-03-06T12:56:32
Scala 2.11.0-20130305-124913-32f04db672
Hotspot 64-Bit Server VM 1.7.0_15
----------------------------------------
[locker.lib.timer:        1:01.513 sec]
[locker.reflect.timer:      39.489 sec]
[locker.comp.timer:       1:19.280 sec]
[quick.lib.timer:         1:14.321 sec]
[quick.reflect.timer:       39.983 sec]
[quick.comp.timer:        1:18.367 sec]
[quick.plugins.timer:       16.413 sec]
[quick.scalacheck.timer:    18.074 sec]
[quick.scalap.timer:        18.556 sec]
[quick.partest.timer:       18.679 sec]

March 2013-03-08T11:20:00
Scala 2.11.0-20130306-063604-5967a664ab
Avian
----------------------------------------
[locker.lib.timer:        3:42.545 sec]
[locker.reflect.timer:    1:42.479 sec]
[locker.comp.timer:       5:26.200 sec]
[quick.lib.timer:         4:02.013 sec]
[quick.reflect.timer:     1:47.344 sec]
[quick.comp.timer:        5:30.512 sec]
[quick.plugins.timer:       20.017 sec]
[quick.scalacheck.timer:    28.173 sec]
[quick.scalap.timer:        26.717 sec]
[quick.partest.timer:       28.362 sec]


At least in this benchmark, it is roughly as fast as HotSpot on small working sets (probably because HotSpot's JIT is still compiling) and up to 4 times slower on larger compilations.
Responsiveness is pretty good, try a few of these 800kB-GUI-examples which include the runtime, the necessary standard library and GUI classes, and the application itself: http://oss.readytalk.com/avian/examples.html

Performance has been improving rapidly, e. g. these are the measurements taken a month before those mentioned above:

2013-02-01T20:36:39
Scala 2.11.0-20130129-171234-d499db3800
Avian
----------------------------------------
[locker.lib.timer:        4:01.447 sec]
[locker.reflect.timer:    1:44.725 sec]
[locker.comp.timer:       7:51.470 sec]
[quick.lib.timer:         4:23.371 sec]
[quick.reflect.timer:     1:52.912 sec]
[quick.comp.timer:       13:34.982 sec]
[quick.plugins.timer:       22.401 sec]
[quick.scalacheck.timer:    30.409 sec]
[quick.scalap.timer:        27.932 sec]
[quick.partest.timer:       28.870 sec]

Anyway, from my POV I can live with a slower VM which doesn't crash on recursion and where having support for value classes, multiple return values and specialization is actually imaginable in the short-/mid-term in terms of willingness of the maintainers and the code-base.

[Francois]

That seems really cool!
What is the boot time of Avian compared to HotSpot? Is it significantly different (better?). If only Scala could have something as quick (or at least, in the same order of time) to boot as Perl/Ruby/Python, it could be such a great tool to build async "sysadmin" service (thinking about cloud, but also more simple file processing, etc...)

Cheers, and it's really cool to see such a work !

-- 
Francois ARMAND
http://fanf42.blogspot.com
http://www.normation.com

[Simon Ochsenreither]

What is the boot time of Avian compared to HotSpot? Is it significantly different (better?). If only Scala could have something as quick (or at least, in the same order of time) to boot as Perl/Ruby/Python, it could be such a great tool to build async "sysadmin" service (thinking about cloud, but also more simple file processing, etc...)

The only thing I remember is that REPL start-up was slower the last time I checked, but there is no reason why it should be that way. But yes, having a Scala shell with instantaneous start-up is my hope, too.
Avian has to keep a lot less baggage around, so (without much experience) I'd say benchmarking and improving the slow parts might help a lot. (Not considering that the REPL start-up would be a lot faster on any runtime if core classes (Predef and friends) would reference (and therefore enforce class loading and initialization) less stuff ... scala.util.Properties is one of the main offenders as far as I remember.)

[Simon Ochsenreither]

Update: I just checked it and start-up time is comparable.

But it seems that I just discovered a new issue ... looks like the recent REPL work broke something:

scala> 1
java.lang.IllegalArgumentException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:751)
    at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:966)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:613)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:612)
    at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
    at scala.tools.nsc.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:612)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:539)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:535)
    at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:566)
    at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:611)
    at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:523)
    at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:379)
    at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:396)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:679)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:665)
    at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:665)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:95)
    at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:665)
    at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:80)
    at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:93)
    at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:102)
    at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

Yet another reason why we should test more run-times ...

[Adriaan Moors]

Hi Simon,

That's great news!

I'd also love to see other community-driven back-ends, and we'll do our best to facilitate including them in our testing, tooling,...

We've just hired someone to work (among other things) on our build infrastructure.

Testing on multiple platforms, and, more generally, accepting contributions to our jenkins setup were both part of the job description :-)

The idea is to generalize the current jenkins setup to manage its configuration on GitHub, so that contributions can be made using pull requests.

cheers

adriaan

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

See you at Scala Days (June 10–12, NYC)!

[Simon Ochsenreither]

 Hi Adriaan,

that sounds great! Would it be possible to ping me as soon as the new guy in charge is ready? I always feel very uncomfortable if I repeatedly ask people whether they are ready to accept contributions yet (website, specification, etc.).

Apart from the 4 organizational points mentioned in the original post, one thing which would really help mid-/long-term would be the possibility to switch off all the rewritings/optimizations, similar to -no-specialization (e. g. the value class rewriting, extension methods, ...).

More precisely, the litmus test would be: "Is this an optimization which should ideally be done by the runtime, not the language compiler?"

I think that would help tremendously, because one could focus on improving the runtime support instead of having to undo the transformations done by scalac first.

Thanks!

Simon

[Simon Ochsenreither]

Another update: There seem to be a few remaining issues on the Avian side (mainly related to Java reflection and the REPL) which I didn't catch because I didn't properly prevent Ant from picking the wrong runtime.
The first fixes were already committed to the Avian repo.

An additional issue seems to be that the tests don't finish on Avian within the time limit set in src/partest/scala/tools/partest/nest/DirectRunner.scala on line 43:

    try if (!pool.awaitTermination(4, TimeUnit.HOURS))
I'll create a PR to increase it (an alternative would be to create a new option so that it can be set by the user without having to recompile partest).

Bye,

Simon

[Adriaan Moors]

Making the time-out configurable would be preferable. Thanks!

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Som Snytt]

I'll add the timeout option, if you like.  I have a couple of partest PRs in the queue.

In a branch, I have --timeout to mean per-test timeout.  (Testicolor doesn't do per-test timeouts.)

How about --wait-time for the total wait time?  Or --fail-after?  Using Duration to parse.

Another slogan from Star Trek: "Out here in the Delta quadrant, every old trick is new again."

On Thu, Apr 18, 2013 at 5:27 AM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

--

[Som Snytt]

Does anyone have one of these classic Scala t-shirts?  I can't find it on ebay.

The one that says:

"I hang with partest!"

Nostalgia is making a comeback.

[Simon Ochsenreither]

I'll add the timeout option, if you like.

Yes, that would be great!

[Som Snytt]

The Par-Test timeout option will require some community support for review.

https://github.com/scala/scala/pull/2418

I spell it Par-Test like Spider-Man; I don't mean it ironically.  With the help of Jenkins, as the song has it, "He arrives just in time."

On Thu, Apr 18, 2013 at 12:51 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

I'll add the timeout option, if you like.

Yes, that would be great!

--

[Simon Ochsenreither]

Here a few test failures where it seems that Scala is to blame:

Quoting Joel:

These are the failures I'm currently seeing:

!!  99 - run/t5375.scala                           [non-zero exit code]
!! 123 - run/t6488.scala                           [non-zero exit code]
!! 702 - run/stream_length.scala                   [non-zero exit code]
!! 897 - run/t4023.scala                           [output differs]
!! 1033 - run/t5353.scala                           [output differs]
!! 1277 - run/reflection-magicsymbols-

invoke.scala  [output differs]

There are probably more, but the timeout is still causing the suite to
exit prematurely, and I haven't bothered to change it yet.

The --timeout option is already in the PR queue.
 

   t5375: our friend CompositeThrowable, which fails consistently on both
Avian and HotSpot on my machine

I already raised this in What's wrong with an API which non-deterministically returns either A or B(Set(A, ...))?.

   t6488: fails because scala.sys.process.ProcessImpl.SimpleProcess.destroy
uses Thread.stop, which is not implemented in Avian and probably never
will be.  Any code that uses Thread.stop is broken and needs to be fixed.

Can someone comment on this? Java Thread Primitive Deprecation FAQ.
 

   stream_length: fails with an OOME; haven't studied it closely enough yet
to know why

Have not looked at it yet.

   t4023: expected output depends on Class.getDeclaredClasses returning an
array sorted in a certain order.  Avian provides the same data as HotSpot
but in a different order, which appears to be valid since the JavaDoc says
nothing about the order.  I think the test ought to explicitly sort the
array if it's going to rely on a specific order.

I have already fixed this right now (it's a test I wrote myself ... argh).

   t5353 and reflection-magicsymbols-invoke: expected output depends on
ClassCastExceptions and CloneNotSupportedExceptions having certain message
formats, respectively.  Avian uses different formats for the messages than
HotSpot, hence the failures.  Again, this seems valid, since the messages
attached to exceptions thrown by the VM are not part of any specification
and may even change from one release to the next.

I think the tests could be made a bit less strict, but I'm wondering whether changing Avian's exception messages to the “standard” ones is nicer. (I'll need to look at them. If the messages are better, I would fix the tests. If they are just different, I would change it in Avian.)

Can someone have a look at the Thread.stop issue?

Thanks,

Simon

[Rex Kerr]

On Tue, Apr 23, 2013 at 6:05 AM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

   t6488: fails because scala.sys.process.ProcessImpl.SimpleProcess.destroy
uses Thread.stop, which is not implemented in Avian and probably never
will be.  Any code that uses Thread.stop is broken and needs to be fixed.

Can someone comment on this? Java Thread Primitive Deprecation FAQ.

I fundamentally disagree with "Any code that uses Thread.stop is broken and needs to be fixed." except inasmuch as the Java threading model is broken and needs to be fixed.

Stop is non-cooperative.  Sometimes you need to get rid of something that's misbehaving.  In uses in the past, I have not noticed that interrupt and stop in fact only work in the same places (and it need not be that way in theory).

I do think it's worth re-examining any code that uses stop, but sometimes (observed) stop functionality is the only way to reasonably achieve something, and the possibility of damaged objects is less bad than e.g. leaving a CPU core running at 100% forever after.

  --Rex

[Lukas Rytz]

We use "Tread.stop" when grading student assignments for the Coursera course. We use ScalaTest but run

every test in a separate thread that we kill after some timeout - there's a lot of infinite loops being submitted :)

I didn't find any other way of doing that.

 

  --Rex

[Simon Ochsenreither]

We use "Tread.stop" when grading student assignments for the Coursera course. We use ScalaTest but run

every test in a separate thread that we kill after some timeout - there's a lot of infinite loops being submitted :)

I didn't find any other way of doing that.

Well, the only safe way is to spin up a new VM for every assingment, because—even with Thread.stop()—a submitter can DoS your whole VM by synchronizing on classOf[Thread] until you stop it.
E. g. if you stop a Thread after x minutes, an attacker can just submit one assignment every x minutes to prevent any grading from happening on that VM.

[Simon Ochsenreither]

I fundamentally disagree with "Any code that uses Thread.stop is broken and needs to be fixed." except inasmuch as the Java threading model is broken and needs to be fixed.

Stop is non-cooperative.  Sometimes you need to get rid of something that's misbehaving.  In uses in the past, I have not noticed that interrupt and stop in fact only work in the same places (and it need not be that way in theory).

I do think it's worth re-examining any code that uses stop, but sometimes (observed) stop functionality is the only way to reasonably achieve something, and the possibility of damaged objects is less bad than e.g. leaving a CPU core running at 100% forever after

I agree with that in general. One should stay away from Thread.stop(), but when working with uncooperative threads, there is not much which can be done differently.

I'm wondering whether there is a safe subset of behaviour which can be implemented in a sane way.
E. g. allow stop()

• if the Thread isn't running (e. g. start() wasn't called yet)
  
• if the Thread didn't lock/synchronize on anything (or at least nothing observable on other threads)
  

Would this make sense?

By the way:

    static void sneakyThrow(Throwable t) {
        Thread.currentThread().stop(t);
    }

Now that's a scary sneakyThrow implementation. :-)

[Som Snytt]

I just came across t6488, which attracted my notice because of catch _ => warning.

Anyone interested can comment on:
https://github.com/scala/scala/pull/2478

I changed destroy on destroy to judicious checking of interrupted flag, but I haven't grokked all of the process package yet.

I'll look at it again in a little while.

Simon

--

[Simon Ochsenreither]

http://mail.openjdk.java.net/pipermail/core-libs-dev/2013-May/016974.html

[Simon Ochsenreither]

Is there any update on this?

With all PRs in the pull request queue applied, I'm now down to TWO test failures for all tests in pos, neg, run and jvm combined. I'd like to get some testing/build infrastructure up as soon as possible now to reduce the chance of regressions.

Can someone tell me what to do next?