Test suite hangs with ThreadPoolTaskSupport, issues with task support detection

[Simon Ochsenreither]

Hi,

while Joel and I bootstrapped and tested Scala on Avian, we both encountered that the test suite hangs in test/files/scalacheck/parallel-collections.

This seems to be a Scala issue, Joel discovered that the same also happens on Hotspot when using ThreadPoolTaskSupport:

Nevermind, I figured it out.  There's a lovely bit of code in
scala/collection/parallel/

package.scala:

   private[parallel] def getTaskSupport: TaskSupport =
     if (scala.util.Properties.isJavaAtLeast("1.6")) {
       val vendor = scala.util.Properties.javaVmVendor
       if ((vendor contains "Oracle") || (vendor contains "Sun") || (vendor contains "Apple")) new ForkJoinTaskSupport
       else new ThreadPoolTaskSupport
     } else new ThreadPoolTaskSupport

If I change that to just

   private[parallel] def getTaskSupport: TaskSupport = new ForkJoinTaskSupport

the parallel-collections test passes on Avian and HotSpot.  However, if I
change it to

   private[parallel] def getTaskSupport: TaskSupport = new ThreadPoolTaskSupport

it hangs on both Avian and HotSpot.  In other words, the test simply won't
complete when using ThreadPoolTaskSupport, regardless of the VM, which
means the test can only pass if the VM vendor is Oracle, Sun, or Apple.
Ouch.

It looks like ThreadPoolTaskSupport is broken on all VMs.

Thread on the Avian mailing list: https://groups.google.com/d/msg/avian/Hugny4JcnDw/shb3r-3BOR8J

An additional issue is that the reading out the vendor is not sufficient to determine whether a VM supports fork join.

Joel's thread dump is attached.

Thanks and bye,

Simon

[Jason Zaugg]

On Sat, Mar 9, 2013 at 3:38 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

Hi,

while Joel and I bootstrapped and tested Scala on Avian, we both encountered that the test suite hangs in test/files/scalacheck/parallel-collections.

This seems to be a Scala issue, Joel discovered that the same also happens on Hotspot when using ThreadPoolTaskSupport:

Good catch. Can you please open a ticket?

-jason 

[Simon Ochsenreither]

I'm already on it. I'll create two: One for the hand and one for the detection issue.

[Simon Ochsenreither]

Here we go:

• SI-7236: Using ThreadPoolTaskSupport causes test to hang
• SI-7237: Insufficient task support detection mechanism chooses fails to choose ForkJoin even when available/supported

[Simon Ochsenreither]

I have increased the priority of SI-7237 to Critical, because if 2.11 starts using invokedynamic, we are basically stuck with a broken task support library until 2.10.2.

[√iktor Ҡlang]

Just a quick question, since we embed a modern version of FJP in the stdlib, why don't we just always use that?

On Sat, Mar 9, 2013 at 4:36 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

I have increased the priority of SI-7237 to Critical, because if 2.11 starts using invokedynamic, we are basically stuck with a broken task support library until 2.10.2.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe - The software stack for applications that scale
Twitter: @viktorklang

[Simon Ochsenreither]

Just a quick question, since we embed a modern version of FJP in the stdlib, why don't we just always use that?

That's a pretty good question. Is it possible that it was done because Java 5 lacked some concurrency classes or fixes necessary for ForkJoin?

[Simon Ochsenreither]

Just a quick question, since we embed a modern version of FJP in the stdlib, why don't we just always use that?

Maybe Aleksandar knows more: https://github.com/scala/scala/commit/c9ae821b7702169147fe324eb160710623c16280

[√iktor Ҡlang]

I'm currently cooking a rib-eye steak, could someone try to instantiate and run the FJP on a 1.5 JDK?

Cheers,

√

On Sat, Mar 9, 2013 at 5:30 PM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

Just a quick question, since we embed a modern version of FJP in the stdlib, why don't we just always use that?

Maybe Aleksandar knows more: https://github.com/scala/scala/commit/c9ae821b7702169147fe324eb160710623c16280

--

You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Aleksandar Prokopec]

We should switch to using FJP everywhere a long time ago, yes.
ThreadPoolTaskSupport was meant to serve as a support for jdk5. Since we dropped the 1.5 support in Scala, the issues with ThreadPoolTaskSupport (in particular, potential creation of unlimited number of threads, and various performance issues) were never fixed (as I recall, this was decided on a Scala meeting some time ago).
And I think that fixing that particular deadlock would require reimplementing much of the logic of FJP.

So imo, we should remove TPTS from trunk altogether. All 1.6 compliant JVMs should support FJP anyway.
I'll take a closer look at this on Monday.

Simon Ochsenreither <simon.och...@gmail.com> wrote:

Just a quick question, since we embed a modern version of FJP in the stdlib, why don't we just always use that?

Maybe Aleksandar knows more: https://github.com/scala/scala/commit/c9ae821b7702169147fe324eb160710623c16280

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

[Simon Ochsenreither]

I have tested ForkJoinTaskSupport and ThreadPoolTaskSupport:

- ForkJoinTaskSupport works on IBM J9 (6SR13), Avian and OpenJDK
- ThreadPoolTaskSupport hangs on all three

Therefore, I have made three pull requests:

- Always choose ForkJoinTaskSupport (2.11): https://github.com/scala/scala/pull/2249
- Always choose ForkJoinTaskSupport (2.10)¹: https://github.com/scala/scala/pull/2252
- Deprecate ThreadPoolTaskSupport and friends (2.11): https://github.com/scala/scala/pull/2251

¹ By the way, what's the priority to choose in JIRA to not get an issue ignored that was marked as “affects 2.10.1-RC3”? :-)

[Adriaan Moors]

On Wed, Mar 13, 2013 at 8:55 AM, Simon Ochsenreither <simon.och...@gmail.com> wrote:

¹ By the way, what's the priority to choose in JIRA to not get an issue ignored that was marked as “affects 2.10.1-RC3”? :-)

It's not about priority. It's about the "Fix by" version. That documents the intention of fixing this issue by that version. The "Affects" version is the version where the bug was first discovered.

When a version is released, we make sure that all issues to be fixed by that version are indeed fixed. Bugs that affect this version but are not marked to be fixed by that version are "known issues".

--

See you at Scala Days (June 10–12, NYC)!

[Simon Ochsenreither]

Scala actors have the same issue. Have a look at https://github.com/scala/scala/blob/master/src/actors/scala/actors/scheduler/ThreadPoolConfig.scala#L48.

Joel reports that this causes concurrent-stream.scala to hang randomly.

I'll have a look at it, but I would really love to understand why this check (especially “on IBM J9 1.6 do not use ForkJoinPool”) was added in the first place.

[Philipp Haller]

The check was added, because earlier releases of J9 1.6 did not implement all methods of Unsafe used in the ForkJoinPool. At the time, we did our share of talking with the people responsible at IBM about completing Unsafe support. It's great that we don't need this check any more.

Cheers,

Philipp

[Simon Ochsenreither]

Hi Philipp,

The check was added, because earlier releases of J9 1.6 did not implement all methods of Unsafe used in the ForkJoinPool. At the time, we did our share of talking with the people responsible at IBM about completing Unsafe support. It's great that we don't need this check any more.

Thanks a lot for this piece of information!

Bye,

Simon