I've been digging around about in the whole fork/join framework since it was included with JDK7. Nothing really new, but seemed like a good time to dig into the specifics. I ran into this article/rant: http://www.coopsoft.com/ar/CalamityArticle.html
Now, obviously, this guy has a biased viewpoint, having written a 'Fork Join Conqueror' (http://coopsoft.com/ar/ConquerArticle.html) and all... Still, he seems to raise some valid points it seems. Anyone care to discuss/debunk his arguments?
I can retort a few of these, but certainly not all of it.
--Exceedingly Complex--
Yes, but, complaining that F/J (ab)uses com.sun.misc.Unsafe is just stupid, in my opinion. The whole point of F/J is that _it_ abstracts away the considerable complexity inherent in aggressively efficient multicore usage so that the code written using the F/J framework is simple. Also, a framework like F/J is one of the few times you actually get to adopt the maxim "Micro-optimize this stuff to pieces, I don't care if the code becomes far more difficult to manage going forward". That's what core frameworks have to do to get the job done. For example, quite a few code in Math is very, lets be nice and say 'uniquely' written, i.e, complete gobbledygook with a comment 9x as large as the method explaining the arcane voodoo magicks going on, but, nobody should be complaining about this. Math is complex so your code doesn't have to be.
Why is F/J such as a mess of inheritance? Yeah, well, I kinda agree with the ranter on that one. inheritance isn't, as far as I know, a clear win for performance (the opposite, usually), and I doubt it makes the API any cleaner either.
--Unstable--
We have but the word of our ranter. The fact that F/J has a long history of tickets, assuming they've all been fixed, doesn't really say much about how unstable this piece of software is. All software has bugs, a project that carefully maintains a list and is open to 3rd party tickets shouldn't be punished for that. Let's flip it around: If F/J did not have an extensive bug list, would that mean it IS stable? That's clearly a silly argument.
Multicore is inherently complicated, though outsiders usually think "Geez, this has got to be much, much simpler than these supposed rocket scientists are making it!". Guess what? Only outsiders hold that view. Once you tussle with trying to write efficient multi-core code that works on all platforms and acts consistently, you start to agree that it's just that complicated, period.
--Lack of attributes--
A lot of these I just don't follow at all. What does this mean?: "There is no error detection/recovery/reporting (other then thrown exceptions)"? What kind of detection/recovery/reporting should F/J be doing? It's core CPU work, errors at this level are invariably unrecoverable and highly unexpected (your CPU or OS is buggy, or the framework is), unless the cause is a bug in hosted code, but propagating exceptions back out of the fork/join workers is exactly how a library is supposed to handle this, so.... what's the complaint again? I don't understand.
Stall detection, anomaly detection, and performance monitoring is nice, but, I'm kind of at a loss in regards to the flow of the rant. First the argument is that this should be simpler, then the argument turns into: But it's missing all these possible but quite complicated features. Which way is it?
--Academic exercise--
This stuff is beyond my comprehension, unfortunately.
--Inadequate in scope--
F/J working on either J2ME or an EE virtual server buit around RMIing calls around the server park? Come on, that's a preposterous idea. It would be nice, but, so are unicorns.
I can only comment on the use of Unsafe and inheritance and performance. There are cases, and F/J is most likely one of them, where inheritance could be a win. Getting separation putting memory distance between shared an non-shared variables can result in them being place in difference cache lines. That can have the effect of prevent multi-thread access from causes each thread to cause the other threads to continuously flush their cache. Cache flushing can be a huge drain on performance.
Until we get a normalized path to CAS that is baked into the JMM or Java Language Specification, we're stuck with unsafe.
Regards, Kirk
On Oct 21, 2011, at 7:49 AM, Reinier Zwitserloot wrote:
> I can retort a few of these, but certainly not all of it.
> --Exceedingly Complex--
> Yes, but, complaining that F/J (ab)uses com.sun.misc.Unsafe is just stupid, in my opinion. The whole point of F/J is that _it_ abstracts away the considerable complexity inherent in aggressively efficient multicore usage so that the code written using the F/J framework is simple. Also, a framework like F/J is one of the few times you actually get to adopt the maxim "Micro-optimize this stuff to pieces, I don't care if the code becomes far more difficult to manage going forward". That's what core frameworks have to do to get the job done. For example, quite a few code in Math is very, lets be nice and say 'uniquely' written, i.e, complete gobbledygook with a comment 9x as large as the method explaining the arcane voodoo magicks going on, but, nobody should be complaining about this. Math is complex so your code doesn't have to be.
> Why is F/J such as a mess of inheritance? Yeah, well, I kinda agree with the ranter on that one. inheritance isn't, as far as I know, a clear win for performance (the opposite, usually), and I doubt it makes the API any cleaner either.
> --Unstable--
> We have but the word of our ranter. The fact that F/J has a long history of tickets, assuming they've all been fixed, doesn't really say much about how unstable this piece of software is. All software has bugs, a project that carefully maintains a list and is open to 3rd party tickets shouldn't be punished for that. Let's flip it around: If F/J did not have an extensive bug list, would that mean it IS stable? That's clearly a silly argument.
> Multicore is inherently complicated, though outsiders usually think "Geez, this has got to be much, much simpler than these supposed rocket scientists are making it!". Guess what? Only outsiders hold that view. Once you tussle with trying to write efficient multi-core code that works on all platforms and acts consistently, you start to agree that it's just that complicated, period.
> --Lack of attributes--
> A lot of these I just don't follow at all. What does this mean?: "There is no error detection/recovery/reporting (other then thrown exceptions)"? What kind of detection/recovery/reporting should F/J be doing? It's core CPU work, errors at this level are invariably unrecoverable and highly unexpected (your CPU or OS is buggy, or the framework is), unless the cause is a bug in hosted code, but propagating exceptions back out of the fork/join workers is exactly how a library is supposed to handle this, so.... what's the complaint again? I don't understand.
> Stall detection, anomaly detection, and performance monitoring is nice, but, I'm kind of at a loss in regards to the flow of the rant. First the argument is that this should be simpler, then the argument turns into: But it's missing all these possible but quite complicated features. Which way is it?
> --Academic exercise--
> This stuff is beyond my comprehension, unfortunately.
> --Inadequate in scope--
> F/J working on either J2ME or an EE virtual server buit around RMIing calls around the server park? Come on, that's a preposterous idea. It would be nice, but, so are unicorns.
> Didn't feel like going further into it today.
> -- > You received this message because you are subscribed to the Google Groups "The Java Posse" group. > To view this discussion on the web visit https://groups.google.com/d/msg/javaposse/-/J22idazMQUEJ. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
Sender: javaposse@googlegroups.com
Date: Fri, 21 Oct 2011 10:41:51 To: Ray Hindman<javaposse@googlegroups.com>
Reply-To: javaposse@googlegroups.com
Subject: Re: [The Java Posse] Re: Fork/Join criticism
Hi,
I can only comment on the use of Unsafe and inheritance and performance. There are cases, and F/J is most likely one of them, where inheritance could be a win. Getting separation putting memory distance between shared an non-shared variables can result in them being place in difference cache lines. That can have the effect of prevent multi-thread access from causes each thread to cause the other threads to continuously flush their cache. Cache flushing can be a huge drain on performance.
Until we get a normalized path to CAS that is baked into the JMM or Java Language Specification, we're stuck with unsafe.
Regards,
Kirk
On Oct 21, 2011, at 7:49 AM, Reinier Zwitserloot wrote:
> I can retort a few of these, but certainly not all of it.
> --Exceedingly Complex--
> Yes, but, complaining that F/J (ab)uses com.sun.misc.Unsafe is just stupid, in my opinion. The whole point of F/J is that _it_ abstracts away the considerable complexity inherent in aggressively efficient multicore usage so that the code written using the F/J framework is simple. Also, a framework like F/J is one of the few times you actually get to adopt the maxim "Micro-optimize this stuff to pieces, I don't care if the code becomes far more difficult to manage going forward". That's what core frameworks have to do to get the job done. For example, quite a few code in Math is very, lets be nice and say 'uniquely' written, i.e, complete gobbledygook with a comment 9x as large as the method explaining the arcane voodoo magicks going on, but, nobody should be complaining about this. Math is complex so your code doesn't have to be.
> Why is F/J such as a mess of inheritance? Yeah, well, I kinda agree with the ranter on that one. inheritance isn't, as far as I know, a clear win for performance (the opposite, usually), and I doubt it makes the API any cleaner either.
> --Unstable--
> We have but the word of our ranter. The fact that F/J has a long history of tickets, assuming they've all been fixed, doesn't really say much about how unstable this piece of software is. All software has bugs, a project that carefully maintains a list and is open to 3rd party tickets shouldn't be punished for that. Let's flip it around: If F/J did not have an extensive bug list, would that mean it IS stable? That's clearly a silly argument.
> Multicore is inherently complicated, though outsiders usually think "Geez, this has got to be much, much simpler than these supposed rocket scientists are making it!". Guess what? Only outsiders hold that view. Once you tussle with trying to write efficient multi-core code that works on all platforms and acts consistently, you start to agree that it's just that complicated, period.
> --Lack of attributes--
> A lot of these I just don't follow at all. What does this mean?: "There is no error detection/recovery/reporting (other then thrown exceptions)"? What kind of detection/recovery/reporting should F/J be doing? It's core CPU work, errors at this level are invariably unrecoverable and highly unexpected (your CPU or OS is buggy, or the framework is), unless the cause is a bug in hosted code, but propagating exceptions back out of the fork/join workers is exactly how a library is supposed to handle this, so.... what's the complaint again? I don't understand.
> Stall detection, anomaly detection, and performance monitoring is nice, but, I'm kind of at a loss in regards to the flow of the rant. First the argument is that this should be simpler, then the argument turns into: But it's missing all these possible but quite complicated features. Which way is it?
> --Academic exercise--
> This stuff is beyond my comprehension, unfortunately.
> --Inadequate in scope--
> F/J working on either J2ME or an EE virtual server buit around RMIing calls around the server park? Come on, that's a preposterous idea. It would be nice, but, so are unicorns.
> Didn't feel like going further into it today.
> -- > You received this message because you are subscribed to the Google Groups "The Java Posse" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/javaposse/-/J22idazMQUEJ.
> To post to this group, send email to javaposse@googlegroups.com.
> To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
-- You received this message because you are subscribed to the Google Groups "The Java Posse" group.
To post to this group, send email to javaposse@googlegroups.com.
To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
The alternative to inheritance is general object composition -- which allows each of the objects to be strewn all of the heap with respect to one another.
Inheritance, however, stuffs all the fields into 1 bucket that all moves together.
Object fields are pointers to elsewhere in the heap in any case, of course. I'm talking about primitives.
On 10/21/2011 4:56 AM, ricky.clark...@gmail.com wrote:
> I can only comment on the use of Unsafe and inheritance and > performance. There are cases, and F/J is most likely one of them, > where inheritance could be a win. Getting separation putting memory > distance between shared an non-shared variables can result in them > being place in difference cache lines. That can have the effect of > prevent multi-thread access from causes each thread to cause the other > threads to continuously flush their cache. Cache flushing can be a > huge drain on performance.
> Until we get a normalized path to CAS that is baked into the JMM or > Java Language Specification, we're stuck with unsafe.
> Regards, > Kirk
> On Oct 21, 2011, at 7:49 AM, Reinier Zwitserloot wrote:
>> I can retort a few of these, but certainly not all of it.
>> --Exceedingly Complex--
>> Yes, but, complaining that F/J (ab)uses com.sun.misc.Unsafe is just >> stupid, in my opinion. The whole point of F/J is that _it_ abstracts >> away the considerable complexity inherent in aggressively efficient >> multicore usage so that the code written using the F/J framework is >> simple. Also, a framework like F/J is one of the few times you >> actually get to adopt the maxim "Micro-optimize this stuff to pieces, >> I don't care if the code becomes far more difficult to manage going >> forward". That's what core frameworks have to do to get the job done. >> For example, quite a few code in Math is very, lets be nice and say >> 'uniquely' written, i.e, complete gobbledygook with a comment 9x as >> large as the method explaining the arcane voodoo magicks going on, >> but, nobody should be complaining about this. Math is complex so your >> code doesn't have to be.
>> Why is F/J such as a mess of inheritance? Yeah, well, I kinda agree >> with the ranter on that one. inheritance isn't, as far as I know, a >> clear win for performance (the opposite, usually), and I doubt it >> makes the API any cleaner either.
>> --Unstable--
>> We have but the word of our ranter. The fact that F/J has a long >> history of tickets, assuming they've all been fixed, doesn't really >> say much about how unstable this piece of software is. All software >> has bugs, a project that carefully maintains a list and is open to >> 3rd party tickets shouldn't be punished for that. Let's flip it >> around: If F/J did not have an extensive bug list, would that mean it >> IS stable? That's clearly a silly argument.
>> Multicore is inherently complicated, though outsiders usually think >> "Geez, this has got to be much, much simpler than these supposed >> rocket scientists are making it!". Guess what? Only outsiders hold >> that view. Once you tussle with trying to write efficient multi-core >> code that works on all platforms and acts consistently, you start to >> agree that it's just that complicated, period.
>> --Lack of attributes--
>> A lot of these I just don't follow at all. What does this mean?: >> "There is no error detection/recovery/reporting (other then thrown >> exceptions)"? What kind of detection/recovery/reporting should F/J be >> doing? It's core CPU work, errors at this level are invariably >> unrecoverable and highly unexpected (your CPU or OS is buggy, or the >> framework is), unless the cause is a bug in hosted code, but >> propagating exceptions back out of the fork/join workers is exactly >> how a library is supposed to handle this, so.... what's the complaint >> again? I don't understand.
>> Stall detection, anomaly detection, and performance monitoring is >> nice, but, I'm kind of at a loss in regards to the flow of the rant. >> First the argument is that this should be simpler, then the argument >> turns into: But it's missing all these possible but quite complicated >> features. Which way is it?
>> --Academic exercise--
>> This stuff is beyond my comprehension, unfortunately.
>> --Inadequate in scope--
>> F/J working on either J2ME or an EE virtual server buit around RMIing >> calls around the server park? Come on, that's a preposterous idea. It >> would be nice, but, so are unicorns.
>> Didn't feel like going further into it today.
>> -- >> You received this message because you are subscribed to the Google >> Groups "The Java Posse" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/javaposse/-/J22idazMQUEJ. >> To post to this group, send email to javaposse@googlegroups.com >> <mailto:javaposse@googlegroups.com>. >> To unsubscribe from this group, send email to >> javaposse+unsubscribe@googlegroups.com >> <mailto:javaposse+unsubscribe@googlegroups.com>. >> For more options, visit this group at >> http://groups.google.com/group/javaposse?hl=en.
> -- > You received this message because you are subscribed to the Google > Groups "The Java Posse" group. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to > javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/javaposse?hl=en. > -- > You received this message because you are subscribed to the Google > Groups "The Java Posse" group. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to > javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/javaposse?hl=en.
I wrote the articles and the software for fork-join. I've been doing fork-join since the mid 1980's so I know a bit about how to do it.
Doug Lea is a professor. He developed an application service. The two often don't mix.
If you take a careful, comprehensive look at the code you'll see that it is more about impressing his peers then about producing an application service that is both efficient and maintainable.
There are ten points in the calamity article. Taken together they show that the F/J framework does not belong as part of core Java. If he wants to have a private package, even part of the extensions framework, then so what? The article was not a critique of anyone, but of the framework's place as the core of all Java development.
I wrote a divide-and-conquer framework without using the Unsafe class. It is faster than the Java 7 versions. It is flexible, simple, and documented fully. There are differences between system programming and application programming. If Doug Lea wanted a system program to live inside the JVM, then he should have built one. You can download my software and compare the two. Then perhaps you'll be better able to comment on efficient, complex, etc.
putting distance between immutable fields and shared mutable fields causes them to show up in difference cache lines. Reads from different threads won't cause cache line flushing.
Regards, Kirk
On Oct 21, 2011, at 11:56 AM, ricky.clark...@gmail.com wrote:
> I can only comment on the use of Unsafe and inheritance and performance. There are cases, and F/J is most likely one of them, where inheritance could be a win. Getting separation putting memory distance between shared an non-shared variables can result in them being place in difference cache lines. That can have the effect of prevent multi-thread access from causes each thread to cause the other threads to continuously flush their cache. Cache flushing can be a huge drain on performance.
> Until we get a normalized path to CAS that is baked into the JMM or Java Language Specification, we're stuck with unsafe.
> Regards, > Kirk
> On Oct 21, 2011, at 7:49 AM, Reinier Zwitserloot wrote:
>> I can retort a few of these, but certainly not all of it.
>> --Exceedingly Complex--
>> Yes, but, complaining that F/J (ab)uses com.sun.misc.Unsafe is just stupid, in my opinion. The whole point of F/J is that _it_ abstracts away the considerable complexity inherent in aggressively efficient multicore usage so that the code written using the F/J framework is simple. Also, a framework like F/J is one of the few times you actually get to adopt the maxim "Micro-optimize this stuff to pieces, I don't care if the code becomes far more difficult to manage going forward". That's what core frameworks have to do to get the job done. For example, quite a few code in Math is very, lets be nice and say 'uniquely' written, i.e, complete gobbledygook with a comment 9x as large as the method explaining the arcane voodoo magicks going on, but, nobody should be complaining about this. Math is complex so your code doesn't have to be.
>> Why is F/J such as a mess of inheritance? Yeah, well, I kinda agree with the ranter on that one. inheritance isn't, as far as I know, a clear win for performance (the opposite, usually), and I doubt it makes the API any cleaner either.
>> --Unstable--
>> We have but the word of our ranter. The fact that F/J has a long history of tickets, assuming they've all been fixed, doesn't really say much about how unstable this piece of software is. All software has bugs, a project that carefully maintains a list and is open to 3rd party tickets shouldn't be punished for that. Let's flip it around: If F/J did not have an extensive bug list, would that mean it IS stable? That's clearly a silly argument.
>> Multicore is inherently complicated, though outsiders usually think "Geez, this has got to be much, much simpler than these supposed rocket scientists are making it!". Guess what? Only outsiders hold that view. Once you tussle with trying to write efficient multi-core code that works on all platforms and acts consistently, you start to agree that it's just that complicated, period.
>> --Lack of attributes--
>> A lot of these I just don't follow at all. What does this mean?: "There is no error detection/recovery/reporting (other then thrown exceptions)"? What kind of detection/recovery/reporting should F/J be doing? It's core CPU work, errors at this level are invariably unrecoverable and highly unexpected (your CPU or OS is buggy, or the framework is), unless the cause is a bug in hosted code, but propagating exceptions back out of the fork/join workers is exactly how a library is supposed to handle this, so.... what's the complaint again? I don't understand.
>> Stall detection, anomaly detection, and performance monitoring is nice, but, I'm kind of at a loss in regards to the flow of the rant. First the argument is that this should be simpler, then the argument turns into: But it's missing all these possible but quite complicated features. Which way is it?
>> --Academic exercise--
>> This stuff is beyond my comprehension, unfortunately.
>> --Inadequate in scope--
>> F/J working on either J2ME or an EE virtual server buit around RMIing calls around the server park? Come on, that's a preposterous idea. It would be nice, but, so are unicorns.
>> Didn't feel like going further into it today.
>> -- >> You received this message because you are subscribed to the Google Groups "The Java Posse" group. >> To view this discussion on the web visit https://groups.google.com/d/msg/javaposse/-/J22idazMQUEJ. >> To post to this group, send email to javaposse@googlegroups.com. >> To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com. >> For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "The Java Posse" group. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
> -- > You received this message because you are subscribed to the Google Groups "The Java Posse" group. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
On Fri, Oct 21, 2011 at 6:53 AM, Edward Harned <edhar...@gmail.com> wrote: > If you take a careful, comprehensive look at the code you'll see that it is > more about impressing his peers
Without passing judgment on the technical value of your article, I find this claim particularly offensive toward Doug Lea considering his history and contributions to the Java platform for the past fifteen years.
-----Original Message-----
From: Edward Harned <edhar...@gmail.com>
Sender: javaposse@googlegroups.com
Date: Fri, 21 Oct 2011 06:53:01 To: <javaposse@googlegroups.com>
Reply-To: javaposse@googlegroups.com
Subject: [The Java Posse] Re: Fork/Join criticism
I wrote the articles and the software for fork-join. I've been doing fork-join since the mid 1980's so I know a bit about how to do it.
Doug Lea is a professor. He developed an application service. The two often don't mix.
If you take a careful, comprehensive look at the code you'll see that it is more about impressing his peers then about producing an application service that is both efficient and maintainable.
There are ten points in the calamity article. Taken together they show that the F/J framework does not belong as part of core Java. If he wants to have a private package, even part of the extensions framework, then so what? The article was not a critique of anyone, but of the framework's place as the core of all Java development.
I wrote a divide-and-conquer framework without using the Unsafe class. It is faster than the Java 7 versions. It is flexible, simple, and documented fully. There are differences between system programming and application programming. If Doug Lea wanted a system program to live inside the JVM, then he should have built one. You can download my software and compare the two. Then perhaps you'll be better able to comment on efficient, complex, etc.
Ed
-- You received this message because you are subscribed to the Google Groups "The Java Posse" group.
To view this discussion on the web visit https://groups.google.com/d/msg/javaposse/-/An7pbSTlXq4J.
To post to this group, send email to javaposse@googlegroups.com.
To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
I agree with the others that a little courtesy goes a long way, even more so if you're trying to persuade people.
Okay, with that out of the way, on to the subject matter. I did not expect the author to reply on this forum... but since you are here: could you explain what you mean with the following observation in your article? 'Since the internal structure is so entangled with the calling client, there is almost no way to alter control variables'
Thanks for your long analysis, much appreciated. I partly agree with you on the complexity part, as far as the implementation goes. However the design of the framework being too complex is not as easily forgivable as the implementation of a single Math method being to complex IMO. The former is more or less exposed to application programmers (class hierarchies in public Javadoc etc.), whereas the latter is just a detail nobody bothers with.
Some of the arguments were a bit beyond my comprehension as well, that's why I started this discussion :)
This debate can be approached in many ways, I could also look in depth at each point of your article... but I'll propose a different rebuttal: first, you realize that all previous work of JSR-133 and JSR-166, i.e. the JMM and the whole of java.util.concurrent, was designed and implemented by much the same people (not just Doug) and shares many of the same traits - for one thing, j.u.c is a quite big and complex library, its implementation has all the things you hate like very complex code and intense reliance on Unsafe tricks, lots of the complexity are exposed in the public API etc. etc.
Now, you admit that all that previous work is considered a MASSIVE success? I, for one, have long enjoyed the power of the Java5+ concurrency APIs and features; they result in application code that is extremely powerful, scalable, maintainable, robust, flexible... AND simple. (At least compared to any pre-Java5 alternatives... if you can do even better, hats off to you, but it doesn't change the fact that JSRs 133/166 were awesome.) And of course it's not just me; this is a well-known success case of the Java platform.
Now you come to us and tell that FJ, which is basically the next chapter of the same book, written by the same authors and basically in the same style, is some piece of junk. Oh and by the way you authored a competing book. Do you see how difficult is to give you any credibility?
And just one more thing: yes FJ's implementation can be still improved in some places - and improved it will be. For one thing, CPU affinity is an obvious issue/opportunity, and I have no doubt that the implementation will eventually take advantage of that. This doesn't even need to wait for a public API, Oracle can introduce internal APIs to obtain and control affinity - just like it does for many other things (e.g. NUMA optimizations for memory management and GC). This can even happen in maintenance updates, no need with wait for JDK.Next. Although of course I'd like to have a public API too some day, but that will need to wait the next JDK.
On Fri, Oct 21, 2011 at 6:13 PM, opinali <opin...@gmail.com> wrote: > Now you come to us and tell that FJ, which is basically the next chapter of > the same book, written by the same authors and basically in the same style, > is some piece of junk. Oh and by the way you authored a competing book.
He also authored a competing framework, which he recommends as the only alternative to Fork/Join at the end of his article.
This is what I find pretty ironic: one of the good things in academia research, which he mocks in his email earlier, is that publications of this kind are usually followed by a whole list of references that have to be diverse and cover all kinds of topics by a lot of various authors.
Maybe he could learn a thing or two from this style of research instead of deriding it.
While I have no opinion on the the proposed "alternative" I think he is right in one case.
The popularity is in no way a measurement of quality and the JDK suffers pretty much from the approach of throwing things into it which are "good enough", but not "the best thing possible".
While often "good enough" is the best approach, it is not acceptable for a standard library.
On Saturday, October 22, 2011 12:43:17 AM UTC-4, Cédric Beust ♔ wrote:
> On Fri, Oct 21, 2011 at 6:13 PM, opinali <opi...@gmail.com> wrote:
>> Now you come to us and tell that FJ, which is basically the next chapter >> of the same book, written by the same authors and basically in the same >> style, is some piece of junk. Oh and by the way you authored a competing >> book.
> He also authored a competing framework, which he recommends as the only > alternative to Fork/Join at the end of his article.
Well, above I was using "book" as a metaphor for "framework" :) but it's also nice to remember JCIP, which is *THE* modern book about applied/practical concurrent programming - if anybody says that book is also "academic", it's crazy talk.
The single serious criticism I've ever heard about j.u.c. and the continuing work of JSR-166, is that it furthers the paradigm of shared-memory concurrency. But, like it or not, it's the appropriate thing to do at the level of the JavaSE platform and Java Language. Then you can use these as [extremely good] building blocks for higher-level languages and concurrency frameworks, like Scala+Actors etc. And j.u.c. is well designed to accommodate future innovation at the runtime or language levels - from Java syntax sugar a la C# 5, to lambdas, coroutines, affinnity..., all these things will be easy to fit, indeed they have been considered in the design.
> The single serious criticism I've ever heard about j.u.c. and the continuing > work of JSR-166, is that it furthers the paradigm of shared-memory > concurrency. But, like it or not, it's the appropriate thing to do at the > level of the JavaSE platform and Java Language. Then you can use these as > [extremely good] building blocks for higher-level languages and concurrency > frameworks, like Scala+Actors etc. And j.u.c. is well designed to > accommodate future innovation at the runtime or language levels - from Java > syntax sugar a la C# 5, to lambdas, coroutines, affinnity..., all these > things will be easy to fit, indeed they have been considered in the design.
Well here we have a real bone of contention. Java reified shared memory multithreading in the early 1990s. It was an error then and is certainly an error now -- at least as being pushed as the "one true way" of concurrency since it makes parallelism very hard. Why should it remain the "one true way" in Java? It is this closed mindset that is a big problem with Java. Java should have actor libraries, dataflow libraries, CSP, that applications folk use. Certainly shared memory multithreading is an important framework level technique, but no applications programmer should ever have to use such a low level model of concurrency and parallelism.
GPars (http://gpars.codehaus.org) brings all of these things to Groovy and Java programmers. JCSP is a CSP framework for Java. DataRush is a commercial offering from Pervasive that offers dataflow. These models thrash explicit use of shared memory multithreading for applications development.
Please can we ditch the idea that Java means using shared memory multithreading and that to use other models we have to use other languages. It is not true and it is damaging in the face of ubiquitous parallel hardware.
> On Saturday, October 22, 2011 12:43:17 AM UTC-4, Cédric Beust ♔ wrote: > On Fri, Oct 21, 2011 at 6:13 PM, opinali <opi...@gmail.com> wrote: > Now you come to us and tell that FJ, which is basically the next chapter of the same book, written by the same authors and basically in the same style, is some piece of junk. Oh and by the way you authored a competing book.
> He also authored a competing framework, which he recommends as the only alternative to Fork/Join at the end of his article.
> Well, above I was using "book" as a metaphor for "framework" :) but it's also nice to remember JCIP, which is *THE* modern book about applied/practical concurrent programming - if anybody says that book is also "academic", it's crazy talk.
> The single serious criticism I've ever heard about j.u.c. and the continuing work of JSR-166, is that it furthers the paradigm of shared-memory concurrency. But, like it or not, it's the appropriate thing to do at the level of the JavaSE platform and Java Language.
It is an appropriate thing to do if you understand impacts on hardware and dynamic vs static (one time assignment models) impacts on scalability.
> On Sat, 2011-10-22 at 04:32 -0700, opinali wrote: > [ . . . ] >> The single serious criticism I've ever heard about j.u.c. and the continuing >> work of JSR-166, is that it furthers the paradigm of shared-memory >> concurrency. But, like it or not, it's the appropriate thing to do at the >> level of the JavaSE platform and Java Language. Then you can use these as >> [extremely good] building blocks for higher-level languages and concurrency >> frameworks, like Scala+Actors etc. And j.u.c. is well designed to >> accommodate future innovation at the runtime or language levels - from Java >> syntax sugar a la C# 5, to lambdas, coroutines, affinnity..., all these >> things will be easy to fit, indeed they have been considered in the design.
> Well here we have a real bone of contention. Java reified shared memory > multithreading in the early 1990s.
I disagree, to use one model in exclusion to all others is dogma. Java supports mutability but it also nicely supports other techniques.
This article is not about Doug Lea. It is about the application service he wrote. Doug Lea is a fine scientist, educator, etc. Application programming is not his forte. The majority of code in the F/J framework is not from other but is one person. I love the JSR166 work. Use it all the time. Most of those classes are what is known as system programming. They are API's. The F/J framework is an application program. It belongs in the extensions framework, perhaps as javax.parallel. It is not the next chapter in the same book. It is a different book.
>Since the internal structure is so entangled with the calling client, there
is almost no way to alter control variables
The client code imbeds the framework in the initial call as a parameter. The framework then calls the client to do the processing. That is simple to program. But there is no way to get at the framework to change number of threads to use per call etc. A much better way is to separate the client call, server processing, and actual computation. Now the server is separately addressable to monitor performance, alter variables and save statistics.
I had many conversations with Doug Lea last year. I did a proof of concept showing him that scatter-gather was a far superior algorithm than work-stealing. Just a simple load balancing increased throughput 5-7 times. Also separating the client from the server allowed better tuning (as above.) He pretty much ignored what I said. Eventually, I built the divide-and-conquer version of Tymeac from the embarrassingly parallel version. Now developers have a choice.
The real test of scalability and performance comes when using these frameworks for applications that generate hundreds of thousands and millions of Tasks (like Scala.) Using work-stealing where Tasks all go into the same queue from which they were spawned and other threads have to go looking for work, is not the best approach.
Let's be honest here. I did not mock academic research. Research and building on prior research is the scientific method. The problem is that research on work-stealing is for operating systems controlling work on CPU's. Application programming is a whole different beast. There is no academic research on work-stealing outside a controlled environment. Cilk and jCilk use a compiler and run time to control Tasks.
I missed addressing the Unsafe class usage. First of all, why do you think they call it Unsafe? If you’re not familiar with the Class, then go look at it.
The Unsafe class is primarily for profilers and debuggers. They need direct access to memory. The concurrency utilities use Unsafe to get around the limits of wait()/notify() using park() and unpark(). There may also be an issue with the atomic instructions on CPUs. Most processors support atomic operation like FetchAndAdd and CompareAndSwap, but not in the same way. Some use LL/SC instead of CAS. So there is surely a need today for using Unsafe for atomic operations. API’s do things normal application programs don’t so it is what it is.
The F/J framework is not an API. It is an application program. That is the major difference. There is absolutely no need to program in pseudo-C to do fork-join. Here is a quote from the ForkJoinWorkerThread:
“Efficient implementation of these algorithms currently relies on an uncomfortable amount of "Unsafe" mechanics.”
Even the author knows he’s overdone it. He doesn’t use a standard deque from the concurrency utilities, he hard codes direct memory access to emulate a deque. The code in all his classes have massive amounts of Unsafe use. Why? Is it to get around the slow, clumsy work-stealing algorithm? I don’t see a need for it. It adds a huge amount of complexity. I can say from decades of experience, complexity always fails, always.
I developed four fork-join frameworks that run on Java SE, ME, and Android without any Unsafe usage. They’re faster than the F/J framework and have professional attributes. Unsafe has no place in application programming.
I would love to see an implementation backing Scala's parallel collections as a proof of concept. It should be even possible to have it as drop-in replacement because the parallelization mechanics are not exposed to the user.
What do you think? Would love to see some performance numbers ...
I really don't get your objection to unsafe. It's more like hand waving than some real concern. It gives access to instructions and until we have a Java level abstraction that is safe, I don't see why access should be taken away. Debuggers and profilers will use the JVMTI and with that you have access to everything without having to play games with pointers in Java.
> I missed addressing the Unsafe class usage. First of all, why do you think they call it Unsafe? If you’re not familiar with the Class, then go look at it.
> The Unsafe class is primarily for profilers and debuggers. They need direct access to memory. The concurrency utilities use Unsafe to get around the limits of wait()/notify() using park() and unpark(). There may also be an issue with the atomic instructions on CPUs. Most processors support atomic operation like FetchAndAdd and CompareAndSwap, but not in the same way. Some use LL/SC instead of CAS. So there is surely a need today for using Unsafe for atomic operations. API’s do things normal application programs don’t so it is what it is.
> The F/J framework is not an API. It is an application program. That is the major difference. There is absolutely no need to program in pseudo-C to do fork-join. Here is a quote from the ForkJoinWorkerThread: > “Efficient implementation of these algorithms currently relies on an uncomfortable amount of "Unsafe" mechanics.”
> Even the author knows he’s overdone it. He doesn’t use a standard deque from the concurrency utilities, he hard codes direct memory access to emulate a deque. The code in all his classes have massive amounts of Unsafe use. Why? Is it to get around the slow, clumsy work-stealing algorithm? I don’t see a need for it. It adds a huge amount of complexity. I can say from decades of experience, complexity always fails, always.
> I developed four fork-join frameworks that run on Java SE, ME, and Android without any Unsafe usage. They’re faster than the F/J framework and have professional attributes. Unsafe has no place in application programming.
> -- > You received this message because you are subscribed to the Google Groups "The Java Posse" group. > To view this discussion on the web visit https://groups.google.com/d/msg/javaposse/-/6a9PfIokayIJ. > To post to this group, send email to javaposse@googlegroups.com. > To unsubscribe from this group, send email to javaposse+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
On Saturday, October 22, 2011 7:46:40 AM UTC-4, Russel wrote:
> On Sat, 2011-10-22 at 04:32 -0700, opinali wrote: > [ . . . ] > > The single serious criticism I've ever heard about j.u.c. and the > continuing > > work of JSR-166, is that it furthers the paradigm of shared-memory > > concurrency. But, like it or not, it's the appropriate thing to do at the > (...)
> Well here we have a real bone of contention. Java reified shared memory > multithreading in the early 1990s. It was an error then and is certainly
We don't need to debate this -- the past is the past, and like it not this is the paradigm of concurrency embedded in Java and in billions of existing LOC of Java apps, this this must be supported forever and also improved continuously in the extent that's possible. I don't claim that we cannot have a new concurrency model as an additional option (hi @Kirk), I just say that the existing system must keep moving'. Deprecating the shared-memory concurrency model is not realistic - it's not like, say, deprecating entity beans for JPA or deprecating Swing for JavaFX; it's not a single library that is only used by a subset of applications, it's a core feature that is used directly or indirectly by 100% of all code (apps, core libs, third-party libs... including the implementation of alternative languages and concurrency frameworks). Even the introduction of j.u.c. in JDK 5 was not a radical break with the past; I for one, had significant legacy code written with straight threads/monitors or with other libs (like Apache commons-pool) that I was able to quickly review to benefit from j.u.c.
> an error now -- at least as being pushed as the "one true way" of > concurrency since it makes parallelism very hard. Why should it remain > the "one true way" in Java? It is this closed mindset that is a big > problem with Java. Java should have actor libraries, dataflow > libraries, CSP, that applications folk use. Certainly shared memory
I agree, but I think this kind of innovation already happens in the right places - the JVM-languages communities (Clojure, Scala etc.). We Java-lang diehards may complain about learning and adopting a new language as precondition for a new concurrency model (for one, I'm envious of Clojure's persistent collections), but that's the only *good* way to do it, because Java lacks even the slightest amount of support for these other models. For one thing, there's no real support for immutability ('final' is not good enough, we need full "constness" support a la C++).
Something like an actor library could be added to core Java, but
1) who would shoulder the effort? Oracle is busy enough improving the core paradigm (as it should), including hard low-level optimizations like lock elision etc. Concurrency is not everything, there's a ton of other big-ticket RFEs that are pet ideas of many hackers, should Oracle do them all? 2) and even assuming that Oracle has the resources to implement all our pet RFEs, should all of them be thrown into the JRE? Many people already complain of bloat when APIs that *are* used by a massive number of apps, like XML / Web Services stuff, are added to the core... 3) so maybe they add actors, only to make 50% of the community pissed of that they didn't pick some other next-gen concurrency idea, such as STM? 4) what to do when people start claiming for language syntax support for the new concurrency system? Add tons of extra syntax on top of the already big and hard-to-evolve Java language?
> multithreading is an important framework level technique, but no > applications programmer should ever have to use such a low level model > of concurrency and parallelism.
> GPars (http://gpars.codehaus.org) brings all of these things to Groovy > and Java programmers. JCSP is a CSP framework for Java. DataRush is a > commercial offering from Pervasive that offers dataflow. These models > thrash explicit use of shared memory multithreading for applications > development.
> Please can we ditch the idea that Java means using shared memory > multithreading and that to use other models we have to use other > languages. It is not true and it is damaging in the face of ubiquitous > parallel hardware.
On Saturday, October 22, 2011 10:11:21 AM UTC-4, Edward Harned wrote:
> This article is not about Doug Lea. It is about the application service he > wrote. Doug Lea is a fine scientist, educator, etc. Application programming > is not his forte. The majority of code in
You mean that Doug (plus JSR166 contributors) wrote FJ from the top of their ivory tower, without studying application behavior, without feedback from early-adopters etc.? You don't have to be a professional Chef in order to provide great veggies or meat; you only need to be a good farmer (and listen to some Chefs, read their recipes books, study their menus...).
> the F/J framework is not from other but is one person. I love the JSR166 > work. Use it all the time. Most of those classes are what is known as system > programming. They are API's. The F/J framework is an application program. It > belongs in the extensions framework, perhaps as javax.parallel. It is not > the next chapter in the same book. It is a different book.
This categorization of FJ as an "application program" is fallacious. If I have some app code that can use FJ, how exactly isn't FJ a conventional API? I would even say that FJ is a relatively low-level API, that will be often useful to build higher-level APIs (e.g. some BLAS, or image processing library) on top of it.
You make some specific criticism to FJ's API and design, and they may have merit. But why didn't you actively participate of the concurrency-interest list and JSR166 discussions (that were completely open - no need to even be part of the JSR EG)? Sorry, private talk with Doug Lea doesn't count, if you are not willing to debate your opinion more openly with the general Java concurrency community. I can find two relevant messages from you in the OpenJDK lists: one from Dec 2010 with the initial claim that FJ was no good and a pointer to an early article, another from Feb 2011 promoting the full Calamity article. Even the first message was too late, FJ was long a done deal, no wonder neither message deserved a single reply.
Your claims of 5-7X better throughput are just crazy talk until you show off the benchmark code, allow the community to validate if this doesn't suffer of benchmark flaws, including biased selection of problems / scenarios.
And a final thing, I see that jCilk extends Java, it has some kind of preprocessor. If you need to do this only to get some cool syntax sugar to avoid verbose API invocations, fine. But if you need it because the jCilk-to-Java compiler makes some important transformations / optimizations that are essential for your concurrency system... then good luck with that... you would be better off by joining forces with some other language that is better designed for concurrency (like Clojure/Scala/Fantom) or that embraces DSLs (like Groovy).
