Receiver annotation syntax: experimental evaluation
Michael Ernst
describe the type of the implicit "this" parameter to a method. Different
people advocate different syntaxes, but have no evidence beyond their own
aesthetic judgment regarding which syntax is best. I have performed an
experiment that helps to settle the issue.
As background, recall that it is easy to annotate the type of an explicit
parameter. For example, consider this method:
class MyClass {
File myMethod(@ReadOnly Date d, Window w) { ... }
}
A call
aMyClass.myMethod(aDate, aWindow)
might modify "aDate", but will not modify "aWindow" or "aMyClass".
Java programmers also need to specify whether the method modifies its
receiver, which is "aMyClass" in the call above. Whether the method
modifies its receiver is orthogonal to whether the method modifies any
other argument.
I administered a survey to 124 undergraduates who had taken one year of
Java programming. Different participants were shown different proposed
receiver syntaxes and asked to guess the meaning. We computed the
percentage of students who guessed correctly, using relatively strict
grading.
56% File myMethod(@ReadOnly MyClass this, Date d, Window w) { ... }
30% File myMethod @ReadOnly MyClass (Date d, Window w) { ... }
23% File @ReadOnly myMethod(Date d, Window w) { ... }
20% File myMethod(Date d, Window w) @ReadOnly { ... }
11% File myMethod @ReadOnly (Date d, Window w) { ... }
3% File myMethod(Date d, Window w @ReadOnly) { ... }
The results show a clear winner, and the differences are statistically
significant at the p=.0005 level using even the conservative Kruskal-Wallis
H-test.
Each participant was then informed of the intended meaning (the receiver is
read-only) and asked his or her opinion of the syntax, on a 7-point Likert
scale (1=strongest disapproval, 7=strongest approval). The results are:
4.1 File myMethod(@ReadOnly MyClass this, Date d, Window w) { ... }
4.3 File myMethod @ReadOnly MyClass (Date d, Window w) { ... }
4.7 File @ReadOnly myMethod(Date d, Window w) { ... }
4.4 File myMethod(Date d, Window w) @ReadOnly { ... }
4.0 File myMethod @ReadOnly (Date d, Window w) { ... }
3.2 File myMethod(Date d, Window w @ReadOnly) { ... }
The participants' opinions of the syntax were largely uncorrelated with how
understandable each syntax is. In other words, asking participants their
opinion of a syntax is ineffective at best and misleading at worst. It
would be interesting to repeat the experiment to evaluate the syntax
intuitions of skilled (as opposed to novice) Java programmers. All of the
syntax choices were proposed by skilled Java programmers!
The survey instruments and experimental data are available on request.
-Mike
Jonathan Aldrich
I think it's great that you've performed an experiment to judge the
relative understandability to uninformed subjects of different syntaxes
for receiver annotations! Like many who are interested in JSR 308, I
think annotations on the receiver are very important, and getting the
design right is highly worthwhile: thus empirical data is welcome.
A caution on your experiment, however. You did not measure "which
syntax is best" in general (the introductory text of your message seems
to make this claim, at least implicitly). Rather, you measured which
syntax is most understandable for a Java programmer *who has never seen
the syntax before*. Looking at the syntaxes you proposed, the result is
hardly surprising--the syntaxes where @ReadOnly was associated directly
with "MyClass" or "this" did best, and these syntaxes also give the
uninformed programmer the most data to drive their intuition. Of course
it is great to have empirical evidence even if it is not surprising!
So how to explain the subjects' preference, once informed of the
intention, for another syntax? I happen to share their preference--my
past annotation designs have used the syntax rated 4.4, but your study
makes a good preliminary argument that the syntax rated 4.7 may be
better (both on account of preferences and a small learnability benefit).
My guess is: your subjects are primarily considering issues other than
understandability to an uninformed programmer. They are thinking about
how succinct the syntax is; whether it is confusing to add "this" as an
extra parameter; how easily the syntax "reads" to someone who knows it;
how it fits consistently with the existing Java design, etc. And as
professional programmers, they are absolutely right to think about these
things in addition to "uninformed understandability." For a
professional programmer, the "uninformed understandability" case comes
perhaps once. After that it is mostly immaterial, and all the other
issues dominate.
If learnability is the #1 most important characteristic of a language
(Visual Basic, anyone?), we should generally be guided by the
performance of novices in studies like yours. However, in languages for
professional programmers, learnability is still an issue to be
considered, but is much less important than longer-term usability of the
language. We run the risk of designing a language that is easily
learned, but which no one wants to learn.
Hence I strongly disagree that asking asking participants their opinion
of a syntax is ineffective at best and misleading at worst (at least to
the extent this is a generalization). Yes, it is ineffective and
misleading with respect to uninformed understandability, but that does
not mean it is not effective and meaningful with respect to other
attributes.
So, with professional programmers as our audience, I think as language
designers we should listen mostly to your subjects' preferences, and
only secondarily to their uninformed task performance. It might be even
better to gather data on professional programmer's performance with
different syntaxes once they are informed of their meaning, although
that may be a much more difficult study to run.
Anyway, keep up the good empirical work, but also keep it in perspective!
Best,
Jonathan
Michael Ernst
[My humble apologies for my slow response, since I wrote this message as
soon as I received yours, but somehow failed to send it and just stumbled
across the draft. Better late than never.]
Thanks for your interest in the experiment about receiver syntax learnability.
I'd like to correct some apparent misunderstandings.
First, no one has claimed that "learnability is the #1 most important
characteristic of a language". Making a strawman argument is not
destructive to an honest and productive discussion. I trust that's not
what you want or intend, which is why I bring this up. Thanks!
It was perhaps just an oversight, but your criteria for language design did
not include expressiveness. Your preferred syntax is strictly less
expressive -- for instance, it cannot express parameterized types (with
qualifiers on the parameters) nor nested (inner) types. I consider
expressiveness, consistency with the existing language, and other factors
to be very important -- I'm sorry if this wasn't apparent to you. I was
investigating whether problems with obviousness (learnability) cast doubt
on any particular syntax proposals, and whether likeability was correlated
with obviousness (surprisingly, it was not!). Obviousness may be
particularly important for a syntax that will be used relatively
infrequently, since most programmers won't become expert in it. My
experiment isn't the final word: every experiment has limitations, and
other experiments might be useful, though they would have costs and
limitations of their own. For instance, you suggested evaluating use by
experts in practice, but you didn't outline a way to overcome familiarity
bias. (That's why my experiment used novices.) Even in limited
experiments for Java's lambda syntax, the results showed a strong
familiarity bias -- people preferred whatever they had gotten used to.
You seem to have made some additional mistakes in your analysis, and in
your arguments that cast doubt on the results.
> So how to explain the subjects' preference, once informed of the intention,
> for another syntax?
They had no such preference. None of the subjects were shown multiple
syntaxes -- they just gave their opinion of the one syntax they had seen.
> I happen to share their preference--my past annotation
> designs have used the syntax rated 4.4, but your study makes a good
> preliminary argument that the syntax rated 4.7 may be better (both on
> account of preferences and a small learnability benefit).
Other than the one truly bad proposal, the other differences in preferences
are not statistically significant. It would be unscientific to base any
decisions on them -- even if it seems to support your opinion.
You are certainly welcome to base a language design on intuition rather
than data -- and successful languages have been designed this way! But I
feel that being open to evidence can enrich the design process.
-Mike
Jonathan Aldrich
> First, no one has claimed that "learnability is the #1 most important
> characteristic of a language"
The way your email read to me--"which syntax is best," "an experiment
that helps to settle the issue," and "asking participants their
opinion...is ineffective at best"--seemed to imply this claim.
I suspected this was not your intention, and I'm glad that's the case.
Nevertheless, I felt it was important to point out that multiple factors
should be used in evaluating syntax, and that IMO learnability is rarely
the most important factor in a language designed for professionals. So
I think we are in agreement, at least about including multiple factors.
> It was perhaps just an oversight, but your criteria for language design did
> not include expressiveness. Your preferred syntax is strictly less
> expressive -- for instance, it cannot express parameterized types (with
> qualifiers on the parameters) nor nested (inner) types.
The purpose of my message was not to articulate a complete set of
criteria--it was mainly to point out that there should be more than one
criterion ;-)
With respect to expressiveness, your post didn't specify how either
syntax can express parameterized or inner types, so it's hard to say
which is more or less expressive in this particular way. Our work,
using a syntax similar to the one you say is inexpressive, did include
(admittedly clunky) support for parameterized types. So I agree that
coming up with a clean way to do so is important!
>> So how to explain the subjects' preference, once informed of the intention,
>> for another syntax?
>
> They had no such preference. None of the subjects were shown multiple
> syntaxes -- they just gave their opinion of the one syntax they had seen.
Ah, I see...this wasn't obvious to me from your original post. At least
there was a preference in the aggregate--though if, as your most recent
message clarified, most of the orderings were not statistically
significant, one should not put too much weight on them.
In any case, taken in context, your experimental evidence is definitely
welcome here! I don't think intuition will ever leave language design,
but as you say, evidence can and definitely should enrich the design
process.
Best,
Jonathan