Howreadable.com - An experiment to measure code readability

[Daniel van Berzon]

Dear All,

I recently conducted an online experiment attempting to objectively measure code readability which I thought would be of interest to this group. I would be grateful for any feedback or suggestions and would love to hear from anyone interested in collaborating in future research.

We compared the readability of various code snippets by showing them to developers and measuring the developer's ability to predict the code's output and their time spent viewing the snippet. We compared pairs of code snippets that differed only in their use of a specific coding construct, in order to measure that construct's effect on code readability.

We were able to see statistically significant differences for certain coding constructs, suggesting the possibility of formulating descriptive rules for code readability.

The results and full write-up are available on

https://howreadable.com

Thank you for your time, and I hope this is of interest,

Daniel van Berzon.

dvbe...@gmail.com

[Richard O'Keefe]

Let's consider the very first example.

var fixedCost = 200;
var monthlyCost = 20;
var result = fixedCost + monthlyCost * 12;
---------------------------------------
var fixedCost = 200;
var monthlyCost = 20;
var result = fixedCost + (monthlyCost * 12);
---------------------------------------
"Readable" to whom? Much depends on what you
are used to. I am familiar with two languages that
avoid operator precedence:
APL (right-to-left): a+b*c and a+(b*c) have the same meaning
Smalltalk (left-to-right): b*c+a and (b*c)+a mean the same.
And are these snippets in ECMAScript, Java, C#, or something else?

"Readable" by what strategy?
I should think that most readers would approach these
snippets *semantically* rather than *syntactically*. You
have have noticed that mathematics, with its huge
repertoire of operations, has no consistent global order.
There will be groups of related operators within which
an order of precedence is defined, but for example
X+Y = {x+y | x \in X \land y \in Y}
is a common definition, but I am not aware of any
agreement on the precedence order of (set) + and
\cup (union).
There need to be two kinds of controls for this.
(a) keep the names and swap the operators.
var result = fixedCost * monthlyCost + 12;
(b) keep the operators and plug in different variable names.
var útkomst = fêsteKosten + moanlikseKosten * 12;
or even x, y, z.

Processing these snippets requires the subject to do at least
two things, possibly three:
(1) read and comprehend the snippet
(2) play computer, including performing arithmetic
(3) check the result and hold it in working memory.
You are *measuring* (1)+(2)+(3) but *reporting* it as (1).
For quite a lot of people, (2) will take significant time.
This needs a control that measures how long the
subjects take to do arithmetic (or whatever else they have

When I tried doing a readability experiment once,
the subjects complained that they could not understand
100-line programs without syntax colouring and an
interactive debugger. So taking into account that about
10% of your male subjects will be red-green colour blind,
another kind of control, where some of your subjects see
*styled* snippets (underlines, italics, bold, size or font
variations) and some of them see plain snippets, is warranted.
Not for *all* of your snippets, but for *enough* to tell whether
you have a problem. Now my colour vision is normal, but these
particular snippets are hard for me to read because the numbers
are purple. Not just purple, but a purple that is *darker* than
other tokens. The very tokens I need to read most clearly are
physically the hardest for me to read. At a minimum, a scheme
(or schemes) of colouring where all the information-bearing
tokens have the same brightness, the same contrast against
the background, is needed.

I do understand that doing experiments like this *looks*
simple but is in practice appallingly hard. This why it is
so important to get feedback from friendly critics *before*
running the experiment.

Personally, I stopped caring about the readability of tiny
fragments the day I wrote my last assembly code program.
I find that the readability issues I struggle with are not at
that scale. But this brings us back to "readable" to whom,
in what sense, for what purpose. You are clearly studying
*something* even if it's not what I mean by "readability".

> --
> You received this message because you are subscribed to the Google Groups "PPIG Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ppig-discuss...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/ppig-discuss/99afa89f-f132-4d74-92d3-07d539b7fbf8%40googlegroups.com.