8.02 Exploring Translations

[Alfie Overacre]

Inthis article, we present a replication of an empirical experiment that evaluates intuitiveness and comprehensibility of keywords relating to different concepts in programming languages, originally conducted by Stefik and Gellenbeck. Novice programmers face many barriers when learning programming. One of these barriers is syntax, which for many languages is not designed based on empirical evidence. The purpose of the experiment was to provide more empirical evidence on the subject, to find out if the results of the original experiment can be replicated and if conducting the experiment in an environment where English is not the native language affects the results. The results of our experiment replicated most of the findings of the original study and provided further evidence that some syntactic choices in many popular programming languages are unintuitive for novice programmers. Our results suggest that the native language of participants who otherwise had good English skills had little effect when compared to the original study. These results may support programming language designers in making evidence-based design decisions and teachers of introductory programming courses in identifying some of the barriers novice programmers face.

Developing software and educating new programmers is very expensive. In addition to educating software engineers at university level, considerable investments have been made to incorporate computer science into basic education in many countries (Department for Education (UK), 2013; Finnish National Agency for Education, 2014). Software engineers are also required to stay up to date on the most recent developments in numerous programming languages, libraries, and frameworks. Professionals who work with program code spend hours on reading it, e.g., debugging and reviewing purposes. Making programming languages easier to learn and comprehend would presumably spare large amounts of both time and money.

The lack of evidence based practice in programming language design is explained to a certain degree by the dearth of empirical evidence in regard to the human aspects of programming languages, which has been documented multiple times, some examples being Stefik et al. (2014), Kaijanaho (2015), and Ko et al. (2015). Empirical evidence is essential in the scientific method and is held to high standards in other fields of research, such as medicine, psychology, chemistry, and physics. For programming language designers to be able to make informed decisions, it is important to gather more evidence that is reproducible and testable.

Many of the commonly used programming languages share a similar base (such as the tradition of C style syntax) but have different variations of syntax and semantics. Which of these variations are beneficial is up for debate, since many decisions in programming language design are based on opinions or personal experience of experts or committees instead of empirical evidence. While there is a case to be made for relying on the insight of experts, there is also evidence that their opinions might not line up with findings in empirical research. For example, Devanbu et al. (2016) found that while programmers have strong beliefs in certain claims about software engineering, these beliefs do not necessary line up with empirical evidence and are primarily based on personal experience. Brown and Altadmri (2017) researched whether expert educators understand the kind of problems beginner programmers face and found that the beliefs of the educators did not match the actual problems that were observed.

To understand and produce program code, every aspiring programmer must first understand the syntax of a programming language, that is, the combinations of characters that form the correct structure for program code. Denny et al. (2011) found that syntax errors were common with students of all levels of ability, even those that were performing better than average. Similarly, in a study by Bosse and Gerosa (2017), syntax errors were the problem most frequently reported by students regarding the difficulties of learning programming. Denny et al. continued their work in 2012 and found that a few types of syntax errors are not only considerably more common among students than others but also take the most time for the students to correct. As with the prevalence of syntax errors, the effect did not depend on the ability level of the students. This finding was supported by Tirronen et al. (2015) who found that in the context of a functional programming course most of student mistakes were reported by only three compiler error messages.

Compilers usually alert programmers about syntax errors with error messages that can lead to frustration if implemented poorly, as discussed by Marceau et al. (2011a, b). They found that syntax error messages are often misinterpreted by students. Some research has been conducted to determine whether improving these error messages affects the rate at which syntax errors are made. An article by Denny et al. (2014) suggests that enhancing syntax error messages does not significantly reduce the rate at which students encounter compilation errors. This finding was replicated by Pettit et al. (2017), though a study by Becker (2016) had the opposite results. Even though it is somewhat inconclusive whether syntactic errors can be reduced with proper error messages, syntax remains a barrier to novices learning programming and more research points to the barrier remaining even with more informative error messages. Even if students start learning with a block-based programming language such as Scratch, where syntactic errors are avoided by allowing the programmer to combine only compatible blocks, sooner or later they will probably become acquainted with general purpose programming languages and syntax as well.

To identify some of the barriers students face, Stefik and Gellenbeck (2011) studied intuitiveness of specific word and symbol choices in programming languages. They asked both programmers and non-programmers to rate words and symbols by how intuitively they describe certain concepts in programming languages. They then compared the intuitiveness rates of the two groups to each other and created a ranking for the choices of each concept to find out which choices were more intuitive than others.

In this work, we replicate the study conducted by Stefik and Gellenbeck (2011) and investigate whether the results from a university in an English speaking country differ from those from a university in a non-English speaking country. We followed the design of the original study and compared our results to theirs to find out if the change of testing environment affected the perceived intuitiveness of the words and symbols. It should be noted that as with the original study, the goal of this experiment was not to find the single best word or symbol for certain concepts, but to get a more general understanding of how the words or symbols relate to the concepts they are supposed to represent.

Overall our results support the findings of Stefik and Gellenbeck (2011). The data suggests that indeed many popular syntactic choices are unintuitive for novice programmers. The native language of the participants did not seem to have much effect on the results, though the participants of our experiment were otherwise proficient in English.

The rest of this article is structured as follows. First, in Sect. 2, we will discuss related work and replication studies. Then, we will describe the design of this study in more detail in Sect. 3. In Sects. 4 and 5, the results of the study are presented and discussed. After that, in Sect. 6, we examine threats to the validity of the study and finally, in Sect. 7, we summarize our findings and conclude.

In this section, we further discuss the relevant background for this study. It is divided into two parts. In Sect. 2.1, we present some of the earlier work that relates to our study. Then, in Sect. 2.2, we move to discuss replication and why it is important in science.

A few years later an article by Stefik and Siebert (2013) described four more empirical studies on syntax and novice programmers. The first two continued and expanded upon the research on intuitiveness of different constructs in programming languages. The first of the two partially replicated the study by Stefik and Gellenbeck (2011) but had a larger scope of concepts. The second study focused on the intuitiveness of larger constructs, such as a loop structure, instead of single words or symbols. The results of these studies support the findings of Stefik and Gellenbeck (2011) and reinforce the conclusions that there are differences in the perceived intuitiveness of different word choices and constructs in programming languages. Also, the perceived intuitiveness varies with the amount of programming experience and across different programming languages.

The two other studies reported in the article by Stefik and Siebert (2013) examined the accuracy at which novice programmers produced program code. The accuracy depended on the language they used and on syntax variations of programming language constructs, such as loops. In these studies, Stefik and Siebert found that both language and syntactic differences affect the accuracy rates. In fact, they found that novices who were using languages that adhere to C style syntax were not significantly more accurate than those using a language with randomized keywords. Their preliminary results also suggest that more intuitive word and symbol choices enhance the accuracy of novice programmers.

The effect of native language on programming has been researched increasingly in recent years. Work by Guo (2018) suggests that non-native English speakers face many obstacles in learning programming, when reading and writing code, relating concepts to their native language and finding suitable instructional materials. One of the barriers they identified was understanding abbreviated keywords or identifiers, which supports our hypothesis that native language affects the perceived intuitiveness of word choices in programming languages. Furthermore, when reading and discussing code with others, beginner programmers are forced to verbalize words in code. This can lead to multiple different verbalizations depending on if the student chooses their native language or English for pronunciation and word order (Hermans et al. 2018). This adds another layer of translation and potential burden. Despite these barriers, however, research by Reestman and Dorn (2019) indicates that native language does not have a significant effect on what compiler errors are the most frequent at least in Java.

3a8082e126