Does Student Writing Improve After Revision?
Mark Stellmack
[The following story cites real data but certain characters' names have been changed for dramatic effect.]
A guy teaches a class. (Let's say the guy's name is "Guy".) Guy wants his students to write a paper. Guy has his students turn in a draft of their writing and Guy is a little disturbed at the generally poor writing quality. Guy figures that one way to improve student writing is to allow students to revise their papers after receiving feedback. "That will make them better writers," says Guy. Guy grades the students' first drafts but he also gives the students feedback intended to improve their writing and he allows the students to turn in a revision that also will be graded. Guy grades the revisions and finds, to his satisfaction, that the scores of 82% of his students improved from the first draft to the revision and only 18% had lower scores or no change in score. Guy says, "It worked! I made the students better writers by giving them feedback on their first drafts and having them turn in a revision!"
At this point, you might be thinking, "Wait...There's a lot of potential for bias here." If you're not thinking that, let me point out that there's a lot of potential for bias here. (Or if you're simply satisfied with Guy's conclusion, stop reading now.)
Guy wants to conclude that he improved students' writing because their scores improved across drafts. How might his conclusion be biased?:
1. Maybe Guy is cheating and he graded the second drafts higher because he wants to show that what he did was effective. Let's assume that this is not the case and that Guy really wants to find out if there truly is an effect.
2. Although Guy might not be knowingly cheating, when the students turn in their revisions for grading, Guy knows that he is grading revisions and he might inadvertently exhibit a bias towards seeing improvement because, hey, the students went to the trouble to revise their papers and all.
3. Guy is the person who gave feedback on the first drafts so the revisions are likely to look better to Guy because the students presumably made most of the changes Guy recommended.
4. Guy may know the students personally and he may take into consideration their personal circumstances when grading (for example, maybe the student struggles with writing for some reason, or maybe the student was sick during the previous week). So maybe Guy gives some students a "break".
If we want to decide if Guy's revise-and-resubmit strategy is really effective, we need to eliminate these potential biases. One way to do so is to have a different set of graders grade the first drafts and revisions from Guy's students. Those graders should be blind to the identities of the authors and blind to whether a particular paper is a first draft or revision.
So there is more to the story: Several blind graders graded the papers from Guy's class. Based on the scores from the blind graders, only 57% of the students showed improvement from first draft to revision (compared to Guy's 82%) and the scores of 43% of the students decreased or were unchanged. Furthermore, on average, the scores assigned to all of the papers by Guy were significantly higher than the scores assigned by the blind graders, suggesting that there was an upward bias in the scores assigned by Guy.
But wait, there's still more: Two years later, to see if the passage of time reduced some of the bias that was evident in the original data, Guy graded some of the same papers again but this time he graded them blind as to the author's identity and as to whether he was grading a first draft or revision. Based on this new set of scores, the same proportion of students showed improvement across drafts but the mean score of the second set of scores was significantly lower than the mean score of the original set of scores. Taken together, these results provide additional evidence that there was an upward bias in the original scores, but they also indicate that students' writing really did improve...in the eyes of the grader that gave feedback on the first drafts.
These results are consistent with our observations that grading writing exhibits a great deal of subjectivity and the results are what we would expect in light of such subjectivity. One instructor might provide instruction and feedback to students that improves students' writing as far as that instructor is concerned, but other readers may not see the same degree of improvement. This could account for the perception some instructors have that student writing is, overall, of disappointingly low quality when students enter their class. It could also explain why instructors may be satisfied that they have succeeded in improving student writing over the duration of their class. Those students move on to the next class and the cycle starts all over again.
What to do with this information? One possibility is to drop the notion that there is a way to improve student writing such that the improvement will be perceived to carry over from one class to the next, that is, that students can be made "better writers" in absolute, objective terms. It may be the case that student writing quality improves over time, but evidence of that is hard to come by. Even if student writing quality is shown to improve over time, without the proper controls, it may simply have to do with maturation of the student and not with anything the instructor has done.
The data do show that student writing improves substantially on particular assignments in the eyes of the grader when the grader gives feedback on an early draft. That essentially describes the peer-review process. When a paper is submitted for publication, the author must satisfy particular reviewers before the paper is published. So teaching students to respond to a grader/reviewer is in itself potentially worthwhile and writing activities should be designed to take advantage of that aspect of the writing/revision process.
[Note that the fictitious character "Guy" represents an amalgamation of several graders. Details of the research described above can be found in Stellmack, M. A., Keenan, N. K., Sandidge, R. R., Sippl, A. L., & Konheim-Kalkstein, Y. L. (2012). Review, revise, and resubmit: The effects of self-critique, peer review, and instructor feedback on student writing, Teaching of Psychology, 39, 235-244. Also see the materials at http://www.psych.umn.edu/psylabs/acoustic/rubrics.htm]