terminology: criteria, rubrics et al.

[Ilya Goldin]

Happy holidays to all!

I'd like to make sure I'm in good company in how I use assessment
terminology. How do you define the following with regard to peer
assessment? What other terms are related? Are there widely accepted
definitions, or are these contentious or ambiguous? Are some of these
obscure and not used?

Criteria
Rubric
Prompt
Reviewing dimensions

My definitions, as a starting point:

Criteria are abstract, Platonic ideals to which students (ought to)
strive, and against which we hope to assess student performance.

A “rubric” is an operational definition of the criteria of interest.

Reviewing dimensions are synonymous with criteria.

A “prompt” is a user interface element, whether on screen or on paper.
Multiple prompts are possible for a single criterion, e.g., Wooley's
work suggests that rating prompts be paired with commenting prompts,
and there may be many other types of prompts. One prompt asks one
question, and may accept zero, one, or more answers, depending on the
type of question.

What's a good term for a component of a rubric?

[Raquel M. Crespo]

Happy holidays to all!

I mostly agree with Ilya terminology, though I must confess that I hadn't thought of a specific term for some of the concepts previously.

I definitely agree defining criteria as 'abstract' considerations and 'rubrics' as concrete implementations of such criteria.

I may differ a bit regarding 'reviewing dimensions'. I see these as the different viewpoints from which such revision can be performed. For example, content vs. format for a paper or presentation. On the other hand, I have mostly seen the term 'dimension' regarding the assessment process dimensions: individual/colaborative, supporting medium, type of exercise (closed response vs. open question), etc.

Criteria are related to such dimensions, but they intrinsecaly imply a judgement. In the previous example, considering content more important than format and thus assigning it a greater weight, or considering that it should include certain elements. Rubrics would be the formal specification of such criteria, detailing more concretely which score should be assigned to specific typical situations (e.g. if the student does not restore the registries in an assembly program, (s)he would loose 2 points out of 10).

Regarding 'prompt', as a non-native English speaker I have always associated it to the "software" prompt (either unix command line or any other text-based program). :-)

And regarding rubric components, I am sorry but didn't think of a particular term for them.

Just in case it is useful, here at my university we are working on an official rubric for evaluating the final project the students must defend at the end of the grade. It is designed as a matrix where each row specifies an element to evaluate (-> dimension), each column the potential scores, and each matrix cell (-> rubric component?) the particular conditions when such score should be assigned.

Best regards,
Raquel

--
-- Computer-Supported Peer Review in Education
-- website: http://cspred.org
-- discussion: http://groups.google.com/group/cspred
-- email the group: csp...@googlegroups.com
-- unsubscribe: cspred+un...@googlegroups.com
--

[Turadg Aleahmad]

It would be nice if there were some common definitions for these terms. I made a grading tool this last semester and I got stuck on some of these same. I mostly match Ilya and Raquel but I'll chime in.

I think Raquel's description of the table is helpful. The rubric is the whole table. The rows are dimensions. The columns are the range of the dimensions. The rubric defines a function for each dimension that maps the domain (student work) to the range (score). In a table, the cells in a row define this map. By "score" I mean to include qualitative or nominal data, not just ordinal. Is there a better name than "score" for this that captures the qualitative aspect?

So I would expand Ilya's set of terms:

Reviewing dimensions are aspects of the assessable work (word for this?) that are being reviewed or assessed. I would just say dimensions. They are like variables.

Criteria I use somewhat interchangeably with dimension and the conditions for scoring in that dimension.

Evaluation function is the how to turn the work into standard scores. It's somewhat subjective but strives to be objective.

Rubric is the set of dimensions, range and evaluation functions. It's often a table where the cells express the evaluation functions.

Prompt in assessment usually means to me the prompt to the learner but I suppose here it is to the assessor. I'm not familiar with it this way but if it helps the reviewer to generate a range value for a dimension then it would be part of the evaluation function. This would be an example of an evaluation function more complicated than a set of cells.

I've been planning to put my grading tool up on Github but haven't gotten around to it. If anyone would like to be notified when I do, let me know. It's built in Google App Engine so it's easy to run online without setting up any servers. You can try it out at http://grubric.appspot.com/ , bearing in mind it's rough.

Turadg Aleahmad
PhD student, HCII

[Turadg Aleahmad]

I just came across this paper from Open Ed 2010 that is relevant to this thread. It offers a formal ontology of rubrics for the Semantic Web. I'd be interested in any criticisms.

Open Rubrics and The Semantic Web: Open Ed 2010 (presentation)

Repositori institucional: An Ontology for Open Rubric Exchange on the Web (paper)

τ