Accounting for Bias in Reviews

[Javy]

I read this review today and while I thought the review itself was pretty good (except for some of the drawn out gags), I was mostly curious about the "bias adjusted" score factor (bottom of the page). To me, that seems to be a silly way a level a score. How do you even turn something like that into a percentage. I guess you could pose the same question about scores in general when it comes to reviewing games, though.

I recently reviewed Anodyne and started the review by saying that I wasn't a huge fan of the game that Anodyne pays homage to: The Legend of Zelda: A Link to the Past. I wanted to be straightforward with my reader about that fact so it would help them further determine whether or not the game was for them. But I'm not convinced that's the way to go about it since that tactic may pull the reader away from the critique of the game, which should ideally be the focus, and onto me, the reviewer.

I was curious about how you all account for genre/series bias in reviews, if you do at all.

[Nate Andrews]

Uh. Certainly not as arbitrarily as that.

[Bryant Francis]

I'm of the opinion that reviewers should just do the best to describe their experience with the game, period. If that means putting up a warning that "they usually don't play strategy games" or something, then that's fair. It's part of the experience, and it helps readers know if they want to add the reviewer to the list of critics they trust.

Referencing them as numbers though? Eh. I'm kind of "meh" about numbers in general, (Jim Sterling gave a great argument for why they're valid, but I just feel like I'm reviewing a car whenever I use them), so maybe the "adjusted score" is a bit far.

[Greg Brown]

Yeah, I'd agree with Bryant; disclosing whether you enjoy or don't enjoy some genres, along with some of the reasons, is probably the best way to go. Inevitably there are going to cross-genre games and other weird edge cases that might lead you to go, "Hey, I don't usually like strategy games, but I really liked this one because of x, y, z." Trying to address that through some *beep beep boop boop* bias calculation seems to be asking for trouble.

[Javy]

Yeah, not too crazy about numbers either, especially since they give readers an excuse to skip the prose and just jump ahead to the score and be done with it. I like Kotaku's simple review rubric because of how personal and straightforward it is: "Should you play this game?" And the answer is usually backed by substantial support as well.

On Thursday, February 21, 2013 5:40:45 PM UTC-5, Bryant Francis wrote:

[Cameron Kunzelman]

I think the bias calculation is at least interesting for making the effort to explain differences in numeric scores in the numbers system. I'm down with the non-score version of reviews, but when a website is committed to that system already, you have to work with the tools at hand. 

I would rather have that than "10/10 so good you must love it"

[Bryant Francis]

I'm always surprised by how much flack they get for it in the comments--it's literally easier to figure out than a review score, and the reviews they write to go alongside it (instead of justifying numbers) are actually really worth a read! Patricia Hernandez' Fire Emblem review feels like experiencing the joy of watching the characters interact, and the different stories that come out of the Kotaku writers feel more authentic than any number calculation ever is.

And yet, without fail, I keep stumbling on people going "this system is stupid." I mean, haters gonna hate, but sheesh. =(

[Amanda Lange]

I would rather not put scores on my reviews, but since we do it at the site, I just try to state my bias as clearly as possible in the text and back the review up with the text as much as I can. I think it's fair to state biases. If you develop enough of a relationship with your readership, they'll start to understand your biases, hopefully.

(When you said Anodyne was a Link to the Past homage I immediately put it on my to-buy list. I suppose that wasn't the intended effect. :) But it's good to know what games a person might like that are like the game that you played even if neither game was your cuppa.)

[Javy]

Yeah, we assign scores at Dusty Cartridge as well.

And if you like A Link to the Past, then chances are Anodyne is for you. It's basically a love letter to that game.

[Javy]

I think a lot of that is that Kotaku readers are a particularly feisty bunch, to put it lightly.

[Nick Capozzoli]

I'm just naturally unbiased, so thankfully, it's not something that I need to worry about :P

 

The "Bias Adjusted" score is an peculiar feature. I actually find it pretty interesting, though I'm sure in practice any impression of objectivity it provides is wholly illusory. It seems like a fun quirk that helps that particular site stand out, at the very least.

 

The latter is probably the only thing that really matters when it comes to scores, anyway. After all, what are they for, if not for being fun to analyze and argue about? I've never found that a review's core really helped to improve my understanding of a game beyond what text can do.

 

I agree with others who suggest being up front with ones biases, with the caveat that I try to do a lot of consideration of my own biases before I set out to review a game. They're not always legitimate...sometimes putting them down on paper shows them to be shallow or poorly-conceived, and you can work through them. On some level, the biases will always be there - best to acknowledge your feelings and make doubly sure that your audience is on the same page. They might get huffy if they don't like your opinion, but it's better to give 'em all the information to work with than to leave them with half of the story.

[Ethan Gach]

Linking this to the article Maddy noted about Gies' Dead Space 3 review, Drew Dixon recently wrote at Bitcreature about the double-standard that's often apparent in many indie vs. AAA reviews.

 

Smaller games get held to the same standard of polish as most high end, high budget games, but those AAA titles are almost never held to the same standard of creativity/saying something/being interesting that's expected of indies. He uses Miasmata as a case study: http://www.bitcreature.com/editorials/dissonant-reviews-miasmata/

 

The end result is that most AAA games have inflated scores by comparison, and Gies' is someone who is well known for giving more weight to the sophistication of a game's systems and mechanics than to its aesthetic presentation, narrative performance, or emotive cache.

[John Brindle]

Final verdict

Score: 8.1/10

Bias adjusted: 7.2/10

Prozac adjusted: 6.5/10

Had a cold adjusted: 8.9/10

Age adjusted: 5.3/10

Expectation of adjustment adjusted: 7.4/10

etc

[Nick Capozzoli]

Bias of "Bias Adjustment" adjusted -> 8.1/10

[Dan Cox]

By using numbers or another easily convertible grading system we create a hierarchic system of values. Games at or above this number are "good"; others, those not at the threshold, are therefore "bad". Without meaning to, it often embeds a binary and, at least in reviewing, the breaking point of the 7/10 grade most commonly.

Which, of course, creates its own problems since there are now expectations affirmed by past use of the scale. My favorite example of which was someone contrasting a game made in 1997 with one from 2007, both of which had the same Metacritic "score" and therefore, to his eyes, must be of equal quality.

[Alan Williamson]

I stopped assigning scores and never looked back. I think 'biased' (and I think what we're really talking about here is 'overtly subjective') reviews are really, really important. If I review a Sonic the Hedgehog game, and you know my background, you'll know I might be more harsh towards it than another reviewer. The bias isn't the problem: your honestly in conveying it is.

On the subject of review scores, Craig wrote a great analysis of them - with some lovely graphs - over at Split Screen.