Game review scores.

[Maddy Myers]

Polygon's shifting review scores on their Sim City review have kicked off some pretty interesting (and, at times, uh, infuriated [see: Ian Bogost]) discussions about whether or not game reviews should have scores, or change their scores. For one thing, Metacritic will only accept the first score that a publication submits, in order to avoid claims that a publication might change their score later to appease an angry game developer (or, whatever, insert your own conspiracy theories here).

My opinion: I think of a game review as a snapshot in time. Changes to a game warrant a new article or subsequent post about those changes, perhaps even a new review.

This is Polygon's review policy page: http://www.polygon.com/pages/about-reviews

Aside from the score change topic, I'm also pretty interested in the idea of talking about whether a game is "functional." I think literal functionality is a huge part of talking about a game; if we can't play it, why would we recommend it? But I also think it's sad when that's the ONLY thing we talk about with regard to a game.

For example, if a game is not only functional but beautifully designed, and also -- let's say -- terrifyingly racist, then how do we score it? I'd give it a zero (or, ideally, I wouldn't have to "score" that game -- I don't really like scores, I'd rather DESCRIBE the game and let readers decide whether they want to engage with it or not). But "racism" isn't listed in Polygon's rubric for deciding whether a game gets a zero ... although ... I think it might secretly be listed in all of their minds, idk.

This is sort of multiple topics, but ... whatever. What do you all think about:

1.) Review score numbers: why have them? Why change them? Should we have them at all?
2.) What gives a game a low score? What makes a game "broken" -- just functionality, or content too, or ... blah blah blah you get the idea :)

[Sylvain L.]

Simply put, when I read a review, I don't want to know if the game is functional: first and foremost I want to know what the game means. Anyway, most of the time, while describing our experience of the game, what it meant to us, its functionality, or its degree of "fun", will be implicit. I can't think of any example, but a deliberately broken and unplayable game could be more meaningful than the most slick and extremely polished gameplay possible. The former would be more valuable than the latter.

And I don't know how we can measure "meaning" (except in extreme cases: a terrifyingly racist game is surely worth a big nothing), so review scores, for me, can only be harmful. Reviews are not meant to measure anything; they're closer to a subjective description. How can I rate my experience? Really, I don't know.

[Cameron Kunzelman]

I don't know if I am willing to go that far, Sylvain. If a game straight-up doesn't work, then it is sort of hard for me to get at the "meaning" of the film.

[Cameron Kunzelman]

Not a film, a game! A game, I say!

[Javy]

This might sound a bit hypocritical since we use scores in our reviews over at Dusty Cartridge and the one we're about to launch, but I don't care for them and would rather write a review without them. However, I do understand how those might appeal to a lot of readers, so I understand why they're there even if I think they do more harm than good. I like the compromise of Koktau's review: the "Should You Play This?" system. There's no numerical score. It's a tab filled with information that, more often than not, makes me want to read on even when I'm pressed for time.

As far as what counts for a low score for a game, if we're talking about below average, I think it has to be broken to a degree. The only really, really negative review I've ever written was for a game that bordered on being broken--multiple forced restarts in boss fights, frame rate issues, and a horrific control scheme, etc.. I spent a good time after I wrote the review justifying the score to myself on the basis that my experience matched the "3" on DC's rubric. I think essentially once you're below the average rating (6-7, right?) on a 1-10 scale, the game's got to have numerous issues that not only disrupt its enjoyability factor but also affect how well it functions.  A bland hack & slash that's still competent functionally is probably going to get an average score from me barring any glaring technical issues because I know it will probably please fans of the genre. As a writer, I see a score--be it numerical or alphabetical--as punctuation. It's meant to emphasize my judgement, not define it. But many readers go to reviews and take nothing away from it other than the score, which is my main issue with scoring reviews being a common place thing--they give the reader that option.

[Nathan Altice]

Most of the time, review scores function as shorthand for the tl;dr set. Scroll through the text to get the quick assessment - better yet, put the score at the top so I don't have to read at all. Besides that, the granularity approaches absurdity as five stars becomes ten points becomes a 'twenty-point scale' becomes comparing a 6.7 with a 6.8 game. What does it mean to be 0.1 points better as a game? The metrics are then propped up with text descriptions that the reviews editors apparently consult when writing a review. Nonsense.

I share Sylvain's desire for meaning, but most reviews are written to inform buying decisions. Is this game worth my $60 or not? Readers build relationships with particular critics (or sites) who share their tastes and tend to trust their scoring decisions. In the age of YouTube and video-oriented gaming sites, the consumer-focused review feels antiquated. Example: I like Giant Bomb. They are striving to push games writing forward, but they are talented entertainers and smart writers. I don't always share their tastes, but they take a very workman-like approach to reviews. They've discussed multiple times in their podcast that their job is to make their audience informed buyers, period. They erect no pretense of criticism. But I find that they've been in the industry for so long that they have a lot of insight about what's good and bad. Further, they feature Quick Looks on the site, which are twenty minutes to an hour of gameplay footage and commentary on new releases. They play a nice range of titles, from Proteus to Super Meat Boy to Call of Duty. For the purposes of buying decisions, I find these videos far more valuable than any written review. I can usually tell within 5 minutes whether I'll enjoy playing a game or not. YouTube Let's Plays serve the same purpose.

Giant Bomb does written reviews too, but I rarely read them. They adhere to a five-star system, no halves or decimals. It's OK, but everything ends up averaging in the 3-4 star range. The problem with scores is that they aim at some kind of objective comparison across time, genre, and tastes. It lets us say x game is better than y game. Sites like Metacritic and Game Rankings exacerbate this problem, allowing us to make quantifiable statements about the 'best game of all time,' divorced from all contexts. How do we accurately compare Mega Man 2 with Assassin's Creed II? Um, scores I guess. It's easier than grappling with what those games did and do, how they've changed, how they work, what they represent, what they teach us about their platforms, and so on.

Polygon is a weird case because they seem to have an identity crisis. They assembled the 'all-stars' of the top videogame sites, hooked into the Verge's impressive CMS, and promised to revolutionize game sites. Their longform features are excellent - thoughtful, well-reported, well-written - but they exist alongside these run of the mill, consumer-centric reviews and vapid news stories that have nothing to do with journalism. I think Polygon missed an opportunity to raise the bar with games criticism. Doing some meaningless review score shift doesn't help (and likely confuses their consumer audience). I also think their site is over-designed and difficult to browse, but that's a matter of personal taste.

The two instances where I find review scores tolerable are a) when they actually mean something - by that I mean a site will actually give a game a zero, average is truly in the middle, and top scores are rare - or b) the whole score system is subverted as a means to get the reader to read, discuss, and engage. Action Button does this to great effect. They gave both God of War II and Bioshock abysmally low scores compared to the rest of the gaming press (a ZERO for the former) and attracted a lot of conversation due to those scores. As counterpoint, they gave God Hand a perfect score. It made the reviews engaging b/c I wanted to see how the writers argued their position. And the reviews changed my opinions on the former two games (and made me buy the latter). That's what good criticism does.

Oh, and Action Button will run multiple reviews of the same game by different writers, sometimes years apart, with wildly different scores. 

So, the FINAL VERDICT on review scores: 9.8 out of 72.3 game-words-inc. points.

[John Brindle]

Bizarrely enough, I'm actually fine with scores. I think this is probably because I hope others will always recognise that they are slightly arbitrary and not a substitute for the review and also a tool used deliberately by a person, not some ultimate mathematical measure of quality. I've always read scores that way and hope others would too. I kind of like the process of looking at a game and thinking, hmm, how would I score it? 78%? No, not quite...89%...no, too high..86%, yes, that feels right. I really do think it is, or should be, a matter of 'feeling right' - and the quality metric can only ever be the reviewers.

Of the games I have played recently, I would give Catherine 89%, Mirror's Edge 69%, Call of Duty: Black Ops 2 54%, Dishonored 95%, and The Binding of Isaac 75%. Cart Life would get 92% due to show-stopping bugs and Howling Dogs would get 99% because it's free so there are very few reason not to play it. The -1% indicates that it exists in a world in which you might not own a computer to play it on or might be killed/put in hospital before you get the chance.  

If I was confronted with a super racist game, and I was ever in the position of writing a review and having to come up with a score, I would probably give it an incredibly low mark out of principle. If it was also a fun or well-designed game, I would state this clearly: "I'm giving this a low mark because you shouldn't buy it and fuck everyone who made it."

[Sylvain L.]

Yeah, well I have a tendency to exaggerate a bit...

Let's nuance: 99.9% of games need to be minimally functional, or else we won't even play them, so, yeah, it would be hard to find any meaning in a game we can't play. And in my example of a "deliberately broken and unplayable" game, the gameplay is not broken, it is exactly how it should be. But then it would be a conceptual game, that we don't have to play to appreciate (Robert Yang wrote about this recently, no?) And anyway, we cannot really dissociate "functionality" and "meaning", because meaning comes out of the functionality.

I still think reviews should focus on meaning first. A score can be useful to measure technical aspects, but I like to think games are more than technical products.

[Ethan Gach]

Personally I would only consider functionality in so far as it makes it difficult for the game to achieve other things.

 

C & L: Dog Days has functionality issues in the conventional sense, but whether or not they prevent the game from being what it tries to be (an open question) is very much up for debate.

 

Some people knocked Catherine because of difficulties they had while trying to climb, often blaming it on the controls and the game incorrectly playing out certain inputs (climbing in the wrong direction/going across rather than up/down, etc.)

 

But it's by no means prohibitive, and depends more on the players skill and ability to adapt to the game's traversal scheme.

 

Imagine if a critic lauded a book for a rich narrative and experimental prose, but then knocked it for being a "difficult read," or for having sentences constructed in such a way that their meaning won't be obviously clear to the reader at first glance.

[Sparky]

When do I give games a low score? When I hate them. When I hate playing them. Functionality plays into this a little bit: I am definitely more likely to hate a game if it erases my progress or has numerous bugs, or makes me use the controller in an uncomfortable way (Mod Nation Racers). But ultimately the score should give a brief idea of my impression of my time with the game. It's not their to give developers or publishers a cookie, it's there to convey my assessment of how worthwhile I think it is to play it, modified as necessary for any issues that will obviously interfere with play later on.

Of course, this requires some familiarity with ME, and this leads me to something I hate about the way scores are aggregated for video games. Go take a gander at any game on Metacritic really quickly, and then go look at any film. Notice how the critic name is prominent for films, but absent for games. Metacritic isn't the only offender in this regard. When Kotaku and more recently Destructoid aggregate scores, they identify reviews by website, not author. This despite the fact that websites often have diverse voices, and the first thing the readers themselves often do is go to check who wrote the review! Despite the fact that gaming sites routinely say things like "A review is one critic's opinion", they disappear the critics from their own reporting about scores.

My problem with the Polygon score is that it's changing in the wrong direction. Everyone knew the launch would be a disaster. EA's major online launches are always disasters, Diablo III's launch was a disaster, and we know by now how these things are going to go. Polygon should have knocked off points for a bad launch prior to publication, then given them back if EA actually managed this gracefully. Moreso than any other kind of reporting, reviews are meant to directly serve and advise the readership. Pretending that everything would turn out fine at launch did Polygon's readers a disservice.

[Alan Williamson]

Game review scores only really fall into a few categories: OMG GOTY!!11 (90+), good (75-89), meh (60-74), avoid (<60). Anything less than 60 is just an increasing degree of insult, isn't it really?

I know I normally come at these things as an idealist who doesn't care about hits and advertising revenue, but is there a point in appeasing people who can't be bothered reading your reviews and want a score? If you don't want to read my thoughts, you can fuck off somewhere else. If you want numbers, go to Metacritic.

Polygon's sliding review scale is daft. Are they going to give Sim City 9.5 again when all the servers work? Zero in ten years when they turn the servers off? It's admirable to acknowledge that they sometimes get scores wrong, but this doesn't seem the right way of going about it.

Also, don't forget that many reviewers use a score as a crutch to make up for shortfalls in their critique: without a score, you wouldn't really know what they thought.

[Bobby Hunter]

Even though pretty much all of us here realize the difficulty of translating emotions, experiences, criticisms--words basically--into a numerical score, I think for a lot of consumers the score works as short hand for the content of the review. This is anecdotal, but I have friends who will just read the score at the bottom of a review or just check Metacritic and base their decisions on that.

I also think there's a distinction to be made between writing a review (for consumers) and criticism, but that's for another time. We're kinda stuck with scores, well, if you happen to write for an outlet that uses them. It is ridiculous to explain the difference between a 9.8 and a 9.7, but I dislike the trend I see of a 7 being considered an average score. I mean if a site uses a 10 point scale, then a 5 should be considered average. What makes a thoroughly average game though? That's subjective and that's where the content of the review comes into play.

Just my thoughts on the matter. I'm loving all the other responses so far!

[Tony Perriello]

To step back from focusing on Polygon for a moment, I think any attempt to try to utilize all of the 10 point score scale beyond what Alan mentioned is admirable, but it will ultimately fall flat (at least in the short term). As mentioned, it seems like most people have been conditioned to years of the "7-10: Worth trying, 0-7: Don't bother it's shit" scale, and it's a bit hard to see beyond that right now. 

That's not to say the view of the 0-10 scale won't change and become more accommodating to using the whole scale in the future, but I don't know. I think it would be healthy to maybe step away from the scale entirely for a bit, and revisit it (if the desire to do so arises) once there has been some distance from it.

Probably not a very realistic scenario, but just my two cents :)

[Johannes Köller]

I suppose the most interesting question to come out of all this for me is whether we should consider such external factors in reviews and scores or keep them focused on the content and meaning of the game while ostracizing business practices in editorials. Perhaps "external" is not the right word though, of course the experience of a game consists in part of the outside experience of buying, installing and playing it (and patching it, entering keys, hoping servers are online etc) but in many cases DRM amounts to little more than a mild annoyance, and reviews then tend to mention it briefly before basically going on to say "Okay, but this is what the game is like once you actually get to playing it." Obviously this doesn't work for Sim City or other such cases in which it's simply not possible to get past the immediate technical annoyances, but I could see two different philosophies about whether this means points should be docked or it should still get a score somewhat reflective of the rare, ideal play experience (and a massive warning label saying that this is not the experience you will have, plus editorials on DRM etc. etc.).

In a sense they've almost made it worse by updating their review, because if they do mean for their scores to reflect such issues then the original 9.5 quite clearly spells "We totally didn't see this coming." Local print magazines have done this kind of revisiting for a while now, and one way they have gone about addressing games that they only got to review before launch and possible server issues is to publish the written review sans score, which is added after they see how it performs out in the wild.

I'm almost glad I don't have to use scores right now.

[Sylvain L.]

Like Johannes says, this whole Sim City debacle and Polygon's shifting score show how videogames are reliant on technological aspects in a way that is more fundamental than what we can see in cinema for example, the other important technological-industrial medium. If I stumble upon the worst projectionist ever in my local theater, I will not take this context into account in my review of the film (I won't do it consciously at least), but for videogames, the technology is an integral part of the experience (the servers in Sim City support some features that would not be possible offline, and the game is just not there without them). And what about bugs? In a movie, the equivalent of a bug would be a mike at the edge of the frame, or the reflection of a crew member in a mirror, etc. (Or for a book, it would be a bad printing problem). But these kind of bugs doesn't prevent me from watching the movie, or even appreciate it. They're mostly superficial. I didn't get the disappearing save bug in The Walking Dead, but I guess I would not have like the game as much if all my choices had vanished just before the final episode.

We could say these bugs concern the game as a product, and the ideal experience, as Johannes says, is the artistic part. If the product is not polished enough, it could hinder my experience of the game in a way that is more fundamental than it is possible in any other expressive medium. So, if your reviews are closer to a consumer's guide, you should probably lower your score when there's a big problem like the Sim City servers, because as a product Sim City is just broken. But if you're more interested in meaning, then maybe you should try to ignore these bugs as much as you can. I'm not sure it's possible to dissociate these two aspects so easily though.

[Nathan Altice]

Sylvain, your point has me thinking about how I'd score something like Sonic Adventure. It's broken in ways that are actively antagonistic to play, but I love it precisely because of those traits. At a technical level, it's a 2.0 (Polygon System [TM]), but as weird pop art it's a 14.3.

And, importantly, this is the perspective of someone playing the game in 2013 having never owned a Dreamcast in the early 2000s.