Ophelea, "Out There"
« I Enjoyed 2007From Big Countries Come Fragile Dragons »

The "Reviews are Broken" Argument


Permalink 16:52:54, by Ophelea
Categories: Press, Consumers

The "Reviews are Broken" Argument

Convergence is happening again/still/ad infinitum. Psychochild pointed me to a post on Slashdot last week about game reviews being broken; this post is based upon an editorial on Kotaku which discusses game reviews being broken primarily from the point of scoring and worth the money.

Gee, really? (Self-aggrandizement off.)

Convergence. While all of this yapping was occurring, I was noticing a few things myself:

After reviewing a game I adored and was particularly proud of the review, I sent the link to a friend. His response? (verbatim)

"i had a hard time getting into it. i dunno why. :( i kinda think being paid for reviews and being rushed on them is making gaming less enjoyable for me. i had to can the review for it since it wouldn't have been timely." What's timely? "2 weeks after release, though 1 is preferred. we have to rush with the reviews."
(This site buys all of their titles retail; they receive nothing pre-launch.)


Three PR companies were pushing particularly hard for PC previews. These were for games of medium-sized publishers and it is unlike any of the people at the companies to push like this. So I asked why?

PRRep: both IGN and GamesRadar have both said flat out they don't have the manpower to cover [said vendor's] games. this is a growing trend with PC games, by the way with titles that are not AAA or multiplatform releases. it's insanity
OpheleaLC: but there are so many of "me"
PRRep: unfortunately, no
OpheleaLC: what happened to the b-list sites? and honestly, my problem? they spend their advertising money on ign and gamesradar so that we struggle....
PRRep: a lot of the b-listers don't cover PC anymore
OpheleaLC: wow, that's insane and that explains why I have so much PC
PRRep: yep!

Nice, again. And not the first time I've heard this concern about PC titles. Not even the second or third…

GamersInfo.net makes it a point to cover children's titles - in homes with children. During the holiday season we are overwhelmed. But, something common with children's titles is you must ask for them. Why? Direct from the mouth of PR:

"Well, we don't really want a hard-core gaming site to give a good kid's game a bad review score, simply because they don't "get" the audience, KWIM? It likely won't have all the elements that hard-core gamers want, and therefore a perfectly great family game will get a bad score ;)"

Follow up:

Sooooo...if that's the case then why does this happen?

That same review I was so proud of, that I worked hard on. I spent over 40 hours on that game before reviewing it. I played with my children. I wrote this approximately 30 days after the game released (and 18 days after I started playing): MySims. GameSpot released their's the day of the title's release. This is the same title that was too complex for the paid writer to complete in 1-2 weeks. (To the GameSpot writer, I'm sure you're good at your job, but children's titles are not your forte.)

EA complimented me on my writing. They thanked me for an in-depth review. They offered to send more EA children's titles. Then they asked how many unique readers visit my site each month; I have yet to receive the product. I do not hold this against EA; we do receive the occasional piece of product from them and some companies have limited product so their decisions are made by numbers.

Quantity over quality.

This blog is unique for me. I've singled out companies, writers, and I'll continue to do so. This skates along the line that borders my ethical boundaries. But, this site was started three years ago on the premise that reviews are broken.

Last week Kotaku agreed. Last week. Can you hear me rage in frustration?

The question on everyone's mind seems to be where to lay the blame? I think this is the wrong question. The system developed over time and is an aggregate of a lot of other systems coalescing. The question should be:

Who is going to fix it?

The answer, unfortunately, is you, the reader. Why do I say unfortunately? Because of all the pieces of the puzzle, all the components that make this bitter soup, the reader is the least likely to contribute, complain, change or even notice that there is a problem.

So, let's start at the beginning: defining the problem.

"Video game reviews today do not serve the public in a meaningful fashion."

That's an incredibly squishy statement. It's true, it defines what is wrong. But, it can be interpreted 90 ways by 90 people. What is our end goal? What do we want the statement to be?

"The purpose of a video game review is to inform the reader of the basic facts of a title (the story, features, unique gameplay elements, bugs, etc.) and to offer the opinion of the writer so the reader may make an informed decision before purchase."

First, you have to decide if you agree with this statement. I always write too much so I'm going to write this entry as if you do.

That being said, the words "opinion" and "fun" by definition are subjective. If there has ever been an objective review written I've never read it. It is wholly impossible when the very nature of "fun" is subjective; difficulty is subjective; pleasing, solid, attractive, ugly, shocking, enthralling, immersive - all of the words used to describe games today are all subjective. Game reviews are subjective. I don't care who you are or what you believe. You can't argue (with me or the wall) about this statement.

Want to argue with me? Then why did GameSpot need to change the way they score their reviews in June? They are correct in their statement that art can no longer be a benchmark in their scoring system because scores are used to compare. Can you compare Bioshock to BrainAge? And would the score be fair?

Ok, reader, time to stand up to the plate. Have you ever asked/commented/read/questioned the writer about their skills/likes/dislikes/history when they handed down their judgment from on high with that almighty number (or star or thumb up/down)?

(I'll come back to kid's games frequently because they're easy to pick on.)

Last week, I was posting a review for a title that I had spent months working with. Why? Well, for one thing I live in the United States and folks here just don't get it. It's an edutainment title and those are usually poorly done, poorly reviewed and poorly marketed. And this title? It was a first attempt at its subject - spelling.

I took time to research how we learn to spell, how are taught to spell, how we should be taught to spell, and I took the title to teachers. Yes, it was a bit of work but honestly? No more than 15 hours over those months. I just couldn't do it all at once (school wasn't in). When looking for the landing page for the publisher I found this at IGN. I've even complained about editors assigning the wrong people to the wrong games and then not following up on their work before.

This is an uneducated, spiteful writer. What's more, based upon the review it is obvious to me he didn't play the game beyond the first five minutes. But the opinion has been rendered. The number assigned. No questions asked. Not one of the reader comments asked, "why the bitterness?" Not one of the reader comments took exception with the statement, "This is how I imagine spelling bees for deaf children." Well, yes it is. And as such it is a great teaching tool.

But the score. Why a score? Why not just the proclamation that the game is bad for reasons a, b, c and d?

We are lazy readers.

This is not a statement of insult; it is a statement of fact. Studies have shown that when reading text upon a screen, it will be read in the shape of an F. In other words, we'll read the meaty parts at the top, and then skim the bottom. Don't think that site designers don't know this and don't take full advantage of it. Why do you think ad placement is at the top?

There are entire classes on writing for the web. The first day sounds like this: the average internet reader has a short attention span, will only read in small chunks, uses a lower than average vocabulary, wants their facts presented to them immediately and easily…

Insulted yet?

Scores make everything easy - for the writer, the reader and the publisher. Let's look back at GameSpot's review changes. Medals. Reviews now have icons to make it easier for the reader to determine the quality of a game. You don't even have to read the writing. Just look at the pictures.

What do those numbers mean for the publisher? It takes up less room on the box and you can put a lot more of them. To be fair, pull-quotes (positive comments from a review) are used but it is a cyclical process. Publishers know that to get the best coverage they need to woo the best sites; to woo the best sites you give them exclusives, box quotes, a lot of lead time, etc. The sites know that to continue to hold the publishers' attention they don't have to give them great scores, but they can't consistently post negative reviews either. So, there's an unconscious bit of adjustment going on.

And having worked in marketing on both ends, I know for a fact that PR doesn't buy reviews nor do sites (those I worked at) give good reviews in exchange for ads. But, if you want a lot of eyes to see your advertising, go to the big site - even if they didn't cover your title because they don't have time for PC. All the while you're wooing them on other platforms.

Sounds insidious and evil, but it's not. It's business. And will the reader go elsewhere? No.

Scores are easy to read. Big sites have shiny giveaways, pre-release information (on the big titles) and shiny user tools.

The internet is all about EASY. But easy won't pay the bills; it doesn't cover effort or talent or time.

Big and little sites are akin to big and little titles. If you missed it last week, Gamecock's CEO made a statement about pre-ordering the smaller titles. Big titles don't need pre-orders; they're in stores no matter what. But "If not enough people pre-order a [small] game, [retailers] will lower their order to where poor sales become a self-fulfilled prophecy..."

Reviews are the same: only the covered games are big and only the big games are covered. But the survival of the sites that will cover the smaller titles depends upon the reader. Everyone covers the big games; so much so that they can't cover the smaller releases. (Sorry, this still boggles me.) But if the smaller sites who take the time and effort to ensure that every title gets coverage can't cover costs, they have two choices: drop the games that aren't cost-effective or shut down.

I'm running out of steam. I could go on and on. Indies are learning. This site was refused a link on a press coverage page because we don't score. Both Metacritic AND GameRankings aggregate scores now. We need two? GameFaqs only take reviews that contain a score. Four years ago this was not the case.

I can't change this - not GI.n or any other non-rating site/pro-subjective site. Why? We simply don't matter enough to the publishers. And we won't until the readers say "enough, we need to know enough to form an opinion, not to be given one".

You're going to have to start searching for your PC reviews or, more than likely, simply not know the games exist.

Rushed reviews by writers who aren't having fun are already the norm.

What is next?

How much is enough?


Comment from: Sylvene [Member] Email
This has given me much food for thought in more ways than one. Where do I start? *chuckle*

I would have to agree that many sites don't match reviewers to games very well. When the review opens with the reviewer wondering what he had done wrong to piss his editor off that he was assigned "My Pretty Ponies" to review, you can tell that you're not going to read a fair review.

2007-11-05 @ 17:37
Comment from: SeanMike [Member] Email
Wow. You wrote so much more about this than I did - but I kind of expected that. :-)
2007-11-06 @ 09:27
Comment from: J [Visitor] Email
I'm constantly aghast at how lazy reviewers treat my favorite hobby. In fact I suspect that one must be grossly overweight and bald to even get a job at Gamespot. You know these guys want to be pro game designers, but they can't even design a decent way to review games.

I'm hoping some reviewer will make a break through and invent a new rating system. A simple number rating isn't enough. Is Super Mario Bros. a 90? Was it a 90 in 1984? Does it really compare to Mario Galaxy today? Maybe the original is a 90 to new gamers but a 50 to hard core gamers. Maybe Brain Age is a 100 for casual players and a 30 for twitch gamers. Does Pong deserve a low score since Rock Star Table Tennis trumps it in every way, or does it get a nifty bump because it is so historically relevant? Can a $10 game get a perfect rating even if it only lasts an hour? Does this perfect rating even mean anything next to Halo 3's high scores?

The best indication of broken-reviewness is here: http://www.metacritic.com/games/xbox360/ The top 5 rated games are FPS! Not very relevant for a sports or adventure gamer is it?
2007-11-07 @ 12:03
Comment from: Harm [Visitor] Email
This is a great read Ophelea! I completely agree with the original concept of broken reviews and commend you on what you have pointed out as well.

I personally find their reviews mildly useful. I ignore the numbers and actually look at the writing more than anything. I admit that I occasionally get skewed by the numbers as I'll be less likely to read the body of the review based on a poor number, but I make an effort to avoid that problem.

I place some of the blame on marketing...as you have pointed out...a number looks better on a box than a lengthy quote.
2007-11-07 @ 12:50
Comment from: dforrester [Visitor] Email
IGN reviews have always scraped the bottom of the barrel; I should know, I used to write for them. One particularly amusing part of the IGN review of Spelling Challenges and More! is this:

Yes, before asking the player to spell "present," they flash the word "present" on the screen, and then give the definition. How did nobody stop and realize how ridiculous that is? Of course I can spell that word; you just showed it to me! Oh man, you only flashed it for a second? How will my meager brain remember six whole letters?!

It seems his meager brain doesn't possess the ability to count to seven correctly, never mind remember the word.
2007-11-07 @ 15:12
Comment from: Ophelea [Member]
Please don't take this as an attack on GameSpot's rating system. I think the way they rate has improved since June, I just don't like the visuals.

I don't care for ANY rating system because - as I pointed out - abuses such as the case on IGN occur so easily.

Or as J mentioned, how exactly can the best 5 XBox titles be FPS titles? And how is that relevant to someone (me) who doesn't play FPS titles?
2007-11-07 @ 16:19
Comment from: Ye Ed [Visitor] Email
This reviewer's basic prob is logorrhea. Stop whining and start editing.
2007-11-08 @ 06:47
Comment from: Aaron James [Visitor] Email
This is a quite incisive rant I have to say. I used to game quite obsessively, unfortunately work has interrupted that. I find myself these days picking up a title, sadly, by the 'buzz' created by the big name sites. I look at the score, if it's 7 or more, I read the first couple paragraphs.

If I like the sound of it, I go on.

I think I'm part of the problem then :(

At least having read this though, I will be a little more discerning in my consumption and valuation of the 'big site' reviews.

And maybe that's what needs to happen?
2007-11-09 @ 04:31
Comment from: Ophelea [Member]

Thank you for reading and not well trusting...I did my best to back up my suppositions with support. But for thinking about it.

And yes, that really is what needs to happen.

More than anything readers need to go to ANY review site, not just the big ones, to form an opinion, not to get one.

If you can stand to, wait a few weeks to pick up a game because the smaller sites (not just this one) don't get their games pre-release, ours are shipped to us the day of release. And we tend to take a little bit longer to play them because most of the writers are volunteer. This means that the writers you read on the smaller sites play like you would - a bit at a time, not all in one day.

Their opinions will be different because it's NOT a job for them. It's a hobby they just happen to write about.

Granted, there are some games that are just good and the buzz is correct. But for that, read the boards to verify. You'll find out more from other players than you ever will from a review.

Thank you again!
2007-11-09 @ 14:54

Leave a comment

Your email address will not be revealed on this site.

Your URL will be displayed.
(Line breaks become <br />)
(Name, email & website)
(Allow users to contact you through a message form (your email will not be revealed.)
July 2015
Mon Tue Wed Thu Fri Sat Sun
 << <   > >>
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

  • Thinking is more interesting than knowing, but less interesting than looking. - Goethe
  • He who can take no great interest in what is small will take false interest in what is great. - Ruskin
  • We don't know who discovered water, but we know it wasn't the fish. - McLuhan


XML Feeds

powered by b2evolution