If this is your first visit, be sure to check out the Forum Rules. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.
Welcome to RPGamer's new forums running under Vanilla Forums! If you're run into any odd or strange issues after our software migration please see this thread for details

The RPG Sanctum - #13: The RPG Review Scale

RosestormRosestorm Host of The SectorcastFull Members
edited June 2011 in Latest Updates
In this episode of The Sanctum, we examine RPGamer's review scale compared to other gaming publications. We also go on various tangents possibly related to the main topic. This show is rated 4/5.

Listen: http://www.rpgamer.com/rpgsanctum/sanctum13.mp3
Download: http://feeds.feedburner.com/rpgamer/DJog
itunes: http://itunes.apple.com/podcast/the-rpg-sanctum/id416583261

Be sure to leave your feedback in the thread and leave your reviews on itunes.
Deputy Editor of Gamersector.com
Check out my podcast Sectorcast, http://gamersector.com/podcast/sectorcast
A lie would be considered the truth if only more people believed it.
«1

Comments

  • CipherinthesnoWCipherinthesnoW Member Full Members
    edited June 2011
    Great topic, I've always wanted to hear a more in depth account as to how reviews are determined and written.

    Also, in regards to Phil's argument that there aren't any rpg's to review lately, he must not have remembered The Witcher 2. I think that is role playing to the T.. :)

    Keep up all the great work.
  • watcherwatcher Veteran RPGamer Full Members
    edited June 2011
    I disagree with the 4/5 rating for this show. Because the unwritten rules of the forums say that someone must disagree with all staff posted ratings for anything. :P
  • PawsPaws BEARSONA RPGamer Staff
    edited June 2011
    image
  • Fowl SorcerousFowl Sorcerous Dread News Editor RPGamer Staff
    edited June 2011
    i still maintain that my system of colours, shapes and patterns would be a much better means of abstraction.
  • OcelotOcelot is not declawed RPGamer Staff
    edited June 2011
    I like happy face, flat face, sad face, myself. With an angry face reserved for the likes of Blazing Souls: Accelate.
    Becky Cunningham, Staff-at-Large
    Twitter: BeckyCFreelance
  • Just DougJust Doug Member Full Members
    edited June 2011
    You could even add a confused face for "I've played this for 40 hours and I'm still not even sure what this is!" (or in internet meme parlance: "don't know if want...")
    "There are more things in heaven and earth, Horatio,
    Than are dreamt of in your philosophy." - Shakespeare, Hamlet Act I, Scene 5

    "You need mad bank for lobster cash." - Sabin1001
  • XR2XR2 Member Full Members
    edited June 2011
    The longer I play video games, the less emphasis I place on reviews. Usually I look at the numbers, and if the numbers look interesting ( not 2.x or 3.x ) then I'll read the last paragraph and look at a few screenshots to see if it looks interesting. At this point I make a note to pick the game up at some point, or figure I'll take a pass.

    Now I actually read reviews when I'm feeling nostalgic for a game I haven't played in a while and don't have time to play at the moment. Or sometimes when I've started playing an older game an I'm having trouble getting into it. This will let me know if it's worth it to keep going, or if I need to approach the game a little differently.

    I love discussing a game after I've played it, but that would involve too many spoilers for a review. But before I play something, I just want the slightest taste.
  • OmbresOmbres Games horder Full Members
    edited June 2011
    Ah come on, what do you all have again Sword of Vermillon!! It's a good game, did have lot of fun with it back then, ok the sword is too short, but the got big boss battle, nice dungeon exploring, sure it is an old game, but was very well done for is time no?
    Always enjoy all the experience in life, you might gain a level or two.
    image
  • RosestormRosestorm Host of The Sectorcast Full Members
    edited June 2011
    Ombres said:
    Ah come on, what do you all have again Sword of Vermillon!! It's a good game, did have lot of fun with it back then, ok the sword is too short, but the got big boss battle, nice dungeon exploring, sure it is an old game, but was very well done for is time no?

    No one mentioned Sword of Vermilion.
    Deputy Editor of Gamersector.com
    Check out my podcast Sectorcast, http://gamersector.com/podcast/sectorcast
    A lie would be considered the truth if only more people believed it.
  • JuMeSynJuMeSyn Code: Kirin Administrators
    edited June 2011
    I did, to the affirmation of others when it was addressed. The 'big boss battle' is rendered a negative when the controls for it are ridiculously limited, 'nice dungeon exploring' implies that the dungeons are pleasant to investigate before the area map is acquired, and as someone who plays old games all the time, this is one of those that didn't look good even at the time, and has aged terribly. I went into Sword of Vermiliion with a hopeful outlook, and only gradually, as the game mired me in repetitive battles that were so omnipresent as to make me hit the maximum level before the game ended, did the protests of my conscious mind at how much pain was being inflicted upon me weigh down my cheerful inner child with a rock tethered to a cliff.
    It's not what he's eating, but what's eating him that makes it ... sort of interesting.
  • OmbresOmbres Games horder Full Members
    edited June 2011
    Ok I see and I get your point, I guess I look at it from past memory, I did remember having fun with it! Sure it far from perfect, but still was fun for me to play it!
    Always enjoy all the experience in life, you might gain a level or two.
    image
  • XR2XR2 Member Full Members
    edited June 2011
    I wasn't a fan of the boss battles, but I managed to get a bugged save near the end of the game somehow so that each hit only did 1pt of damage, which made things a lot more manageable. I did like the music though, specially the village music. Guess that's what sound tests are for ;)
  • SpartakusSpartakus One Knight Stand Full Members
    edited June 2011
    The review scale is a lost cause. Whether you use 1-10 or 1-100, 5/50 is never an average grade and games hardly ever score beneath it, which is irrelevant because any score beneath 7/70 is considered bad. Grades 1/10 to 5/50 are merely strawmen to be invoked on rare occasions as an alibi.

    RPGamer's 1-5 score works well because the numbers are so small each step of the ladder is significant.
    Currently playing:
    Dragon Age: Inquisition | Final Fantasy VII | The Banner Saga | Ys: Origins | Metal Gear Solid: Ground Zeroes
  • RewindRewind Member Full Members
    edited June 2011
    The only review scale that I really like is a 2 point scale - thumbs up if you like it, thumbs down if you don't. That's enough for me to summarize an opinion, and I can read the review to find out why.
    PSN: TLDragoon
  • Severin MiraSeverin Mira News Director/Reviewer RPGamer Staff
    edited June 2011
    XR2 said:
    The longer I play video games, the less emphasis I place on reviews. Usually I look at the numbers, and if the numbers look interesting ( not 2.x or 3.x ) then I'll read the last paragraph and look at a few screenshots to see if it looks interesting. At this point I make a note to pick the game up at some point, or figure I'll take a pass.

    For me anything between 2.5-3.5 in our scale makes it worth investigating the review to see if it's one of those that may fit within my personal taste. There's a number of games scoring in that range that I personally love, and where the content of the review becomes most important in helping me decide if it's one I might enjoy. Then again it's also fun to read reviews below that and be thankful I didn't have to play it :p

    Also, angry face is far too lenient for Blazing Souls.
    "My doctor says that I have a malformed public-duty gland and a natural deficiency in moral fibre and that I am therefore excused from saving universes."
    Twitter: severinmira | Xbox Live: Severin Mira | PSN: severinmira (EU), severin-US (US) | NNID: severinmira | Skype: severinmk
    Final Fantasy XIV: Sevirain Kristinger (Leviathan)
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited June 2011
    The only thing I dislike about the reviews here is that the scores are often the opposite to what I think. Really good games often just get rated 2/5 while really really horrible games get 4/5 or better!

    I mean sure, I could just use reverse thinking and buy all games rated 1/5 and 2/5 here, but... it feels like the wrong developers get appreciated here and I'm afraid that other gamers will actually listen to the review score and not buy an amazing game and buy the sucky games instead just because they are rated better.

    Then I'd rather have no score at all, so the players are forced to create their own opinion from the facts listed.
    (Note: this also requires the review text to be about the facts and doesn't include the words "good", "bad", "liked" and "disliked".)
  • OcelotOcelot is not declawed RPGamer Staff
    edited June 2011
    If you want "just the facts, ma'am," read the Wikipedia summary of the game's basic features. Reviews are supposed to be subjective, what makes them good or bad is a reviewer's ability to back up his/her opinion with examples. Nobody is ever going to agree with every review, but if somebody finds themselves disagreeing with the majoirty of reviews on any site, (not just RPGamer) that person might consider that it's not the reviewers, it's them. Like the majority of medium-sized gaming sites, RPGamer has 5-6 regular reviewers, all of whom have quite different preferences and tastes in gaming. It's not unusual for a regular reader to find that they have different preferences from a particular reviewer, but if somebody finds themselves disagreeing with all of the reviewers on a site, they just might have their own quirky preference in games.

    There's absolutely nothing wrong with enjoying games that a lot of other people don't... I know I've done so on various occasions. However, doing so doesn't mean that reviews that dislike a game you enjoy are wrong, or that a site tends to trash "good" games and promote "bad" games. Now, I'm personally not a huge fan of numerical review scales, but as somebody who writes reviews for a living, I do my best to use my employers' review scales as a way to communicate my general feelings about a game in shorthand. I don't generally write reviews for RPGamer, but I do help to proof them, and I can tell you that every review we post includes a discussion between the reviewer and at least two other people as to whether the scores given match the text of the review. We have some heated debates on the matter, in fact. The end result is that the scores are generally a good indicator of how the reviewer felt about the game, so if you don't like the scores, you probably simply disagree with the reviewer about the game. Nothin' wrong with that, but it doesn't make either the reviewer or any particular reader the holder of the ultimate truth about a game.
    Becky Cunningham, Staff-at-Large
    Twitter: BeckyCFreelance
  • ChickenGodChickenGod Overdosing Heavenly Bliss Moderators
    edited June 2011
    Severin Mira said:
    For me anything between 2.5-3.5 in our scale makes it worth investigating the review to see if it's one of those that may fit within my personal taste. There's a number of games scoring in that range that I personally love, and where the content of the review becomes most important in helping me decide if it's one I might enjoy. Then again it's also fun to read reviews below that and be thankful I didn't have to play it :p

    While this may not be quite on topic, I was inspired by you and Rya to browse through the reviews and find games that were scored 2 or lower that I really liked, and games scored 4 or higher that I hated. I browsed all reviews from the top 7 reviewers, and what I found was kind of interesting.

    Games rated 2.0 or under that I really enjoyed were as follows: Stella Deus, Growlanser 3, Shadow Hearts, Wild Arms 2 (1.0?!), Grandia 2, Legend of Dragoon, MegaMan Battle Network 5, Final Fantasy 8, Devil Survivor, Final Fantasy 5, and Arc The Lad 2. As can be seen, many of these games are generally well-liked and its kind of surprising to see them ranked so low. Still, I only managed to find a total of 12 games, and that number is remarkably low. Also consider the fact that there are no two reviewers who have consistently rated these as being bad. Another review can be found for many of these games, often changing the score from "Bad" to "Great" for games such as Shadow Hearts and Grandia 2, so I think the reviewers here are doing a pretty solid job on this front.

    The titles rated 4.0 or higher that I really didn't care for are thus: Shining Force GBA, Disgaea, Valkyrie Profile 2, Jeanne d'Arc, Valkyria Chronicles, Demon's Souls, Fable, Nier, Skies of Arcadia, Fire Emblem: Shadow Dragon, Breath of Fire 5, Dragon Quest 4 DS, Final Fantasy 12, Final Fantasy X-2, Kingdom Hearts: Days, and Mana Khemia 2. Unsurprisingly, of the 16 I listed, most of these games are very well liked and reviewed positively elsewhere as far as I know, such as DS, Nier, Skies, and FF12. I wouldn't be at all surprised if anyone took a look at this list and just thought that I was crazy for disliking at least 3/4th of these, so the reviewers must be doing pretty darn good in this category as well. One thing I will note is that unlike most of the games rated poorly, I couldn't really find a more neutral/negative score for the vast majority of these, which would indicate that there really isn't a solid "2nd Opinion" for someone like me who happens to hate well acclaimed titles such as Skies, Valkyria Chronicles, or FF12.
    "Looks like Teach just got tenure!" - Teach
  • OmbresOmbres Games horder Full Members
    edited June 2011
    Well like Ocelot say, review are really subjective, we do all have our preference, but then again a good reviewer, like the one in RPGamer, always describe well everything about the game and what they liked or not, and most of the time, I was always able to know if a game is good for me after reading about it. I might add that you learn the taste of certains reviewers and the one that have similar taste to your, so you can learn that way who you should follow more his/her idea and vice-versa
    Always enjoy all the experience in life, you might gain a level or two.
    image
  • DarkRPGMasterDarkRPGMaster A Witness to Destruction Moderators
    edited June 2011
    ChickenGod said:
    While this may not be quite on topic, I was inspired by you and Rya to browse through the reviews and find games that were scored 2 or lower that I really liked, and games scored 4 or higher that I hated. I browsed all reviews from the top 7 reviewers, and what I found was kind of interesting.

    Games rated 2.0 or under that I really enjoyed were as follows: Stella Deus, Growlanser 3, Shadow Hearts, Wild Arms 2 (1.0?!), Grandia 2, Legend of Dragoon, MegaMan Battle Network 5, Final Fantasy 8, Arc The Lad 2, Devil Survivor, Final Fantasy 5, and Arc The Lad 2. As can be seen, many of these games are generally well-liked and its kind of surprising to see them ranked so low. Still, I only managed to find a total of 12 games, and that number is remarkably low. Also consider the fact that there are no two reviewers who have consistently rated these as being bad. Another review can be found for many of these games, often changing the score from "Bad" to "Great" for games such as Shadow Hearts and Grandia 2, so I think the reviewers here are doing a pretty solid job on this front.

    The titles rated 4.0 or higher that I really didn't care for are thus: Shining Force GBA, Disgaea, Valkyrie Profile 2, Jeanne d'Arc, Valkyria Chronicles, Demon's Souls, Fable, Nier, Skies of Arcadia, Fire Emblem: Shadow Dragon, Breath of Fire 5, Dragon Quest 4 DS, Final Fantasy 12, Final Fantasy X-2, Kingdom Hearts: Days, and Mana Khemia 2. Unsurprisingly, of the 16 I listed, most of these games are very well liked and reviewed positively elsewhere as far as I know, such as DS, Nier, Skies, and FF12. I wouldn't be at all surprised if anyone took a look at this list and just thought that I was crazy for disliking at least 3/4th of these, so the reviewers must be doing pretty darn good in this category as well. One thing I will note is that unlike most of the games rated poorly, I couldn't really find a more neutral/negative score for the vast majority of these, which would indicate that there really isn't a solid "2nd Opinion" for someone like me who happens to hate well acclaimed titles such as Skies, Valkyria Chronicles, or FF12.

    I bolded everything I have played and enjoyed. Just 7 of them in the lower than 2.5 category out of the whole site is pretty good on the reviewing front, since it tells me that I'm probably going to get an unbiased look at the negatives for those games (all of which I already know), and that the reviewer lowering the score is somebody I'd like to see the interests of (because who knows, they may give me a good reason to try out a game I don't think I'll like). It also tells me you guys should keep up the good work.
    User Database Check
    Pages cleared: 98
    Users cleared: 2940
    Current Year: 2010
    Spammer-Banning Battle Theme: You Will Know Our Names
    Number of Spammers banned: At least 2000 at this point.
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited June 2011
    Well you can't change anything about reviewers having different opinions than the readers. I'm not even complaining about this.

    What can be changed however is whether a review score exists or not and how objective it is (please no more discussion about how objective a score can be, it can certainly be made more objective, even if it's never 100%).

    From Ocelot's reply you can clearly see that RPGamer does have a review score and intentionally makes it subjective ("It should reflect how the reviewer felt about the game"). This is the main point where I disagree with Ocelot.

    I don't think review scores should be intentionally subjective, at least not when done by real game journalists. I usually see it like this: Let's say a we have a blog reviewer. I expect his reviews to be as objective as possible, but I expect the "comments" to the blog entry to reflect subjective opinions. I read a review because I want to know how a game is like, not because I want to know what the reviewer thinks about it. I want to know that in FFXIII there are no towns you walk around in, buy stuff and talk with random NPCs. I do not want to know that the reviewer considers this as a huge flaw and thus can only gives the game a 1/5 score. On the comments however I'm interested in the opinion, because there are many. This is the main difference. If there are many subjective opinions, I will always see both viewpoints about a feature. I will see "Okay person X doesn't like feature A because of M but person Y does like feature A because of N", now I reflect myself and think about what I would think. Here subjective opinion is a good thing, because you can compare. When a single reviewer reviews a game however, this is not present and that's why I think being subjective here is a huge mistake.

    That, and I'm generally again any kind of scoring system. Just summarizing "good points" and "bad points" is enough. Seeing 5/5 or 1/5 is completely meaningless if it only depends on the subjective opinion of a single person.
  • PawsPaws BEARSONA RPGamer Staff
    edited June 2011
    The scores in the little box on the side are there for those who need it. The bullet points and the 1500 words comprising the rest of the page are for everyone else -- and for people who don't think numbers are cool.
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited June 2011
    The scores in the little box on the side are there for those who need it.
    But the point is: Nobody needs it. Even the people, who think that they need it, don't.
  • OcelotOcelot is not declawed RPGamer Staff
    edited June 2011
    All I'm going to say here is that while a few people may only want to read basic facts in a "review" of a game, no reviewer gets paid to recite facts about a game. We get paid to provide critical analysis of a game. Although those of us who care about our craft try to be fair and to write our reviews in a way that is relevant to as many gamers as possible, without analyzing a game's strengths and weaknesses, it's simply not a review. If I were to turn in a "review" that only summarized a game, it'd get sent back to me and I'd have to rewrite it as a review. Critical analysis (which necessarily contains subjectivity) is what the vast majority of the readership wants. If it wasn't, our reviews wouldn't get page hits and therefore we wouldn't get paid. Sorry if you want something different, but again, go read Wikipedia or something and stop trying to make your opinion about reviews out as if it's what "real" journalists should do (pro tip: insulting the professional qualifications of people with whom you don't happen to agree is both lame and insulting).
    Becky Cunningham, Staff-at-Large
    Twitter: BeckyCFreelance
  • XR2XR2 Member Full Members
    edited June 2011
    I happen to love the little box of numbers on the side of the review. I rarely bother with more than the conclusion of the review, at least until after I've played the game.

    That said, the question of exactly what a review is for, and what perspective to write it is an interesting question. I don't think constraining a review to strictly factual information about a game is even possible, let alone desirable. For example, typing out a description of how Atlas's press turn battle system works make it sound terrible. Teams take turns? Main character death? With two different types of instant death attacks? Sounds awful, but it's one of my favorite battle systems from the last decade. Or if we take Persona 3 and stick to the facts that it has random dungeons, main character death, and you only control the main character with no subjective opinion of how these elements work together is like trying to judge a recipe by reading an ingredients list. Now maybe Atlas's latest works aren't to your taste, but there's more to a game than the sum of it's parts, and that's what a review needs to try to quantify.

    Now my question is how much a review should try to be 'objective', and how much they should be subjective. Obviously complete objectivity isn't possible, so then the question is how objective can and should a reviewer be? Can a reviewer accurately determine that something is good but not to their tastes? Or the opposite, that it's not really that good, but does a good job of scratching their particular itch? For me, I love the original Lunar game on the Sega CD. I love the humor, the strong character definition (especially for it's time), and how the game made the quest personal for Alex. So if I were to score this based on how I feel about the game (purely subjectively), I'd give it a 4 or 4.5. However, the game does have it's flaws. The colors are muted, the battles can get time consuming, and the interactivity is horrible (no spell costs and it reorders them according to last use, and have poor names besides, just for one example). So if I were trying to be 'objective' about it, I'd have to give it a 2.5 or a 3.0.

    Either way, I want the reviewer's impression of how they felt the game came together, not just a list of features and implementation details.
  • XR2XR2 Member Full Members
    edited June 2011
    There are also other questions on how to approach a review. Obviously playing the game to the end once should be a requirement, but is that enough? What if the game is like Fallout 3, and a lot of the content is optional? What if there are a lot of side quests, but they don't really add to the game? What if there is a good path and an evil path to the game? How much should a game's ability to try and tailor itself to the experience the reviewer is looking for count in the score? What if you have a different play style, play a different type of character, and take a different path through the game? How much of what a reviewer says could even be relevant to your experience?

    A good review has a lot of ground to cover. It has to explain where the reviewer is coming from, how the game works, what the review felt worked well, what the review felt didn't work well, who the target audience is, what kind of gamer might like a particular game and why, and what type of gamer might not like a particular game and why. All of this while touching on characters, plot elements, game systems, graphics, art, technical details, music, etc without giving too much away. And it has to do all this is as few words as possible so that someone may actually read it. I'm guessing it's not as easy, or as straight forward as it looks.
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited June 2011
    It's funny because I think Atlus's combat system are horrible right out of the objective reasons you mentioned. For example Persona 3 has horrible gameplay but a good story. I played the game because of the good story for 20 hours (a lot for me), but that doesn't make the gameplay any better.
    Yet a review should neither say "1.0/2.0 because of sucky gameplay" nor "4.0/5.0 because of great story", it should just not have a score at all and simple say "Gameplay bad because: Objective fact #1, objective fact #2, ..." and "Story good because: Objective fact #1, objective fact #2, ...". Yes "good" and "bad" are still subjective, but since it's bound to objective facts, the reader can make his own opinion. For example he sees "ah, the reviewer didn't like the game because it had randomly created dungeons, hey but I like randomly generated dungeons, so this might be something for me". That's still okay.

    About your second post I'd just say the review should mention it if there is a lot optional content and if there are different paths. He doesn't need to play all of them, but he should at least have reliable sources on it.


    @Ocelot
    That a reviewer wouldn't get paid for a review, isn't really a good reason why the review is bad. Maybe the person paying is wrong in the first place as well.
  • SpartakusSpartakus One Knight Stand Full Members
    edited June 2011
    ChickenGod said:
    Games rated 2.0 or under that I really enjoyed were as follows: Stella Deus, Growlanser 3, Shadow Hearts, Wild Arms 2 (1.0?!), Grandia 2, Legend of Dragoon, MegaMan Battle Network 5, Final Fantasy 8, Devil Survivor, Final Fantasy 5, and Arc The Lad 2. As can be seen, many of these games are generally well-liked and its kind of surprising to see them ranked so low.

    I'm usually lenient with reviewers but...wow, what were they smoking. I seriously think the ones who wrote those reviews should revisit the games with an open mind. Yeah the junction system in FF8 is broken and tedious but in no way does it detract that much. The music alone should bump it over the 2.0 mark.
    Currently playing:
    Dragon Age: Inquisition | Final Fantasy VII | The Banner Saga | Ys: Origins | Metal Gear Solid: Ground Zeroes
  • QuinQuin ne cede malis RPGamer Staff
    edited June 2011
    Hold on, back up a sec...
    Rya.Reisender said:
    Persona 3 has horrible gameplay

    Okay, I've gotta hear the "objectivity" on this one.
    [SIGPIC][/SIGPIC]
    befriend (v.): to use mecha-class beam weaponry to inflict grievous bodily harm on a target in the process of proving the validity of your belief system.
  • SpartakusSpartakus One Knight Stand Full Members
    edited June 2011
    I partially agree. While there is nothing wrong with the gameplay mechanics the tower is just crazy boring to play through. It baffles me that so few reviewers can't see the flaw in having the majority of the game play out in a huge, featureless and randomly generated dungeon. It was so dull it was gamebreaking for me. The fatigue system compounded the issue by not letting me plow through as much as I could but making me start over and over and over...
    Currently playing:
    Dragon Age: Inquisition | Final Fantasy VII | The Banner Saga | Ys: Origins | Metal Gear Solid: Ground Zeroes
Sign In or Register to comment.