If this is your first visit, be sure to check out the Forum Rules. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.
Welcome to RPGamer's new forums running under Vanilla Forums! If you're run into any odd or strange issues after our software migration please see this thread for details

Active Topical Banter - Episode 4: Feelings out of a Number

Fowl SorcerousFowl Sorcerous Dread News EditorRPGamer Staff
edited May 2014 in Latest Updates
Scoring is a fickle beast for a reviewer. Our crew attempts to tame and/or slay the monster with all of our of bits intact.

Listen
archive
Subscribe
more different subscribe

Talking Points:
-humans as abstractors
-illusion of objectivity
-Scott's joke system of colours and shapes scoring
-Scott's not completely joking 4(ish) point scale
-justification of love/hate
-numerical weighting (anything below 6/10 is trash)
-metacritic is ********
-hate.8 out of ten


Next Time: Engagement and Immersion

Comments

  • smacdsmacd Full Members
    edited May 2014
    Game reviews and how people deal with them can be an interesting topic. I know my tendency is to look at the score and reviewer, then scroll to the bottom to read the summary paragraph, and then decide whether I'll read the entire review or if I had enough information. I know that they are subjective (and should be), and its rare for me to make a decision about any game based on just one review. I am bothered by the "school" scale used for games, I mean, if everything under 75% is "bad", why not just use one number to represent that? On the other hand, while I'm not a fan for the methods used by Metacritic, nor issues like Obsidian's Fallout NV crap, but I do like the idea of the site and I like using it as a rough gauge for how well received games are. For example, Watch Dogs getting low 80s after being massively hyped up and being the most-preordered game of all time is pretty telling, even though the low 80 value is fairly meaningless itself. I get the message that the game definitely didin't live up to the general hype, but that I will still reserve my judgement of the game to what specific people think of it, or my own experience.

    Most everything I would have said was mentioned at some point by someone, so gotta say that I think you guys really covered the gambit this time. Although, probably a topic for another show, but I can't tell you how much it bugs me when the unwashed masses on the internet verbally abuse or threaten reviewers based on opinions.

    Also, I'm a Nyx fan! There are definitely some voices on the site I'm more a fan of than others, though I think everyone's voice is important- including those that I disagree with on virtually everything.

    (off topic) As for my username, I say "smacked" rather than "smack dee", but whatev. Its probably pretty obvious from my real name where I came up with it. I should probably come up with something better, but I've been using it for so long that its pretty much as normal as seeing my own name. My fiancee hates it though.
  • ghaleon80ghaleon80 Pass Christian, Mississippi Full Members
    edited May 2014
    I personally think metacritic is not a good site to use for game reviews. I've often found that to get a good review you have to use different sites depending on the genre. Lots of other game sites have people who aren't fans of a genre reviewing that type of game which sometimes results in a game getting a unfair rating. There needs to be more genre based websites such as RPGamer or at least more genre specific reviewers on some other sites. I don't like the idea of a reviewer who's profile shows their favorite game types as nothing but FPS or Sports games reviewing my RPGs
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    Metacritic is good to see the score and that's basically it. Taking the average of all scores on the internet gives a very good and fairly "objective" view on the game quality.

    For review text it's much better to focus on reviewers you actually like and usually agree with. Everything else is pretty useless. Yeah, I can draw some info from reviews of reviewer that have a different taste than me because someone this points listed under "negative points" are the reasons why I end up buying the game or I could just go like "He gave the game a 5/5, so I will probably not like it at all". But it's very inaccurate. A reviewer will look at a feature he likes differently than a feature he doesn't like. And if you like the feature then reading the reasons why someone else didn't like it won't give you useful info. On the other hand someone who likes the feature just as much as you will analyze it just as you would, so the information is much more useful.

    My main problem is usually that I don't really remember reviewers at all. On RPGamer I know that I have pretty much the opposite opinion as 80% of the staff, I really should learn who are the remaining 20% I can actually listen to, but I keep forgetting it (it's hard because the real names are used in the reviews but not on forums).
  • MacstormMacstorm Ysy St. Administrators
    edited May 2014
    Metacritic is good to see the score and that's basically it. Taking the average of all scores on the internet gives a very good and fairly "objective" view on the game quality.
    I disagree completely and would go into more detail why, but make my points clear in the podcast. Not all scores are created equal and MC doesn't do a good job of being meaningful.
    "The universe is already mad. Anything else would be redundant."
    Twitter @FinalMacstorm
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    It doesn't matter if all scores are created equal or not as you always take the average of all scores.

    If half of the reviewers think 3/5 means average score whereas the other half thinks 80% means average, then it will simply end up with 3.5/5 or 70% representing average on Metacritic.
  • MacstormMacstorm Ysy St. Administrators
    edited May 2014
    It doesn't matter if all scores are created equal or not as you always take the average of all scores.

    If half of the reviewers think 3/5 means average score whereas the other half thinks 80% means average, then it will simply end up with 3.5/5 or 70% representing average on Metacritic.
    That's an overly simple way of looking at it, but would be true if all sites reviewed all games. Since that's not the case, you end up with meaningless scores all across the board.
    "The universe is already mad. Anything else would be redundant."
    Twitter @FinalMacstorm
  • ShayminShaymin The Gratitude Pokemon Halifax, New SealandFull Members
    edited May 2014
    As a regular (and host of the forum podcast) for the site mentioned that gave a double 7.5 (independently) for Mario Kart 8, that was a fun few days of trying to explain to people that they use the whole scale and it's their opinion. Sadly, just as one would leave another would come in and the process would reset. Fun times. :dispirited:

    Scott and Mac have Metacritic nailed... I wish it was easy to ignore it, but it still gets a lot of traffic and I doubt most people are going there for movie reviews in a world where Rotten Tomatoes exists. I think if they ditched the concept of the metascore and simply listed reviews with a link to a site's rubric if available, they'd be doing a service to the industry. But that wouldn't bring in the clicks.

    I've been trying to train myself to not look at scores over the last few years, and relying on Twitter/podcast/message board impressions for a lot of my game buying decisions. To that end, I've cut myself down to two sites (the aforementioned fan site and of course, RPGamer) - one has an easy way to just see the review text, and for RPGamer... the old "hold your hand in front of the score portion" trick comes into play. I hope that review scores stop being so meaningful over the next few years, but let's just say I'm not holding my breath.
    "The flowers all over its body burst into bloom if it is lovingly hugged and senses gratitude."
    Twitter | A gaming podcast by grownups | Nintendo World Report news editor
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    Macstorm wrote: »
    That's an overly simple way of looking at it, but would be true if all sites reviewed all games. Since that's not the case, you end up with meaningless scores all across the board.
    If you have many reviews from which you draw the average score (which metacritic has) then the error margin caused by this is small.
  • MacstormMacstorm Ysy St. Administrators
    edited May 2014
    If you have many reviews from which you draw the average score (which metacritic has) then the error margin caused by this is small.
    Not for every game, especially niche games, which then get lumped in with those mainstream AAA games that do. Thus the meaningless scores.
    "The universe is already mad. Anything else would be redundant."
    Twitter @FinalMacstorm
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    Hmm, I guess that's true. I mostly use Metacritic for popular titles because of that.

    For smaller titles I usually just use my own "first impression" by looking at screenshots or gameplay videos if available. I often write reviews myself.

    But anyway, that you say it only works for popular titles just underlines my point: The bigger the mass of reviews averaged together, the better (more objective) the score.
  • MacstormMacstorm Ysy St. Administrators
    edited May 2014
    Hmm, I guess that's true. I mostly use Metacritic for popular titles because of that.

    But anyway, that you say it only works for popular titles just underlines my point: The bigger the mass of reviews averaged together, the better (more objective) the score.
    I can agree that for major titles (Call of Duty, Halo, GTA) that it probably works well enough like you say. But niche gaming is screwed in that process. This is fine for those willing to look beyond MC, but it means that niche will always have an uphill battle, which is sad because they are lumped in and look lesser in comparison. The brainwashing that a metascore is a universal truth that can be equally applied to all games is a problem.
    "The universe is already mad. Anything else would be redundant."
    Twitter @FinalMacstorm
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    But do you really think people actually use Metacritic when they are looking for niche games? Because for me it seems pretty obvious that its advantage stems from the mass of reviews rather than the quality.

    It's kind of like Amazon ratings. Their review and rating quality of a single one is pretty bad but if there are 100+ the average is still pretty accurate in determining how well the product is perceived.
    If I see an Amazon product that only has one rating which is 1 star or 5 star I usually don't consider that as meaningful.
  • MacstormMacstorm Ysy St. Administrators
    edited May 2014
    Niche gamers don't need it to find info on niche games, but it will forever keep the quality games that could reach beyond the niche down.
    "The universe is already mad. Anything else would be redundant."
    Twitter @FinalMacstorm
  • PawsPaws Purr RPGamer Staff
    edited May 2014
    If you have many reviews from which you draw the average score (which metacritic has) then the error margin caused by this is small.

    Four is not many.

    It gets even murkier when you consider there's no way to vet reader reviews for people who actually own the game, and those just being twits.
  • Rya_ReisenderRya_Reisender Solipsist Snowflake Full Members
    edited May 2014
    How do you get to "Four". Even smaller titles that are fairly new have more ratings than this on Metacritic. Let's just take Child of Light as an example: 39 review and 221 user ratings.
    It gets even murkier when you consider there's no way to vet reader reviews for people who actually own the game, and those just being twits.
    Only if you want to differentiate that at all. If someone "downvotes" a game he hasn't played, there must be a fairly big reason for it (you would only do that if you really hate something).
  • PawsPaws Purr RPGamer Staff
    edited May 2014
    Four is the minimum required to get a rating, and Child of Light is not a small title. Also, despite your darling predilection, no, people who haven't played a game shouldn't review it.
  • Severin MiraSeverin Mira News Director/Reviewer RPGamer Staff
    edited May 2014
    Some games I've reviewed are rather light on the total (i.e. Four), yet still have an average.
    "Time is an illusion. Lunchtime doubly so."
    Twitter: severinmira | Xbox Live: Severin Mira | PSN: severinmira (EU) | NNID: severinmira
    Final Fantasy XIV: Sevvi Taubemira (Leviathan)
  • TexsideTexside Member Full Members
    edited May 2014
    Great podcast.

    I thought the point about Metacritic was spot on; I've read in the social sciences about how averages are already dangerous things, especially with a small sample size, and I think this highlights that.

    I also think -- and I will say, up front, that I do not know what calculations (if any) Metacritic employs -- that averaging scores across sites is a dangerous notion. I know this was talked about in the podcast, and I have to agree; I think it's very hard to say RPGamer's scores match up with Site A and Site B, even if you make a sliding scale. Different review systems look for different things, and emphasize different things in the abstraction of making a numerical score. Averaging them feels dangerous, because we lose this information; if a score hides (for example) Nyx's opinion about Adventure Time despite technical flaws, an averaged amalgamation hides every reviewer's experience, which defeats the point of reviews entirely. That lumping of all those scores together strips the scores of the information that makes them meaningful, even if we assume every reviewer involved played the game and Metacritic has weighted each site's scoring properly.

    I've used RPGamer reviews to decide whether to purchase games in the past. Usually, I start at the scores because they're at the side of the screen and they're a handy way to get an idea of what I'll be reading; I dunno if this is common, but it's a good way to prime myself for what to expect. The actual text of the review has been what shaped my decision, and I don't think a score alone could let me form an informed decision. That score is ultimately an abstraction of the reviewer's experience, and that experience is the part that helps me decide where my dollars are going.

    I've personally found reviews valuable, though, for things unrelated to the score, if I'm looking at a game I've played before. Seeing what others thought, especially if they disagreed with me, is just intellectually fascinating. I've seen criticism I've disagreed with, but I've also seen sharp criticism of games I love in reviews. It depends on my mood -- if I'm tired after work and have a headache, I may just want to pretend that Rhapsody is actually Dragon Fantasy Quest VI+IX: Let Us Cling to Radiata Trigger, Game of the Year Edition, Endorsed by Anthony Bourdain, Neil deGrasse Tyson, and Jareth the Goblin King, and not want to hear sense -- but sometimes, I want to think about why I liked something and why I overlooked something others couldn't stand. It tells me something about what I'm playing.

    Review scores definitely​ cannot do that.
  • silktailsilktail mild peril Full Members
    edited May 2014
    I'll give this podcast 4 apples out of a Cornucopia.

    ...

    That's not a score, I just want rid of this fruit. I have plenty enough!
    My main problem is usually that I don't really remember reviewers at all. On RPGamer I know that I have pretty much the opposite opinion as 80% of the staff, I really should learn who are the remaining 20% I can actually listen to, but I keep forgetting it (it's hard because the real names are used in the reviews but not on forums).
    Maybe bookmark the Staff Bios page, to get a handle on people's handles?
    http://www.rpgamer.com/bios/bios.html
    Shaymin wrote: »
    I've been trying to train myself to not look at scores over the last few years, and relying on Twitter/podcast/message board impressions for a lot of my game buying decisions. To that end, I've cut myself down to two sites (the aforementioned fan site and of course, RPGamer) - one has an easy way to just see the review text, and for RPGamer... the old "hold your hand in front of the score portion" trick comes into play.
    I've been doing something like that too. (I usually use another window to cover up RPGamer's score).
    Except I do it to get a feel of how the game should score (for me), before I look at the actual score given.


    I do find a numeric score useful for comparing the games reviewed by the same people. And subsequently a 100 point scale could be justified for full time reviewers.
    However you do have to know their scoring system (where their "average" lies, etc) and you can't simply compare different games, reviewed on different scoring systems.

    Metacritic might as well be holding an internet poll, if you don't know scoring system of each source used.
    Or better yet they should switch to the 4(ish) point scale, because a 100 point scale accuracy is completely ridiculous here.


    But enough of metacritic! Lets talk about something more important:-

    ...

    I remember Sam (Wachter-O'God)'s accidental romance story. :love_heart: She has talked about it in a past RPGCast before.
    She can have a pair of pears from the cornucopia for it. (I'm never getting rid of this fruit am I?)

    Let's see ... it was the "Game of the Year 2011" podcast:
    http://board.rpgamer.com/forum/showthread.php?19065

    I'm afraid it is well over an hour into a pretty long podcast, but it had its own song and everything.
    I recall followed their suggestion to send in April Fools fanart. Disappointingly the Fanart section shutdown before that happened (probably unrelated), so no-one saw it.
Sign In or Register to comment.