Meta criticism of information aggregation and filtering services: the danger of losing control about our decisions

Metacritic is a very valuable service that aggregates reviews into a “Metascore” for entertainment products like games. For “Supreme Commander 2″ Metascore is calculating a user score of “6.1″.

sup2

Included in this is the score of “4″ from user “Breck”.

aggregatewrong1

The issue: “Breck” is not rating “Supreme Commander 2″ with a “4″,  his comments are making very clear that he is rating a “final fantasy” game. How has this wrong “4″ now changed the calculation of the user score of “6.1″?

This is very likely just another simple case of a data quality issue that can be easily fixed and should not at all to be understood as criticism of Metacritic as a service (especially taking into account that user and review ratings by nature are highly subjective).

But there are generic questions behind this case that lead us to a meta criticism of information aggregation and filtering services.

How often do we rely on information services that filter/aggregate and search for us information and we do not know:

  • what sources this service is using (and what sources not)?
  • what algorithms for filtering, sorting and aggregation this service is applying?
  • what quality control mechanisms are executed on a regular basis for that services?

The type of information that is made accessible to us has an influence on our decisions and somehow on our thinking. As long as  logic and data is packaged together in a black box we are in danger of giving up bit by bit  autonomy about our decisions and our thinking. How can we avoid that?

Recommendations going forward:

  • Decide about what information services we are using based on their transparency about the data and the logic they are using
  • Do not rely on a single service for information filtering even if itself is an information services that is aggregating information from many sources (like in our case here you could additionally look at GameRankings or other similar services; you are probably interested in this discussion “Metacritic or Gamerankings?” on GameSpot forums)


Bookmark Buttons
Bookmark this: Digg Bookmark this: Del.icio.us Bookmark this: Facebook Bookmark this: StumbleUpon Bookmark this: Google


November 6th, 2010 at 12:03 pm and is filed under Issues explained. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply