Meta criticism of information aggregation and filtering services: the danger of losing control about our decisions
Included in this is the score of “4″ from user “Breck”.
The issue: “Breck” is not rating “Supreme Commander 2″ with a “4″, his comments are making very clear that he is rating a “final fantasy” game. How has this wrong “4″ now changed the calculation of the user score of “6.1″?
This is very likely just another simple case of a data quality issue that can be easily fixed and should not at all to be understood as criticism of Metacritic as a service (especially taking into account that user and review ratings by nature are highly subjective).
But there are generic questions behind this case that lead us to a meta criticism of information aggregation and filtering services.
How often do we rely on information services that filter/aggregate and search for us information and we do not know:
- what sources this service is using (and what sources not)?
- what algorithms for filtering, sorting and aggregation this service is applying?
- what quality control mechanisms are executed on a regular basis for that services?
The type of information that is made accessible to us has an influence on our decisions and somehow on our thinking. As long as logic and data is packaged together in a black box we are in danger of giving up bit by bit autonomy about our decisions and our thinking. How can we avoid that?
Recommendations going forward:
- Decide about what information services we are using based on their transparency about the data and the logic they are using
- Do not rely on a single service for information filtering even if itself is an information services that is aggregating information from many sources (like in our case here you could additionally look at GameRankings or other similar services; you are probably interested in this discussion “Metacritic or Gamerankings?” on GameSpot forums)