The Digital Analyst

The cute girl from last Friday’s Christmas party agreed to go on a date with you!  Nice work, my friend!  Time to find a good spot to grab dinner.  If you’re like the rest of my net-savvy generation, you pull up Yelp and start looking for places.  The list of restaurants come up and as you’re scrolling down, you can’t really tell the difference between their ratings.  This sushi restaurant has 3.7 stars, but this Italian restaurant has 3.8 stars and the Mexican place down the street has 3.6 stars.  Is a 3.7 star sushi restaurant really that much better than a 3.6 star Mexican place?  Okay, time to check out what people had to say in their reviews.  One person says they got food poisoning from this place, but another says they proposed to their wife there and another said it’s just okay, but not better than the place around the corner.  Who do you trust?

There is a fundamental problem with the online rating systems we use to give and find feedback on businesses and products.  I’m going to dive into these problems below and try to give a little insight into how I believe we can solve them.  Then I’ll try to provide a glimpse of the vision I have for the future of rating systems and how we use them to discover the world around us.

The Problem with 5 Stars and Written Reviews

The most common rating system we use online is the 5-star scale.  This scale was adapted from the familiar ratings we found in newspapers growing up.  Newspaper critics helped people discover the best new restaurants, movies, plays and events by rating them and justifying that rating with a written review.  When the internet began implementing feedback mechanisms, the 5-star and written review system made the most sense because it was familiar.  Websites could average out the ratings and feature written reviews that users found the most helpful.  It worked pretty well in the beginning, but then as more and more people began rating and reviewing things, the utility of that feedback became murky at best.

The problem that began to emerge with 5-star ratings is that as more people rate things, the less differentiated those things become from the rest of the market.  The clearest example of this problem emerges when searching Yelp in cities where most businesses have over 1,000 ratings.  When the search results populate, you’re going to be stuck looking at a range of ratings from 3.5 to 4.2 stars generally.  The 5-star scale was meant for critics to provide a singular opinion.  It was not designed to be averaged across thousands of opinions.  It also wasn’t designed for amateur reviewers because we tend to rate things at the extreme ends of the scale.

YouTube realized this problem after they analyzed the distribution of user-ratings when they had a 5-star scale below every video.  The graph above shows that users were much more likely to give videos 5 stars or even 1 star than to bother with the nuance of giving a 2, 3, or 4 star rating.  Since sharing this graph in a blog post, YouTube moved to a simple thumbs up or down.  Kudos to them for realizing they made a mistake and fixing it!

There is also the problem that everyone has a different interpretation of the 5-star scale itself.  Some people might hand out 5-star ratings like candy on Halloween and some might hold onto one full-size candy bar for the best costume of the night, yet both of those people are counted equally when averaged.  The subjectivity 5-star scales inject into the rating process even further decreases the usefulness of the average ratings.

If we can’t tell the difference between businesses based on their ratings, the next source we generally turn to is reading the reviews other people have written.  Written reviews suffer from the same problems as the 5-star scale though – they are highly subjective.  When you ask your friends how much they like a business or product, you know who’s opinion to trust (and probably don’t even ask the people you don’t trust).  There is no way to tell who’s opinion to trust and who’s to discard online.  Is the restaurant really that dirty or is the person who wrote that review an OCD germaphobe?  Did this customer really receive poor service or was it written by the owner of a competitor down the street to boost their own business?  Again, who do you trust?

The Solution is Simplicity

How do we solve the rating system problems described above?  KISS!  Keep It Simple, Silly!  Okay, not THAT simple.  The en vogue thing in social media right now is to use an over-simplified system where users just “like” or “+1″ things.  It seems like it should work – the more people who “like” something, the better it is, right?  Wrong.  When you throw out the ability for people to express dislike or negative experiences, you create a system that doesn’t reflect reality.  What does a “like” or “+1″ really mean if people don’t have the option of “disliking” or “-1″ something?

So maybe instead of KISS, it should be KIPS.  Keep It Pretty Simple.  Simplify the rating system into a straight-forward measure of like and dislike.  This rating system approach has gained popularity with the emergence of websites like RottenTomatoes and Reddit.  RottenTomatoes is an interesting case because they have developed a system to convert ratings into a binary positive/negative rating.  They knew that if they just averaged all of the 5-star ratings for movies across the country, their website would be full of movies with 3.5-4.2 stars.  Instead, the movies currently in theaters have scores ranging from How to Survive a Plague’s 100% fresh rating to Playing for Keeps’ rotten 3%.  While RottenTomatoes aggregates movie reviews from vetted sources, Reddit provides a model where the rating process is given over to the community.  Every user has the ability to rate any link submitted to Reddit by giving it an upvote or downvote and links are ranked by their net upvotes (upvotes minus downvotes).  Reddit is quickly becoming one of the most influential sites online and it’s popularity is highly attributable to the beautifully simple rating system at the core of its functionality.

Another part of the solution for improving our rating systems online is the integration of social media’s secret sauce, photos.  Photos have been the fuel that drives websites and apps like Facebook, Reddit, Instagram and FoodSpotting.  At the end of the day, we are very visual creatures.  It makes sense then that any solution for improving rating systems should pay special attention to photos.  Anyone who has used FoodSpotting understands the maxim “a picture is worth a thousand words” is especially applicable when choosing where to eat.  Yelp understands this and has begun to build photos as a central part of their experience, but they are still chained to their flawed 5-star ratings and written reviews.

The Future is Rating Pad Thai

If rating systems are already slowly moving towards a pretty simple binary rating scale augmented with user-submitted photos, where is the next point of disruption and opportunity for innovation?  I believe the future of rating systems hinges on our ability to add layers to our experience.  Right now, we go to a restaurant, come home, and write a couple paragraphs describing what we ate, the service and the overall ambiance.  The rating we give is meant to describe your overall experience with that restaurant, which, similar to “likes” and “+1″, is a little too simple.  Maybe you went to a restaurant, loved the ambiance and food, but your server was new and in over their head?  Life is complicated and our experiences are influenced by a number of different factors.  I believe the rating systems of the future will let you rate each of those complicating factors separately, so you can give the food and ambiance of the restaurant a positive rating, but give the service a negative rating.

Adding these layers respects the complexity of our experiences and provides more useful insight to people who use the ratings to make decisions.  If someone has a hankering for pad thai, they can search for the highest rated pad thai dish near them instead of just hoping the highest rated thai restaurant has the best pad thai.  It lets people search for the best masseuse instead of the best spa and the best beer instead of the best bar.  Adding layers opens up a whole new world of social discovery when combined with a simplified rating system and photos.