It isn't just restricted to app ratings. How often to you see a low star rating and then read the review only to find that there is a conflict between what has been said and the number of stars awarded? What is going on? Are the users simply logically inconsistent beings? Of course they are!

More cartoon fun at xkcd a webcomic of romance,sarcasm, math, and language

Reviews are important and we all know that the star rating system is even more important and that an app with a low star rating probably doesn't even get considered by the user. What is worse is that after dismissing a low-rated app the user probably doesn't go on to actually read the reviews to find out if the rating is justified in any way.

Incorrect star ratings are therefore important enough to be the subject of some research. Rahul Aralikatte, Giriprasad Sridhara, Neelamadhav Gantayat and Senthil Mani, all of IBM Research decided, to find out what the situation actually was on the Android Play Store.

"Unfortunately, due to a variety of reasons, often the star rating provided by a user is inconsistent with the opinion expressed in the review. For example, consider the following review for the Facebook App on Android; “Awesome App”. One would reasonably expect the rating for this review to be five stars, but the actual rating is one star! Such inconsistent ratings can lead to a deflated (or inflated) overall average rating of an app which can affect user downloads, as typically users look at the average star ratings while making a decision on downloading an app."

The researchers manually examined close to 9000 apps and found that 20% had inconsistent, both overly good or bad, ratings for the reviews they received. This is an amazingly high proportion and could result in signification distortion in the average rating.

In an effort to auto detect such inconsistent ratings, they tried out a range of machine-learning approaches. Of those tried, all of the traditional approaches only managed mediocre performance, except for the neural network which achieved a 92% accuracy - this part of the study is worth looking at for a comparison of methods.

As a test they then got 23 app users to write reviews and assign star ratings. The neural network achieved an 87% success in predicting the star ratings. This suggests that Google, and every other review facility, should run the neural network over the star ratings and either flag or discount or even revise any anomalies. The network was also used to estimate the number of erroneous star ratings in the store and came up with a figure of between 16% and 26%.

Sadly there is no indication of why star ratings can be so inaccurate. Is it the user interface or is it the user? As programmers we all know what we think of users...

More Information

Fault in your stars: An Analysis of Android App Reviews

Related Articles

Amazon customer reviews - can we trust them?

Google Introduces Manual Review For Apps

Fear And Loathing In The App Store 4 - When Apps Vanish

Fear And Loathing In the App Store 6 - Apple's Reasons For Rejection

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info