Fake Reviews & The Google Play Store

Posted by on Oct 9, 2014 in Affiliate Fraud, App Stores

The Fair Credit Reporting Act ensures that Americans are granted a free credit report from Equifax, Experian and TransUnion once every twelve months. If you are fortunate enough not to encounter a scammer along the way to getting your free credit report, then you’ll most likely end up at  annualcreditreport.com

Unfortunately, not everyone is so lucky. The FTC has something to say about Websites that are exploiting folks looking for their free credit report:

Other websites that claim to offer “free credit reports,” “free credit scores,” or “free credit monitoring” are not part of the legally mandated free annual credit report program. In some cases, the “free” product comes with strings attached. For example, some sites sign you up for a supposedly “free” service that converts to one you have to pay for after a trial period. If you don’t cancel during the trial period, you may be unwittingly agreeing to let the company start charging fees to your credit card.

There’s no shortage of exploitation in this vertical on the Web, just do a couple of searches for yourself and take a close look at the ads. Be careful if you’re a scammer looking to get in on the action: the FTC may come knocking!

Moving away from the Web, and using your favorite Android device, a quick search for “free credit score” on the Google Play App store yields hundreds of results.

google-play-reviews-00

google-play-reviews-01

Of interest in this article are search results #2 and #3:

#2: Free Credit Report & Scores by Sinsation: this app has 10,000 downloads and a review score of 4.6 out of 5 stars.

google-play-reviews-02

#3: Credit Score Pro Free Reports by Amazing Apps Inc: 5,000 downloads and a 4.7 out of 5 review score

google-play-reviews-03

In this piece, I am not going to prove how bad these apps are. Legitimate users have saved me the hassle here by taking the time to write a review and warn anyone who was considering a download:

google-play-reviews-04

So these apps are up to precisely the type of nastiness that the FTC warned against, i.e., promising a free credit report but then having users sign up to a service to get it.

What the average user who has been tricked by these apps does not know, is that the apps are connected to one another in two very interesting ways.

Connection #1
The first connection is something you’d expect from an app/Web site discussed here: Affiliate Marketing.

Observation of the network activity behind each app [1, 2] shows us that they redirect through an affiliate link which then forwards on to the merchant that will pay the affiliate in the event of a sale: creditscorepro.com.

Note that both apps start with a GET request to a different .info domain (thegreatestever.info and thedatingconnection.info) before routing through to creditscorepro.com using the same affiliate id (AFID=278315), suggesting that the same entity is behind both apps. This is surprising because they are published under separate publishers (Sinsation & Amazing Apps Inc).

Now creditscorepro.com is not the bad guy here, they are trusting their affiliates not to get up to these types of shenanigans. You can take a look at their terms on a number of affiliate networks to verify this.

From their program on LinkConnector, note the language:

Due to FTC regulation, no “Free Credit Report” messaging

From their program on AffiliateWindow, note the language:

Due to FTC regulation, no “Free Credit Report” messaging

Also note that creditscorepro.com will pay the affiliate $26 per user that is sent their way. This is important because it is in this manner that the author of the apps is monetizing his/her efforts:

  1. Create an app and publish on the Google Play app store
  2. Market it as a means to get a free credit score
  3. Refer users of the app to creditscorepro.com through an affiliate network
  4. Creditscorepro.com pays affiliate network for each successful lead
  5. Affiliate network pays author of the app (profit!)

The affiliate behind these apps is using the ratespecial.com affiliate network, so we don’t get to see the precise terms that creditscore.com has on this network (due to them not being publicly available). It makes sense though that they have very similar terms to those on LinkConnector and AffiliateWindow.

What’s interesting on the ratespecial.com network is that their Terms and Conditions for the entire network do not accommodate affiliates using apps of this nature, in the text below they essentially limit their affiliates to the Web and Email:

5.    Restrictions. For any Engagement, subject to any greater restrictions in the applicable Accepted Offer Terms, Affiliate Network and or Publisher may promote the applicable RPS by banner advertisements, button links and/or text links (collectively hereafter the “Links”), contextual links for popup advertisements and email that is compliant with all applicable laws. Subject to the prior written and continuing approval of RateSpecial, promotional Links may contain the trade names, service marks, banners, buttons, and/or logos provided by RateSpecial on the Affiliate Network and or Publisher Portal for display on the websites used for Affiliate Network and or Publisher’s Engagement. Use of creative material that is not approved by RateSpecial will disqualify any resulting events from being “Actionable Events”. If the applicable Accepted Offer Terms says “WEB ONLY”, the foregoing materials are only allowed on the websites of Affiliate Network and or Publisher (and for avoidance of doubt cannot be used in email or in Links). If the applicable Accepted Offer Terms says “EMAIL ONLY”, Affiliate Network and or Publisher (and its agents) must limit the related promotional activities to emailing to lists, which for the avoidance of doubt are limited only to those created, managed, and treated in compliance with all applicable law. If the applicable Accepted Offer Terms says “CONTEXTUAL LINK ONLY”, then Affiliate Network and or Publisher may only promote RPS using Links, using them to direct potential customers of RPS directly to the website(s) designated in the Accepted Offer Terms.

Of course, as mentioned above we don’t get to see precisely what the Accepted Offer Terms are, but it’s probably in line with the terms of programs on other affiliate networks.

Connection #2
Two minutes into investigating apps of this nature one can’t help but wonder

“If these apps are such rubbish, then how is it that they each have thousands of downloads and so many favorable reviews?”

James Grubbs, a helpful reviewer from one of the apps above, has it all figured out:

google-play-reviews-05It just makes sense, these guys are faking their reviews.

If you take a deeper look at the reviews of each app, you’ll find that in fact there are quite a couple of reviews that are remarkably similar, but under different names:

google-play-reviews-07 google-play-reviews-06The red outline was added by me, it highlights portions of each review that are exactly the same.

After going through the awful reviews and carefully inspecting the network activity behind each of these apps, it’s obvious that they are designed to deceive. And yet in spite of this they number installs in the thousands and overall have an astounding review score of 4.6 out of 5 stars.

There’s something not quite right here, one or two fake reviews surely can’t do this.

With this in mind, I thought it would be interesting to investigate fake reviews on a larger scale in an effort answer the following question:

Are fake reviews a significant problem on the Google Play app store?

If one can’t trust the reviews of an app, then one must question the integrity of the entire store. After all, when the app store displays apps by “Top Paid”, “Top Free” and the like, it is surely including the number & nature of the reviews of each app as a factor when deciding where to rank them. It doesn’t take much to arrive to this conclusion, for there’s simply no anonymous “link” system to serve as part of a powerful ranking function as there is on the Web.

So to get to the bottom of this, I decided to first define a fake review using the two apps above as a basic frame of reference, keeping in mind that we want to minimize the chance of labeling a legitimate review as fake.

We take into account the following:

  •  The fake reviews we want to detect are those that intend to improve an app’s overall score in the store and not lower it, so we’re going to look for reviewers dishing out a minimum of a 4 star review for an app. Of course, there are scammers that will try to lower another app’s score (their competitors, in doing so they lower their ranking) but that’s not our focus today.
  • Fake reviews are going to be mostly generated by automation, and the result of something the SEO pundits call spinning: “rewriting existing articles, or parts of articles, and replacing specific words, phrases, sentences, or even entire paragraphs with any number of alternate versions to provide a slightly different variation with each spin”
  • Smaller reviews are obviously going to be very similar, if not exactly alike, to a number of other reviews. This just makes sense, just because the review “I love it” is everywhere does not mean they are all fakes. Scammers know this too, so they camouflage their reviews by keeping them short. Of course, every now and again the scammer gets creative, so instead of keeping things short, they spin long stories, which make for easier pickings when it comes to detection.

A brief overview on our method for detecting spinning is in order, for this is ultimately how we detect a fake review:

  • Break down each review into a set where each element is a word
  • Sets with less than N elements will not be considered. We have to be careful what value we choose for N here, the shorter it is then the more likely we are to increase the false positives rate. The two apps from above seem like a pretty good baseline, and it’s hard to argue that they are not fake. If you have a compelling argument as to why I’m wrong on this, please do get in touch. Otherwise, each review breaks down into 75 and 57 elements for each respective set. With this in mind, and in an effort to keep this simple enough so that someone without a degree in statistics/mathematics can easily follow along, we set our minimum review length to 50
  • We then take an intersection of each set with every other set. If the number of elements in an intersection is greater than a configurable threshold (aka their similarity in this case), then the sets are derivatives of one another and the associated reviews are marked accordingly, i.e., we have a fake review. The two reviews above are 73% similar to one another, so I chose to work with a 70% similarity in this experiment

The variables used are thus defined as follows:

  • Minimum set length (number of words in a review): 50
  • Minimum review score (score that the reviewer gave the app): 4
  • Minimum review similarity (similarity to another review in order to be considered as a fake): 70%

The Data
The data set used for our analysis includes all reviews of all apps found in the Google Play app store. On 2014-09-09 we indexed what was publicly available on the Google Play app store and found 2,719,686 reviews written by 2,084,818 reviewers of 852,137 apps.

Whilst this is great data it is by no means a 100% copy of what is in the Google store. The reason for this is because Google only exposes a fraction of the reviews for each app, you can verify this for yourself by loading up an app through Google Play on Android or via your Web Browser and trying to scroll through all of the reviews, note that only a small number of reviews are served.

Results
With a minimum set length of 50 and a review score of at least 4, we found 16,450 eligible reviews from 15,368 authors. Comparing every review to every other review, we found 172 reviews with at least 70% similarity to another review from a total of 82 authors. In this tab delimited file we tabulate the results in the form of details for each review along with the reviews that were similar to it and the similarity score.

Now consider a directed graph where edges are reviews, grey nodes are authors (labelled with the author id) and red nodes are apps (labelled with the app id that was reviewed). You can interpret the image below as “author 108446078933376544807 provided a fake review of the app with id appinventor.ai_northeastapps12.FreeCreditReport”

google-play-reviews-08

We know the review is fake because it is at least 70% similar to another review in the app store. Don’t be surprised that it is a free credit report app. Here’s the review written by author 108446078933376544807:

im a really big fan – everyone should use it its very important to know your credit score especially at this time around the holidays. im really glad i checked because it got me much better rates on a loan i applied for. i would suggest it for anyone who wants to check their scores on the go.. good thing is its sooo simple no bones about it, just gives you the report and score directly to your phone. easy to use it only took a couple minutes and i had both reports and scores. thankful that i found it

And here’s the same review of another app (appinventor.ai_GSDesign39200.BestPennyAuctions), from a different author (104192597425876737411), that was 98% similar:

fine im a really big fan – everyone should use it its very important to know your credit score especially at this time around the holidays. im really glad i checked because it got me much better rates on a loan i applied for. i would suggest it for anyone who wants to check their scores on the go.. good thing is its sooo simple no bones about it, just gives you the report and score directly to your phone. easy to use it only took a couple minutes and i had both reports and scores. thankful that i found it

So other than the word “fine” two separate authors for two separate apps had precisely 98 words and 507 characters that were exactly the same.

If you carefully examine the tab delimited file above, it’s evident that there are a number of reviews marked as fake that are not necessarily malevolent in nature, for example, when the same reviewer reviews two separate but similar applications: google-play-reviews-09

So author 114210272673494778688 reviewed the free and paid version of an app with precisely the same review:

this is an excellent memory game for young kids. it is biblically based and helps kids develop their memory and picture association. the sounds help kids remember where the animals are and can be turned off for more of a challenge.   my son (4 months) is too young to play it himself but he loved watching the animals and hearing the sounds. i would definitely recommend this game for anyone with kids!

The author is clearly not trying to hide what he is doing (100% review similarity from the same author) and the review itself seems genuine. Ordinarily one may be quick to add a condition which ignores the same reviews from the same authors, but then we’d be ignoring little clusters like these:

google-play-reviews-10The author here is simply pasting the following review verbatim for each app:

just like the content included with the main application, this add-on content is fantastic quality, incredibly realistic-sounding and certainly much better than what alternative applications are offering! whether this particular add-on is for you is personal preference, but at least with lightning bug you have a wide range of choices that sound great, work well and use very little storage space…

This author is obviously adding no value, and as we’ve shown it’s easy enough to detect this.

A quick look at the original author that kicked this project off:

google-play-reviews-11Michael Cohrs (author 105002329817833480149) did a fake review for CreditScorePro. This review was 100% similar to the review performed by Alvin Moses (104842832943339004409):

i tell my friends i would like to consider myself financially responsible so an app like this is just perfect for me. especially before making big purchases, its so important these days to know your score. that makes me so happy that i found an app on google that does exactly that, with no other stupid popup ads or anything like that to deal with. overall i guve it 4 stars

This review is also 100% similar to the review from Joyce Brothers (author 114529626562499288016) who reviewed FreeCreditScore, which includes a review from Kenda Minda (author 112746229244334005942, our original fake author) that is 96% similar:

google-play-reviews-12Easily detected rubbish reviews for near-five star rubbish apps.

Thus far the variables for defining a fake review have yielded a relatively small data set to explore, but it’s still fascinating what’s in there:

google-play-reviews-13In this cluster we have 5 authors responsible for 10 reviews of 4 apps:

  • Author 117831797660947426021’s review of com.martingamsby.orplus is 98% similar to the review from 107129133670198971212 and 89% similar to the review from 109575713192283562186
  • Author 117831797660947426021’s review of com.gobid is 100% similar to the review from 111772687247355872036 and 100% similar to the review from 102652010949230761474
  • Author 117831797660947426021’s review of com.mzdevelopment.musicsearch is 94% similar to the review from 102652010949230761474. Interesting from these two reviews is that the reviewer is merely shifting the order of words in the set (which we throw away and ignore thanks to the nature of how we detect fake reviews)
  • Author 117831797660947426021’s review of com.gappsolutions.skyman is 94% similar to the review from 102652010949230761474

Given that a cluster this interesting can be found with what is a fairly limiting constraint for eligible reviews (recall the min set length of 50), I decided to adjust the minimum set length by 20% to see if it would yield any more interesting results

  • Minimum set length (number of words in a review): 50   40
  • Minimum review score (score that the reviewer gave the app: 4
  • Minimum review similarity (similarity to another review in order to be considered as a fake): 70%

With these settings, we found 35,938 eligible reviews from 33,120 authors. Comparing every review to every other review, we found 336 reviews with at least 70% similarity to another review from a total of 161 authors. Results available in this tab delimited file.

The same cluster from above transforms into a larger cluster which now includes 10 authors responsible for 26 reviews of 11 apps.

google-play-reviews-14

The additions:

  • Author 116451052766263427395’s review of com.sndapps.zombiepiano is 100% similar to 106235084813202092445’s review of the same app
  • Author 116451052766263427395’s review of com.javi.hungry.dragon is 100% similar to 117054880620692184006’s review of the same app
  • Author 116451052766263427395’s review of com.DarkCarnival.Sharkz is 96% similar to 112114267199436446596’s review of the same app
  • Author 116451052766263427395’s review of com.tekbrix.findplus is 100% similar to 106718374237046178982’s review of the same app
  • Author 116451052766263427395’s review of com.selpheeinc.selphee is 100% similar to 109575713192283562186’s review of the same app
  • Author 116451052766263427395’s review of com.yoni.skyattack2 is 100% similar to 109575713192283562186’s review of the same app
  • Author 112114267199436446596’s review of air.com.nanico.skifleet is 100% similar to 109575713192283562186’s review of the same app

These reviews are obvious fakes: created or controlled by a single entity. At best they’re different instances from the same automation. Either way, what’s clear is that these reviews are junk. And if the reviews are junk then what does it say about the app reviewed? Moreover, what about the author? Well we know what it says about the author (automated), which really begs the question of what this says about the rest of the reviews written by these automatons (and the reviewed apps) that did not meet the conditions of this study. To get an idea of how much of a problem this is, I rebuilt the small graph above but added all reviews from the authors identified (now in orange) in the cluster above:

limited author ids - graph - bad authors to all reviews

Just answer the question!

So, are fake reviews a significant problem in the Google Play store?

Absolutely, and here’s why:

  • We ran a very simple algorithm against a fraction of the reviews in the Google Play store and yet we still found a surprising number of fake reviews, including some interesting clusters of badness. Obviously, such a simple algorithm was designed to find simple scammers, bottom of the barrel at the end of the day, really. So if we can find the simple guys with such a simple technique, who’s to say what the more capable scammers are up to? What’s arguably the most important issue here is that this confirms Google is not trying to detect & remove fake reviews, not even the simple ones
  • There’s quite a market for selling reviews. Spend ten minutes on fiverr and you’ll quickly get a feel for how much one pays for positive (and negative) reviews: $5 for 8 or $40 for 100. More enterprising publishers may consider employing the services of app2top.org who boast having sold more than 21,000 stars and 6,000 reviews. Their prices average $0.25 per review and are able to deliver up to 200 reviews per app per day

google-play-reviews-15My thanks to Ben Edelman for his thoughts on early versions of this article.

« »