Do Promotional Reviews Skew the Data Set?

Jun 22, 2022
Do Promotional Reviews Skew the Data Set?

As more consumers look to reviews to inform their purchase decisions, and more brands are rewarded for having reviews, many have turned to promotional reviews.

Promotional – sometimes called incentivized reviews – happen when a consumer receives a free product, a discounted product or discount product code for future purchases, or another incentive for reviewing the product. As noted in our previous blog on the authenticity of reviews, promotional reviews are not to be confused with counterfeit or “purchased” reviews, where a consumer is paid money to give a review a positive product, regardless of their true experience. As we noted, this may help in the short term, but within even 8 weeks, products with counterfeit reviews see a drop in star rating, search ranking and sales.

Rather, promotional reviews are still authentic reviews by users. There is no reward for leaving a good review and no penalization for a bad review. What’s more, promotional reviews must be disclosed.

The FTC has an Endorsement Guide for reviewers/influencers as well as brands and manufacturers to follow when leaving or soliciting reviews. The guide applies to eComm sites as well as YouTube reviews, and blogs and social media.

The guide boils down to this: Any money off a product must be disclosed in the review. This includes free products for reviews (promotional or incentivized reviews). The standard is: if other consumers know about the connection, would that information affect how much weight they give to the review?

We looked at a number of promotional review programs across various sites and various categories to see if promotional reviews had any nominal impact on ratings and reviews.

The Three Types of Promotional Reviews we Examined:

Amazon’s Vine Program: This is an invitation-only program. Customers who “consistently write helpful reviews” are invited by Amazon. They receive products by sellers participating in the program and are encouraged to provide “honest and unbiased reviews”. No money is exchanged. The program limits the # of reviews to 30 reviews per ASIN.

Influenster’s Vox Box: Users sign up for the VoxBox campaign and, if selected, receive a box free products from brands to review.

Website/DTC promotional Reviews: Many brands offer free products and discounts for honest reviews on their brand website.

We also looked across various categories: Personal Care, Oral Care, Skincare, Laundry, Food & Beverage, Baby Consumable, Cosmetics, Household Cleaners and OTC Meds; and at > 10,000 reviews, dated from 2019 – 2021 from Sephora, Amazon, Influenster, and various DTC sites.

What we Found:

In total, promotional reviews have a very slight, though not statistically significant, effect on overall star rating: 4.25 compared to 4.19 for unsolicited reviews. DTC sites are the only site where there is a significant difference in star rating between solicited and unsolicited reviews, which may be impacted by manufacturers filtering reviews on site given they can control the platform.

Average Star Rating Comparison by Channel Average Star Rating Comparison by Category

Influenster’s Vox Box, which primarily focuses on beauty brands, had unsolicited reviews higher than promotional reviews, at 4.55 pts compared to 4.31 pts.

Sephora’s Data showed non-promotional reviews at 4.55 pts compared to 4.59 pts for promotional reviews.

Amazon’s Vine program saw a similar breakdown, with promotional reviews having an average of 4.26 pts compared to unsolicited reviews at 4.12 pts. Here, because Vine makes up such a small amount of overall reviews, the total for all products was 4.14 pts.

The most influence in terms of incentivized reviews were on brand websites, which were over a 1 star higher for promotional reviews when compared to its unsolicited reviews.

Being unable to see the type of solicitation message brands include with the incentivized reviews, we are unable to have a clear hypothesis as to why this is. It may be that consumers rarely review products on a site unless asked. Some brands may choose to limit the number of 1 & 2 star reviews they receive. As a rule, 4Sight does not extract reviews from DTC sites unless it is necessary.

How Many Promotional Reviews in Data by Channel

We also wanted to see how many promotional reviews were in the data for each channel.

Promotional V. Unsolicited Review Numbers by Channel

Notice that only Influenster has more promotional reviews than unsolicited reviews (particularly interesting as they’re promotional reviews were a lower star rating than average).

Aside from DTC and website review data, it’s clear that using promotional reviews in any analysis on the category – for consumer wants and needs, what’s driving the category, any whitespace, etc. – doesn’t impact the outcomes of the analysis one way or another. Consumers asked to provide a review in exchange for a product or discount had the same experience as reviewers without the incentive. When it comes to DTC sites, one needs to be a little more cautious in analyzing reviews for insights to ensure that the sample size is not biased toward promotional reviews that skew the data.

For non-DTC site reviews, there doesn’t seem to be any compelling reason to remove these reviews from a data set, but we were curious if there was any reason to leave these reviews in the data set.

We hypothesized that reviewers participating in these programs felt more obligated to leave thorough reviews. To determine if this was true, we calculated the average number of words per review.

Average Word Count Per Review by Channel

The results were a bit all over the place. For Website/DTC reviews, the numbers were exactly the same, and Influenster unsolicited reviews had slightly higher word count. Both Sephora and Amazon had significantly higher word count for their promotional reviews, with Amazon’s Vine program reviewers leaving 3Xs the amount of words per review.

On the whole, this suggests to us that promotional reviews may have a small effect on star rating, but might also be worthwhile to include in an analysis, as they will likely provide deep, thorough insights. It’s clear that incentivized reviewers treat the task very seriously, want to give an honest opinion, and want to help inform others in their purchase decisions.

Top