Following the devastating fire in the Notre Dame Cathedral, Ubisoft pledged €500,000 to aid in its restoration and made Assassin's Creed Unity—set in Paris during the French Revolution—free for a week. Shortly after that, hundreds of positive reviews flooded into the game on Steam. It was, as far as I know, the first high-profile instance of "reverse review bombing," in which users post "off-topic" reviews to artificially inflate a game's score.
It seems clear enough that this is the case: Unity's overall user review rating on Steam is "mixed," while recent reviews are "mostly positive." But was it actually a review bombing at all? And if so, how does it fit with the actions Valve has recently taken to lessen the effects of negative review bombs? I'll be honest, it's not something I thought about, but as seen in this lengthy new meditation on the topic, Valve really has.
Valve said the Unity situation doesn't quite match the pattern of negative review bombs because the increase in reviews was accompanied by an increase in players, which "isn't necessarily the case with a typical off-topic review bomb." More players generally means more user reviews, so from that perspective it appears more like a game that's gone on sale or received a major update, which has boosted player numbers and thus reviews.
Reading the reviews leads to a similar conclusion. "Some reference Notre Dame or the giveaway. But most just look like standard reviews of a new player, or a player that's returning to a product they bought a while ago," Valve wrote. "Ubisoft has released significant updates to AC:Unity since launch, and it appears that some players who bounced off it at launch have returned, and found themselves enjoying the game more."
Because of that, the surge in positive reviews is not a by-definition review bomb, according to Valve: "It certainly doesn't fit our original definition in the 'aimed at lowering the Review Score' section, but back in 2017 the community's terminology around 'review bombs' was also focused only on concerted negative efforts. It'd be nice to change that terminology to something that doesn't imply positive or negative, but that's really up to the community."
That uncertainty led to discussion about whether the Unity reviews are "off-topic," which is how Valve determines whether or not they'll be removed from the overall review score calculation, and contemplation of "context changes," which may or may not be relevant to "general" Steam customers looking for a new game. Overall it seems like a lot of effort to be poured into a relatively minor issue—Valve said marking the new reviews as off-topic would decrease the overall review score by 1.3 percent, which would have zero impact on its fortunes—but it is interesting to see how deep this particular rabbit-hole can go:
"In this case, the Notre Dame tragedy has made it so that AC:Unity happens to now include the world's best virtual recreation of the undamaged monument. That's a context change that could be increasing the value players are getting from the game, so perhaps the game really is better than it was before? Or maybe that's unrelated, and it's actually players feeling good about Ubisoft's significant donation to rebuilding the monument? Irrespective of the reason, perhaps this is a short-term temporal effect? Should temporal effects even be included in Review Scores? If a game was heavily focused on a time of the year, like Christmas, we suspect we'd see it have Review Score fluctuations around Christmas-time, as more people bought it and thought it was better on average than people who bought it at other times of the year.
"If visiting the virtual Notre Dame is a reason players have reviewing the game more positively, we'd expect the Review Score to continue to reflect it in the future, albeit at a lower volume. But that's still the case even if it's not the reason—the future Review Score would revert to where it was prior to this event."
All of that leads to this final conclusion: "We're not really sure what to do here." The influx of new Unity reviews doesn't really look like a review bomb based on the definition established a new months ago and even if it was, it might not be off-topic anyway and marking them as such wouldn't materially impact the review score. And thus, the final decision was made to just leave it all alone.
"But hopefully, this post has helped you understand that thinking behind why we've ended up there," Valve wrote. "If you have any thoughts on how we should approach this case, or similar cases in the future, we'd love to hear from you in the comments below."
Steam users have taken up that invitation, and unsurprisingly the reaction is largely negative: Despite Valve's deep-dive on its reasoning, a large number of commenters believe the company is being "hypocritical," to use one relatively polite descriptor, by burying negative off-topic reviews while allowing those that are positive (and therefore presumably a boon to sales) to stand.
There's also an interesting bit in the two-part FAQ that also provides a little insight into how Steam's internal algorithms work. Explaining that Unity's visibility on the storefront wouldn't be affected if the reviews were marked as off-topic, Valve said review scores come into play when sorting games into lists in different places in the store.
"Games receive a 'boost' based upon which user review bucket they're in (Mixed, Mostly Positive, etc). The actual boost amount is quite small relative to other factors in the Store, and it's essentially the same for all the Mixed or above review buckets," Valve explained. "But there's a big boost drop-off as soon as a game drops below Mixed into the negative buckets, which occurs at the point where less than 40% of the user reviews are positive. A Mixed game receives over 500% more boost than a game in Mostly Negative. That might seem scary, but we're still talking about a boost that's small relative to many other store factors, and it's the minority case - 71.7% of titles on the Store are Mixed or above."
"In the case of AC:Unity, the positive review spike looks significant in the Recent Reviews view, but in the overall Review Score it only shifted from 59.7% up to 61%, both of which are squarely in the Mixed reviews bucket."
As an aside, Assassin's Creed Unity is probably not "the world's best virtual recreation" of pre-fire Notre Dame: That honor likely goes to the late art historian Andrew Tallon, who used lasers to create a detailed scan of the structure with more that one billion data points.
