FORTUNE — On Monday, New York State Attorney General Eric Schneiderman announced the conclusion of “Operation Clean Turf,” a yearlong sting that caught 19 different companies, most of them SEO (search engine optimization) or reputation management firms for hire, that were writing fake reviews for small businesses that paid them. The NYAG fined the businesses in varying amounts that total $350,000. (As part of the operation, people from the NYAG’s office even posed as owners of a Brooklyn yogurt shop.) In the press release, Schneiderman says that the investigation “tells us that we should approach online reviews with caution” and calls the process of posting fake reviews online, “the 21st century’s version of false advertising.”
Yelp’s VP of corporate communications Vince Sollitto tells Fortune that Yelp (YELP) has known about the investigation for over a year and cooperated, helping to identify for the NYAG’s office specific businesses it should target. Yelp’s line on the case is that this is great news, and indeed, it is — the operation is a warning to companies that write fake reviews, and the businesses that hire them to do so, that law enforcement is cracking down. “What we believe this particular announcement does, more than anything,” Sollitto says, “is put those businesses on notice that this is not a harmless little exercise designed to boost your ratings. It’s a deliberate attempt to mislead consumers, it violates our guidelines, but more importantly it violates the law.”
There’s a second lesson from this investigation: that the writing of fake reviews (the NYAG’s office calls the practice “astroturfing”) is getting worse, and those doing it are becoming more advanced in their attempts to circumvent Yelp’s review filter.
MORE: Yelp’s CEO ignored advice from Elon Musk and Peter Thiel and succeeded anyway.
The NYAG’s press release points out that many review sites have filters, and calls Yelp’s “the most aggressive.” For Sollitto, that’s an implied win; he believes the NYAG is noting that Yelp’s filter is “presumably the most progressive and successful.” But one criticism of Yelp over time has been a lack of transparency about the filter. “I think Yelpers understand Yelp’s review filter better than the business owners do,” says David Mihm, director of local search strategy at software company Moz. “Yelp‘s literal fortunes are tied to how many business owners will pay them for advertising, but they don’t seem to have much empathy with the set of customers that they need money from.” Mihm (as well as other critics) says there is still, “this issue of education and confusion on the part of business owners as to the steps a customer has to go through to get a review posted.”
While Yelp’s most avid users love the convenience of the service, the same cannot always be said for some on the other side of the equation. One restaurant owner in New York, for example, tells Fortune, “Imagine someone reviewing all your articles who doesn’t know anything about writing.” Yelp CEO Jeremy Stoppelman acknowledges, “There is this idea that, ‘we’re the chefs, and they’re the Yelpers.’ An ‘us vs. them’ mentality.” But, he retorts, the power is in the aggregate — a restaurant may get a few unfairly harsh marks, but with enough reviews, the score should be reliable. (Negative reviews are in the minority; 79% of reviews are three stars and up.) If a restaurant has a lot of reviews on Yelp, and the average is less than three stars, well, it probably isn’t very good.
What isn’t fair, for businesses or customers, are fake reviews. It’s a problem Yelp has sought to combat since its inception in 2004, and one that every review site with user-generated content has had to consider. Yelp has had a “review filter” since two weeks after its launch, when Stoppelman and colleagues saw the first obviously fake reviews. He says the filter is an “essential part” of the site.
So, how exactly does the filter work? “Obviously I can’t answer that question,” says Sollitto, “because there are a lot of people, as evidenced by this new investigation, who want to know exactly that so they can avoid it. What we’ve said from day one is that we take extraordinary measures to protect consumers from fake reviews. And what the media likes to talk about is businesses that aren’t happy with that. Well, we like to talk about how consumers are being protected and should be happy about it.” Talking about the technical aspects of how the filter works, Sollitto says, wouldn’t do anything to strengthen the message the company has made clear since day one: that the filter exists and is central to Yelp’s service.
Still, Mihm mentions a friend of his in Portland who owns a framing shop, J Rollins Art of Framing. She has three reviews on Yelp, plus 13 filtered reviews (it is possible to view the filtered reviews; they are not deleted entirely) and she does not understand why that’s the case. Mihm, like the NYAG’s office, believes Yelp’s filter is indeed the “more aggressive” among filters (Google and TripAdvisor, too, have review filters) but doesn’t necessarily interpret that word the same way that Sollitto does. “I think they probably do have the cleanest reviews outside of Google (GOOG), but I think a huge number of their filtered-out reviews are genuine,” says Mihm. “It seems to me that many more false positives get caught in their filter.”
More: Fortune’s 40 Under 40
Some in the industry feel that Yelp has done its policing right. Allyson Downey, cofounder and CEO of weeSpring, a baby products review site, says that Yelp influenced her own user-review company (“We describe ourselves as ‘Yelp for baby products’ because it’s the quickest way to explain what we do,” she says). WeeSpring, a TechStars company, particularly admires, “what [Yelp has] done to maintain the authenticity of their reviews, especially their consumer alerts tool, where they red-flag businesses that are caught buying reviews.” (Yelp places a consumer alert banner on pages of businesses it believes have paid for reviews, though takes it down after 90 days if the business has cleaned up its act.)
Sollitto does share that the filter weeds out 25% of reviews posted. That does not mean that 25% of reviews are fake, but that 25% of reviews are either suspect or simply less helpful. Still, it is a surprisingly large portion. And a study conducted in April by Maritz Research found that, of 3,400 people (an admittedly small sample size) who use review sites like Yelp, in most cases only half of them trust the reviews they’re seeing (59% on TripAdvisor and Zagat, 53% on Yelp). The study’s results suggest that consumer trust in user-generated reviews is eroding.
If the fake review problem persists, one potential solution would be for Yelp to tighten its requirements for posting a review. Google is moving away from anonymous reviews; you must now be logged in to Google Plus to write one. Yelp could require Facebook (FB) login to guarantee your identity, or require customers to check in to a place before they can review it. It could have the restaurant or store provide some evidence to Yelp that proves the reviewer was there. It could require you to upload a photo of yourself at the business. But of course, all of these options would run the risk of annoying, and scaring off, would-be reviewers. So much for the power of the aggregate.
