Black Box Search vs. Black Hat Publicity Hounds

Frank Pasquale

J.C. Penney was on top of the web world last holiday season, showing up at #1 for dozens of retail search queries on Google. Type "dresses," "area rugs," "bedding:" you'd get Penney's items as your first search result. Had the venerable retailer become a "Wikipedia" of online shopping, reliably providing the "people's choice?" Or was something more troubling going on?

The New York Times has related an excellent study (which I had worried was impossible to do) examining the issue. The study "suggests that the digital age’s most mundane act, the Google search, often represents layer upon layer of intrigue." For J.C. Penney was using "black hat" search engine optimization, a definite no-no in search engine land.

The possibility of black hat SEO is old news, recognized as far back as Helen Nissenbaum and Lucas Introna's 1999 article on the politics of search engines and Alejandro Diaz's thesis "Through the Google Goggles." Reporter David Segal pushes the ball forward by asking about the timing of crackdowns on black hatters:

Why did Google fail to catch a campaign [of result manipulation by J.C. Penney] that had been under way for months? One, no less, that benefited a company that Google had already taken action against three times? And one that relied on a collection of Web sites that were not exactly hiding their spamminess?

Mr. Cutts emphasized that there are 200 million domain names and a mere 24,000 employees at Google. “Spammers never stop,” he said. Battling those spammers is a never-ending job, and one that he believes Google keeps getting better and better at.

Here’s another hypothesis, this one for the conspiracy-minded. Last year, Advertising Age obtained a Google document that listed some of its largest advertisers, including AT&T, eBay and yes, J. C. Penney. The company, this document said, spent $2.46 million a month on paid Google search ads — the kind you see next to organic results. Is it possible that Google was willing to countenance an extensive black-hat campaign because it helped one of its larger advertisers? It’s the sort of question that European Union officials are now studying in an investigation of possible antitrust abuses by Google.

So now J.C. Penney can only look back with regret on its once-dominant position. I am not convinced by "conspiracy theories" here, just as I was not upset by Google "bias" in responding to the "miserable failure" presidential googlebombs. But I think the article may be too sanguine about Google's vaunted "wall of separation" between ad workers and search workers.

First, who audits the wall of separation? What measures are taken to avoid reciprocal influence? Do we have anything beyond the company's word to verify "Google’s commitment to separating the money side of the business from the search side?"

Sure, there is a case for keeping search and ad algorithms secret. Secrecy is common in Silicon Valley; as Tim Wu notes, "Google, . . . despite its commitment to network openness, keeps most of its code and operations secret, and today’s Apple, unlike the Apple of 1976, guards technical and managerial information the way Willy Wonka guarded candy recipes." In general, companies have a right to choose between patents (open for all to understand but not subject to copying during their term) and trade secrets (closed but free to be imitated once independently invented) to protect their innovations.

However, we may have to rethink the social value of secrecy in certain critical industries. Ordinarily, confidentiality does not matter much to the larger economy. The state can let Wonka keep his new recipe for chocolate secret, secure in the knowledge that customers can find other alternatives if the price is too high. But what happens when secrecy affects the critical intermediaries that help consumers determine whether the price is too high, or what product best meets their needs?

The deregulatory trend of the past few decades was premised on the idea that that same basic economic reasoning that applied to sales of goods like chocolate could also guide the finance and high technology fields. But in the information economy, finance firms determine what is funded, and dominant internet firms heavily influence what is found online. They are often dominant private powers, effectively establishing a "Googlement" (or "Government Sachs").

We need to understand far more about the material bases of their decisions than we currently do. As Joseph Turow has demonstrated, there has been an explosive growth in intermediaries between advertisers and publishers online. I've questioned the emerging economic incentives in a series of works on search engines.

The NYT article claims that Google "draws a pretty thick line between techniques it considers deceptive and “white hat” approaches, which are offered by hundreds of consulting firms and are legitimate ways to increase a site’s visibility." But what about certain ambiguous practices? I will explore that question in my next post, as webmasters reel from the latest algorithm changes. For now, I highly recommend Ira Basen's radio documentary "Engineering Search" for those interested in the shadowy world of search engine optimization.

X-Posted: Madisonian.
Posted 5:46 PM by Frank Pasquale [link]