Search engines are the undisputed gateway to the internet, and Google, with a 90% share of the search market in the UK and 72% in the US, wields unprecedented economic power. Google’s own revenues last year exceeded $21 billion, but this pales next to the hundreds of billions of dollars of other companies’ revenues that Google controls indirectly through its search results and AdWord listings. Recently, it seems that Google has started quietly exercising this control in ways that can suppress competition, stifle innovation, and erode consumer choice.
Google has always used various penalty filters to remove certain sites entirely from its search results or place them so far down the rankings that they will never be found.
Figure 1: Google search penalties
Typically, Google search penalties take a result’s "natural" ranking position and artificially lower it by some factor (depending on the severity of the penalty). Usually, the severity of a penalty is of little consequence, as any result placed below the first three pages is effectively invisible to users.
Whereas these penalties used to be reserved for spam, or sites caught attempting to cheat Google’s algorithms, they are now increasingly targeted at perfectly legitimate vertical search and directory services. It may not be coincidence that, collectively, these services present a nascent competitive threat to Google’s share of online advertising revenues.
Google grants certain sites immunity to these algorithmic penalties through ‘whitelisting’ (a.k.a. ‘manual lift’). But, these manual interventions are cloaked in secrecy and seem to be reserved mainly for high profile sites that would be conspicuous if absent from Google’s search results.
These vertical-search-targeted penalties, coupled with a manual whitelisting process that seems heavily biased towards established brands, are a significant barrier to new entrants and inevitably suppress innovation.
The experience of Foundem illustrates the innovation-suppressing effect of these penalties.
Founded in 2005, Foundem is a UK-based technology company specialising in Vertical Search. Vertical search is different from and complementary to keyword-based, horizontal search: for example, whereas a traditional horizontal search engine like Google can return a list of sites that sell flights, an appropriate vertical search engine like Foundem or Kayak can go deeper into those sites to return details of the actual flights.
Because vertical search requires an “understanding” of a vertical, and because each vertical is different, vertical search services tend to be highly specialised, focusing on just one or perhaps a small handful of verticals. Before Foundem, there was effectively a technology barrier that prevented horizontal search engines from going deeper and vertical search engines from going broader.
Figure 2: The Search Technology Barrier
This barrier prevents horizontal search engines from going deep and vertical search engines from going broad. Foundem’s unique WebSentient technology allows it to overcome the barrier to go both deep and broad.
Foundem’s patented WebSentient™ technology may be the world’s first and only universal vertical search platform, enabling Foundem to deliver class-leading vertical search services to virtually any vertical with just a straightforward configuration. Foundem is probably already the world’s broadest vertical search service, covering product price comparison, travel, jobs, and property, and it can expand rapidly into myriad additional verticals, including many that lie beyond the reach of traditional services.
In December last year The Gadget Show, the UK’s leading technology television programme, named Foundem the UK’s best price comparison site [Video Clip].
Figure 3: The Gadget Show recommends Foundem (December 2008)
And, in October 2009, Which?, the UK’s leading consumer body, surveyed the UK’s twelve leading flight search sites and ranked Foundem third—five places above Kayak (a dedicated travel search service, and one of the leading lights of the US Vertical Search industry).
Figure 4: From Holiday Which? October 2009
Foundem also powers state of the art vertical search and price comparison services for many of the UK’s leading media companies, including Bauer, IPC Media, and Future.
Figure 5: Powered by Foundem Examples
Despite Foundem’s strong credentials and proven track record of innovation, since June 2006 it has been suffering from a Google penalty that systematically excludes all of Foundem’s content from Google’s search results.
Figure 6: Illustrating the effect of Foundem’s Google search penalty on all of Foundem’s price comparison result pages (e.g. “compare prices shoei xr-1000”)
Before the penalty, Foundem featured and ranked normally in Google searches, as it continues to do in Yahoo and Bing.
Figure 7: Example – Foundem continues to rank normally in Yahoo and Bing
For the query “compare prices shoei xr-1000”, Foundem is one of only two or three truly relevant pages. For this query, Foundem ranks #1 in Yahoo, #7 in Bing, and #144 in Google. This broad pattern is repeated for all of Foundem’s price comparison results.
As the example illustrated in Figure 7 shows, for the query “compare prices shoe xr-1000”, Foundem currently ranks #1 in Yahoo, #7 in Bing, and #144 in Google. To put this in perspective, this is an example of a highly targeted query aimed specifically at a price comparison for a particular motorcycle helmet, and Foundem’s page is one of only one or two available pages that are actually relevant to the query (and, until very recently, Foundem was the only relevant result). Yet, the Google penalty ignores this relevance and artificially drops Foundem’s result to page 14, well beyond the reach of any Google users.
This is not an isolated example, this same pattern is repeated for all of Foundem’s tens of thousands of product price comparison pages.
Just weeks after Foundem’s Search penalty began, Foundem was also struck by an equally sudden and severe AdWord penalty (AdWords are the ‘sponsored links’ above and to the right of Google’s search results, paid for by advertisers on a per click basis).
Google assigns automated ‘Quality Scores’ to its advertisers’ landing pages (the page you land on when you click). The scores are meant to reflect the page’s quality and relevance to the keywords being bid for. Foundem’s AdWord penalty meant that, overnight, Google artificially lowered all of Foundem’s Quality Scores from a typical rating of “excellent” (10/10) to a typical rating of “very poor” (1/10). This instantly raised all of Foundem’s costs per click (minimum bids) from around 5p to around £5. Foundem was now being asked to pay a staggering and prohibitive 10,000% more than its unpenalised competitors.
After a protracted series of appeals, Foundem’s case was eventually escalated to a senior AdWord evangelist who explained that Foundem’s Quality Scores had been automatically downgraded by a new algorithm designed to detect and penalise vertical search services such as price comparison and travel search. In other words, Google had systematically and artificially lowered Foundem’s Quality Scores, not because of any quality or relevance issues, but purely because Foundem had been identified as a vertical search engine.
The same AdWord evangelist then championed Foundem through Google’s “whitelisting” process and, six weeks later, Google granted Foundem immunity from its AdWord penalty by manually “whitelisting” the site names ‘www.foundem.co.uk’ and ‘www.foundem.com’. Within hours, all of Foundem’s Quality Scores and costs per click returned to normal.
Figure 8: Illustrating the Effect of Foundem’s AdWord Penalty and Eventual Manual "Whitelisting"
It is important to note that the sudden dramatic changes to Foundem’s Quality Scores were entirely unrelated to the underlying quality or relevance of Foundem’s pages: the pages did not change before the sudden drop from 10/10 to 1/10, and did they did not change before the sudden increase from 1/10 back to 10/10. These quality score changes were purely the result of a penalty being applied and then lifted. It is also important to note that once penalised in this way, a site’s only means of escape is by manual intervention (e.g. “whitelisting”) by Google.
One of the reasons it is so difficult for legitimate sites struck by these algorithmic penalties to obtain relief is Google’s reluctance to acknowledge publicly, or even internally, any kind of manual override of its algorithms. In September 2007, Google confirmed Foundem’s AdWords whitelisting in writing. Yet, as recently as September 2008, Google seemed to conceal the existence and crucial role of these penalties and their manual remedies: “Google told me that it never made judgments of what was ‘good’ and ‘bad’ because it was all in the hands of the algorithm” (Joe Nocera, Stuck in Google’s Doghouse, New York Times, September 12 2008).
As described above, Foundem’s strong and growing list of credentials now includes being evaluated and confirmed as high quality by Google’s own AdWord Quality Team.
Moreover, Google has now confirmed that Foundem’s Search and AdWord penalties were both triggered by the same penalty algorithm. It is therefore clear that Foundem’s search penalty, like its AdWord penalty, has everything to do with Foundem’s role as a vertical search engine, and nothing to do with the quality of its service or pages. It is equally clear that Foundem’s only means of escaping its search penalty is for Google to manually intervene and grant Foundem immunity from it, just as it has already done for its AdWord penalty.
Yet, despite all of this, all of Foundem’s numerous formal and informal attempts to obtain manual “whitelisting” for the Search equivalent of its AdWord penalty have failed. Three years on, Foundem remains unjustifiably penalised in Google’s search results.
The only significant change to Foundem’s Google search penalty in the three years since it began occured just days after journalists from the Guardian and the BBC began to question Google about Foundem’s case. On 14 July 2009, Foundem’s home page suddenly emerged from the penalty and started ranking normally again for certain phrases, such as “price comparison”:
Figure 9: Illustrating Foundem’s Home Page ranking for relevant keywords, such as "price comparison"
Sadly, this change is entirely superficial. All of Foundem’s actual content—its tens of thousands of price comparison pages—remain penalised beyond the reach of Google’s users.
Figure 10: Illustrating Foundem’s ranking for all of its content pages, for even very specific keywords such as “compare prices canon eos 7d”
The only material effect of this change, therefore, was to muddy the waters: where Foundem’s search penalty had been easy to describe (as a “complete exclusion”), it is now more complicated.
Google’s overwhelming dominance of search means that no one can doubt the immediate and substantial economic impact of a Google penalty. Even Amazon, one of the web’s strongest and most established brands, would be placed in dire financial jeopardy were Google to exclude Amazon’s pages from its search results, or even just artificially bump them down a few tens of places. Imagine, then, the impact of these penalties on fledgling businesses that have yet to gain significant brand recognition. Any policy that, inadvertently or otherwise, whitelists established sites while leaving emerging sites penalized inevitably suppresses innovation.
Remarkably, Google exercises this immense power without oversight of any kind.
Any attempt by Google to justify or rationalise these penalties is significantly undermined by Google’s growing emphasis on its own vertical search services. At the same time as Google is quietly penalising third-party vertical search services, it is continuing to create its own competing services and giving them increasingly preferential placement in its search results. Since May 2007, for example, Google has been placing its own price comparison results in prime positions for all product-related searches.
Users expect Google to try to return the most relevant results to their queries, while filtering out any obvious Spam. Users do not expect Google to deliberately filter out high quality sites, irrespective of relevance; yet, with the introduction of these vertical search penalties, this is precisely what Google is now doing.
Google is performing a precarious balancing act between the enduring public perception of it as comprehensive and impartial and the reality that it is no longer either. In the court of public opinion, Google continues to nurture the appearance of absolute automated objectivity, while, in the court of law, it now vehemently asserts and defends its right to manually and subjectively promote, penalise, or omit whatever it chooses.
Every time Google ranks its search results, it is by definition expressing an opinion. There is a commonly held misconception that, in court, Google is merely defending the right of its algorithms to autonomously express this opinion. But this is not the case. When Google’s opinion lay entirely in the hands of an algorithm that acted in a uniform and methodical way and relied solely on information it found for itself, it was easy to argue that its opinion was fair and objective. With the introduction of vertical search penalties, however, Google has become increasingly reliant on manual intervention to ensure that high-profile sites are not visibly penalised. By introducing special treatment for particular site names manually fed to the algorithm (such as ‘whitelists’), objectivity is lost, and the opinion becomes undeniably subjective.
In court filings, Google confirms such manual intervention, stating categorically that PageRank—one of the factors used to rank Google’s search results—is manually manipulated: “Even if PageRank were entirely determined by an algorithm, which…it is not,…PageRank is no less Google’s expression of opinion about the ‘appeal, popularity and relevance’ of a website than a movie review using a scale of: ‘I hated it’ … or ‘I loved it’” [Attorneys for Google Inc., June 16 2006].
Remarkably, at the same time as Google’s attorneys were arguing in court that "PageRank constitutes Google’s subjective opinion concerning the relative importance of a website…", the information on Google’s website directly contradicted this: "PageRank performs an objective measurement of the importance of web pages", [Source]. This page was eventually altered in May 2007 in a way that accommodates Google’s manual manipulations: "PageRank reflects our view of the importance of web pages" [Source].
Outside of the court room it seems that Google is not quite so comfortable with its shift away from automated impartiality toward manual bias; subtle changes to the text on Google’s website reveal a company struggling to come to terms with this new, and less palatable, reality.
Until recently, Google’s online explanation of its search results read: "A site’s ranking in Google’s search results is automatically determined by computer algorithms…", but this was altered in May 2007 to read "A site’s ranking in Google’s search results relies heavily on computer algorithms …" [Source].
Similarly, where Google used to reassure us that "there is no human involvement or manipulation of results…", [Source], it now offers the considerably less reassuring "We have always taken a pragmatic approach to help improve search quality" [Source].
Even now, Google still regularly denies all manual intervention:
"At Google we do not manually change results.” (Udi Manber, Google VP in Charge of Search Quality, April 16, 2008) [Source].
"Our third philosophy: no manual intervention…The final ordering of the results is decided by our algorithms…, not manually by us. We believe that the subjective judgment of any individual is…subjective, and information distilled by our algorithms…is better than individual subjectivity." (Amit Singhal, Google Fellow in Charge of the Ranking Team, 7 September, 2008) [Source].
Foundem’s case illustrates that there is a pressing need to re-establish the level playing field on which the web was founded, so that innovative sites and services can once again compete on utility and relevance, rather than on the whims of an all-powerful search engine that seems increasingly intent on maintaining the status quo.
Foundem is being penalised simply because it is a vertical search engine—a clear example of Google, intentionally or otherwise, acting to suppress innovation in a field viewed by many as the natural next step in Search.
Because Google’s new penalties are targeted at perfectly legitimate services, they rely on a lack of transparency and openness for their very existence. Unlike previous Google penalties, which were aimed at spam and sites caught attempting to cheat Google’s algorithms, these new penalties are difficult to defend publicly. Instead Google conceals, confuses, and obfuscates; keeping the debate focused firmly on the “if” a site is penalised, rather than on the “why”.
Google’s overwhelming dominance of the UK and US Search markets means it probably isn’t allowed to behave like this. Certainly, EU competition law is clear: dominant organisations are not allowed to act purely in their own self interest.
In the short term, Google needs to manually lift Foundem’s search penalty, just as it has already done for Foundem’s AdWord equivalent of this penalty. Google also needs to introduce an effective, accessible, and transparent appeal process by which all inappropriately penalised sites can obtain whitelisting. There should also be an open and informed discussion about the rationale behind these sweeping and highly questionable penalties.
If Google is allowed to continue on its present course unchecked, it will have far-reaching consequences for the web’s future development. In the words of Google’s CEO, Eric Schmidt, “Incumbents very seldom invent the future”.
[First Published: 18 August 2009]
[Significantly Updated: 14 October 2009]
[Added Link to Foundem’s Penalty Update: 30 December 2009]