July 5th, 2001
So you thought search engines offer up neutral results? Think again
By Paddy Kamen
Is an Internet search engine more like a grocery store or a library?
While search engines provide access to vast amounts of information, they are
much more like grocery stores than libraries. Both grocery stores and search
engines offer suppliers a chance to reach you, the potential customer. Both
offer valuable "real estate" that suppliers pay for. Both provide
the public with a service. But it’s not a public service.
Consumers need to be aware of the difference.
In the grocery store, suppliers pay more to have their product on certain shelves.
It’s no accident that you find some products with 25 "product faces"
at eye level but you have to look hard for even a few "product faces"
of something else on the bottom shelf.
Search engines also offer suppliers of goods, services and information an opportunity
to connect with the public. With millions of people surfing the Net at any given
moment, every business with a Web site wants you to look at their site. And,
like getting on the right shelf in the grocery store, they all want to be listed
in the place you are most likely to see them: the first page of search results
relevant to their area of business.
Search engines provide an immensely valuable service to online businesses by
sending potential clients from around the world 24/7. And because there is much
greater demand for their indexing (or listing) and retrieval services than they
can meet (it’s estimated that, at best, only 65 per cent of sites on the Internet
are indexed at any given time) search engines are uniquely positioned to charge
for the exposure they have to offer.
It’s not that search engines or the Web sites that they index are doing anything
wrong. While the rapid commercialization of this communal space may be disillusioning
to some, the fact is that the interests of everyone involved are different.
"Search engines are a business, not a public utility," says Saman
Faroz, owner of Worldshape Inc., a Toronto-based Internet consulting firm. "Like
most Internet businesses, they’re under revenue pressures and looking for new
and creative ways to make money."
Danny Sullivan is one of the world’s foremost experts on search engine services.
As the founder and editor of Search Engine Watch (http://www.searchenginewatch.com)
Sullivan has been researching and writing on search engines from a consumer
perspective since 1996.
"In the search engine business, `monetizing the search’ is the phrase
that describes making money from the search results you present," says
Advertising was the most obvious place to begin monetizing the search and all
of the major engines now carry search-specific ads: if you enter a query for
"cereal" an ad for Corn Flakes may appear beside your search results.
But advertising is easy to identify and something we have all grown used to.
What many consumers aren’t aware of, however, is that what appear to be unbiased
search results may in fact be paid-for listings.
Each search engine has its own way of describing paid-for links. And consumers
often can’t tell the difference.
Go to Alta Vista, for example, and search for "cheese". At the top
of your results you will see three "featured links". These are paid
for. Next you will see a note that says: "We found 1,022,064 results."
Under that you will find 11 links (these are not paid for). Under that you will
see an "Extend your search" category that lists a further six sites.
Some of these are also paid for. At the bottom of the page you can click on
directories. This will take you to directory results that are powered by LookSmart.
All commercial sites in the LookSmart directory are paid.
Therefore, out of 20 links on the first Alta Vista results page for cheese,
as many as nine are paid for. This first page is the most valuable "real
estate", as most users will not explore the 1,022,044 remaining results.
"Consumers often give more credit to search engines than they should,"
says Bill Sweetman, vice-president of interactivity for Delvinia Inc., a Toronto-based
Interactive media research company. Sweetman cautions that searching is a "user
beware" activity. "While there are search engines that try their best
to find the sites most relevant to your terms of search, for others the motive
is to feed you the highest bidders in that category."
Frances West agrees. "It’s hard to know which ones are using pay-for-placement,"
says the Internet specialist with the Toronto Public Library system. "Some
will tell you that they charge for faster indexing but what’s really insidious
is when paid-for links are placed in the results."
GoTo has always been a pay-for-inclusion engine and is very upfront about its
paid listings. Enter the query "cheese" on GoTo and you’ll see the
first 27 results are paid for with the price per click paid by the Web site
right under its link. Prices for "cheese" listings range from a high
of 0.26 to a low of 0.01 cents. After item number 27, none are paid for.
Sullivan says that most users would find it difficult to identify the paid-for
links on a results page. He cites Google, AOL search and Excite as engines that
make the differences very clear.
"Those that make it hard to differentiate are AltaVista, HotBot, Lycos
and iWon. That’s because they use euphemisms like `featured sites’ or `partner
sites’ rather than the clearer `sponsored links’ or `sponsored sites’."
There are many ways in which search engines garner revenue from Web site owners
hungry for good listings. While advertising is the largest revenue source for
most search engines, other revenue generators include paid submission, paid
inclusion, paid placement and content deals.
Paid Submission: Rather than wait for the search engines to find his or her
site, the Web site owner pays to have the site evaluated. There is usually no
guarantee of a listing with paid submission.
Paid Inclusion: Normally a search engine will list only the home page of a Web
site. With paid inclusion, the engine will delve more deeply into a site and
list other pages in additional categories. This is also referred to as a "deep"
listing and likely leads to more traffic to the site in question.
Paid Placement: This is a simple pay-for-position arrangement. As in the AltaVista
example above, some of the links are placed at the top of the list and, in their
case, referred to as "featured listings". (According to AltaVista
public relations spokesperson Kristi Kaspar, "featured listings" will
soon be referred to as "sponsored listings.")
Content Deals: Ever wonder why a Barnes & Noble window always comes up with
book titles on just the subject you’re searching under when you’re in the Yahoo
search engine? That’s a content deal. As another example go to Lycos and search
under "depression". You’ll see "read about depression" under
the Popular Sites heading. The link takes you to WebMD’s content.
OTHER SERVICES: Search engines also make money by licensing their search software
to other search engine brands. According to Sullivan, GoTo’s results are available
via Netscape Search, NBCi, Lycos, iWon and HotBot. There are many similar arrangements
among the major players.
Sweetman points out that consumers may be unintentionally misled into thinking
they are searching more than one engine when they are really accessing the same
search tool under a different brand.
"If you wonder why you’re getting the same results from different search
engines, it’s because they’ve licensed the search tools from the other company."
Another revenue stream for search engine companies is the corporate market.
Michael Zahra is general manager for Yahoo Canada. He says that while advertising
is Yahoo’s largest source of revenue, it also provides portals and intranets
to large corporations and makes money from auctions, small business services,
domain registration and the sale of transactional components for online shopping.
Yahoo’s search results carry paid placement "sponsored links". They
also offer a Business Express paid submission service, which provides a fast
review of sites with possible inclusion in the main listings. Paid submission
is mandatory for commercial areas.
It’s important to keep in mind that Yahoo uses a combination of directory (human
surfed and edited index of sites) and search engine (machine crawled) links
on its results page. Google powers its search engine results.
Google is one search engine that takes great pride in the fact that they do
not manipulate their search results through payments.
"Our algorithm (or formula by which the engine searches) is fully automated
and there is no way to influence results," says Cindy McCaffrey, vice-president
of corporate communication for Google.
Google currently offers the largest index of the Web with 1.3 billion links.
Its engine crawls the Web once a month and applies a proprietary technology
that assesses the importance and relevance of a page as it relates to specific
queries. It looks at the number of other sites that link to any particular page
as well as the importance of those linking sites.
If, for example, my small freelance writing site links to the Toronto Star,
it would not deem the Toronto Star site more important. But if the Toronto Star
site had a link to my site, it would boost my site higher in the index. If all
major media outlets in Canada had links to my site, Google would rate it even
McCaffery says that Google is best placed to keep up with the phenomenal growth
of the Web because it is fully automated. "You can’t possibly keep up with
the growth when human editors are involved," she says.
While listings on Google can’t be influenced by payment, like other search
engines, it does sell paid placement in the form of "sponsored links"
that appear at the top of search results and in boxes along the right side of
the results page.
Google doesn’t appear to have the same degree of search-specific advertising
or sponsored links as some other search engines. The query "cheese"
brought up no sponsored links. Neither did the query "survivor". Third
time lucky, though: when the query "domain names" was entered many
sponsored links appeared.
Google supplements its ad revenue by licensing its search software to other
portals. It also creates site-specific search technology to enable corporations
with large sites to provide search tools to their visitors.
META SEARCH SERVICES: Also known as metacrawlers, meta search services provide
access to results from several search engines at once. These are a godsend to
the serious searcher for their obvious time-saving benefits. One might be forgiven
for also thinking that the search would yield fewer commercial results than
a single engine search.
A recent Search Engine Watch survey shows, however, that the meta search is,
in most cases, substantially commercialized.
"What people may not realize is that many meta search services are now
also bringing back paid placement listings and mixing these in among the more
traditional editorial-style results that they retrieve," writes Sullivan
in a May 23 article entitled "Meta Search or Meta Ads" on Search Engine
Of the eight meta search services reviewed for the article, only one Ñ
Vivisimo Ñ carries no results from paid placement search engines. The
percentage of paid links turning up in the results from the other meta searchers
ranged from 14 per cent to 60 per cent.
Consumers who are concerned about getting paid-for results may, in fact, be
better off using a single engine search because most meta search services do
not differentiate between paid and unpaid placement results.
Clearly there are times when consumers want to use search engines to get information
about what products are available in the marketplace or to lead them to the
things they want to buy.
If, however, you want to tap the Internet’s capacity to stimulate thought,
share creative endeavors and learn about the world, don’t lose heart. Frances
West recently revised a useful tool known as the virtual reference library (VRL).
Available through the Toronto Public Library at http://vrl.tpl.toronto.on.ca,
the VRL is a directory of links compiled by library staff.
West also recommends the Librarians Index to the Internet at: http://lii.org
put out by the California Library system.
Ideally, standards will evolve to help consumers of search engine results clearly
differentiate between links that are prominent because they are paid for and
links that are unbiased. In the meantime, just as you shouldn’t trust your grocer
to put the healthiest food near the checkout, you should approach search engine
results with some caution.