Permission-Based Indexing

A Solution to the Supplemental Syndrome

Purpose and Intent

The purpose of this article is threefold:

  1. To demonstrate how Google can use its existing technology to be able to successfully determine which sites should and should not be fully indexed and crawled, thus benefiting both Google and the webmasters who wish to and should be included;
  2. To allow webmasters the opportunity to comment and provide additional thoughts. The hyperlink to this article will be posted to Google engineer Matt Cutts’ blog as well as the major social bookmarking sites (e.g. http://www.digg.com/) in order to gather as much feedback as possible from webmasters and end users alike;
  3. To provide a means for other search engines to increase the quality of their engines by implementing the solution in this article, and therefore increase the quality of those engines as well.

This article is not intended to assume an adversarial position, as I am a supporter of what myself and many others perceive to be the intent of the BigDaddy update — increasing quality of the engine by removing links that are created for the sheer purpose of search engine manipulation and therefore bear no relevance or value to the end user.

This article is intended to demonstrate how webmasters and search engines can work together and cooperate in a positive manner that will mutually benefit webmasters, search engines, and most importantly the end user. By allowing webmasters increased control and by communicating with webmasters as outlined in this article, increased search engine quality and relevancy will result.

Background

The Google search engine constantly strives to improve its engine by indexing as much quality content as possible, while eliminating as many sites that are there for the sole purposes of manipulating and spamming the engine as it can.

On March 29, 2006, Google launched its BigDaddy update, and it appeared to remove many of the preexisting spam and manipulation attempts and was a drastic and much-needed wakeup call to many webmasters who had elected to participate in link co-operatives, reciprocal linking schemes, and other such tactics designed to manipulate the engine. For some examples of the types of things Google is attempting to remove, please visit Google engineer Matt Cutts’ blog post concerning the BigDaddy indexing timeline.

The Unfortunate Side Effect: Good Sites Suffer

The BigDaddy update had the unfortunate side effect of partially or fully delisting pages from many sites with legitimate purposes, sites that did nothing specifically to manipulate the engine. In particular, one woman named Nancy posted to Matt’s blog. Nancy owns a rather unique and comprehensive fan site devoted to Eagles band member Glenn Frey. Her site is very clean, coded well, and contains a great deal of useful information and content concerning Frey’s career both as a member of the Eagles and as a solo musician. It can easily and quickly be determined that this is a site that deserves to be in any search engine or directory that Nancy wishes to have her site included in.

The reason the Glenn Frey site not to be included appears to relate to one of the reasons Matt Cutts discussed in his timeline — a relative lack of inbound links to her site. Cutts discusses inbound links as a factor when sites are indexed; a lack of inbound links to a page or site may lead to partial or no indexing of said page or site. This has led to a number of complaints from webmasters who believe that the number and quality of inbound links is in no way an indicator of the quality or usefulness of a page, and whether or not that page should be indexed.

06/12/2006
Issues with Inbound Links –>
This website is for sale!