After going back and reviewing the links given me by Max at Yahoo, I may have figured out a reason the site was dropped from the index. So thanks for the info Max - it may have proved helpful after all. This is from Yahoo’s site:

To see if your site is indexed but is not showing up in search results for a particular query, search for your site using the title and an identifiable unique phrase from the text on your page.
If you want a page to appear in web search results, make sure that page includes some unique text to be indexed. Pages with no unique text or no text at all may drop out of the index or may never be indexed. Also, refer to the Site Guidelines to see whether your site is in compliance with Yahoo’s content policies. Sites that violate the Yahoo! Site Guidelines may be removed from the index.

Lastly, it is possible that your site was unavailable for crawling during an index refresh, and it could re-appear following another index update.

Well I know the site passed the first two tests. After all, it WAS indexed. Next, Max says that the site doesn’t seem to be in violation…which leads me to the last statement on the page: “it is possible that your site was unavailable for crawling during an index refresh”. I happen to know that my site was up because I host it myself and there were no server problems or service restarts lately. Of course there’s the chance that there was a network issue somewhere between Yahoo and my site that caused interference, but that’s also unlikely.

HOWEVER, what I just remembered is that out of all the pages I had indexed by Yahoo there were a few that had changed locations due to restructuring on my site. I use custom error pages, so it’s not a big deal for visitors, but now that I think of it–the spider probably sees those as the 404 error page they are, and not as the pretty page my visitors see. Unfortunately, Yahoo doesn’t happen to mention what they would do in such a situation but my guess is that if the first few links the spider tested were 404’s then maybe it gave up and delisted the entire site without continuing to test the other pages which had been previously indexed. I have no idea how realistic this theory is, but I think it’s my strongest one so far…

This entry was posted on Monday, September 26th, 2005 at 12:34 pm and is filed under Business. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.