[TAG] Search Engjne Spiders

[Benjamin A. Okopnik]

On Thu, Apr 20, 2006 at 04:46:45PM -0700, bob van der Poel wrote:
> 
> Any light on how search engines like google work? I've recently moved my 
> web stuff to the free site provided by my new ISP. For some reason 
> google refuses to list any of my "really good stuff". And, doing some 
> searches I don't think that any other (or very few) pages on this site 
> are being found. My page http://users.xplornet.com/~bvdp has a fairly 
> unique pattern I can test "wynndel broadband xplornet". So far the only 
> hits are back to my OLD web pages which announce a move. BTW, those 
> pages are gone.
> 
> I've discussed this with Xplornet and they have come to the conclusion 
> that their site is "sandboxed", but don't know why, etc. And, frankly, 
> I'm not sure they care much either :)
> 
> I have tried to "seed" things by filling out the google form (sorry, 
> forget the exact spot right now).

Well, there's a surefire way to find out if Google knows about you:

```
ben at Fenrir:~$ google http://users.xplornet.com/~bvdp
Sorry, no information is available for the URL users.xplornet.com/~bvdp

  ? If the URL is valid, try visiting that web page by clicking on the following link: users.xplornet.com/~bvdp
  ? Find web pages that contain the term "users.xplornet.com/~bvdp"
'''

I guess it doesn't. I'm assuming you went to
http://www.google.com/addurl.html to add yourself - yes? And you made
sure to put in a list of keywords that are relevant to your site - yes?
I've always found this to work just fine for my clients, although it
does take up to a couple of weeks (and it's amazing how impatient people
can get in the meantime. :)

> I find the whole issue pretty interesting, especially since in the past 
> when I've created new pages, etc. they have been found by google within 
> hours. Days at the most. These new pages have been up for about a month 
> now and not a hint from google.

So, try submitting to a bunch of other engines in the meantime. They all
"steal" from each other, AFAIK - and the more places there are on the
Web that refer to your site, the higher the chance that Google will run
across it sooner.


* Ben Okopnik * Editor-in-Chief, Linux Gazette * http://linuxgazette.net *