[TAG] Search Engjne Spiders

[Thomas Adam]

 --- bob van der Poel <bvdp at xplornet.com> wrote: 
 
> Any light on how search engines like google work? I've recently moved my 
> web stuff to the free site provided by my new ISP. For some reason 
> google refuses to list any of my "really good stuff". And, doing some 

Probably then, the google site is listed in 'robots.txt' on the
webserver as not being valid when sending out bots to spider your site
(or a subdomain).   It's a common practise to ban most harvesting bots,
and to allow only a few.   

Google work by sending out spiders -- programs that essentially use
referral-following (URL-hopping if you like) to see what's what.  The
greater linked your site is, the greater the chance you'll get
'spidered'.  (I am sure this is explained on google's site.  Go and
look).

> searches I don't think that any other (or very few) pages on this site 
> are being found. My page http://users.xplornet.com/~bvdp has a fairly 
> unique pattern I can test "wynndel broadband xplornet". So far the only 
> hits are back to my OLD web pages which announce a move. BTW, those 
> pages are gone.

So?  Even if your site is hit by google (or some other indexing
surveyor) that's still no guarantee your search results will show up.
Consier a company such as Google.   Do you have any idea just how much
*data* they house?  Whenever you goto 'www.google.com', and perform a
search -- you're connecting to a different cache of information each
time, so the results you get returned for a known search may well
differ.   And note that "specific" != "likely chance of singular hit".
In fact that's quite the opposite.  If your site is the only one listed
with a specific phrase, and has not been referenced elsewhere on the net
for google to pick up, what do you think the chances of it being listed
are?

> I've discussed this with Xplornet and they have come to the conclusion 
> that their site is "sandboxed", but don't know why, etc. And, frankly, 
> I'm not sure they care much either :)

To me it's a moot point, and is slightly egotistical to want to see your
site on a search engine.  If that happens all well and good.  Doubtless
it might happen eventually.

-- Thomas Adam



		
___________________________________________________________ 
24 FIFA World Cup tickets to be won with Yahoo! Mail http://uk.mail.yahoo.com