[TAG] Search Engjne Spiders

[bob van der Poel]

Thomas Adam wrote:
>  --- bob van der Poel <bvdp at xplornet.com> wrote: 
>  
>> Any light on how search engines like google work? I've recently moved my 
>> web stuff to the free site provided by my new ISP. For some reason 
>> google refuses to list any of my "really good stuff". And, doing some 
> 
> Probably then, the google site is listed in 'robots.txt' on the
> webserver as not being valid when sending out bots to spider your site
> (or a subdomain).   It's a common practise to ban most harvesting bots,
> and to allow only a few.   

I've actually checked that out. The ISP assures me that there are no 
robots.txt files buggering things up. I should have mentioned that in my 
original post, I guess.

[snip]

> To me it's a moot point, and is slightly egotistical to want to see your
> site on a search engine.  If that happens all well and good.  Doubtless
> it might happen eventually.

Of course it is moot. Which is why I posted this message. Figured it 
might of some interest to others. Sorry if you think I'm just being 
egotistical or whatever.

-- 
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: bvdp at xplornet.com
WWW:   http://users.xplornet.com/~bvdp