[TAG] Search Engjne Spiders
bob van der Poel
bvdp at xplornet.com
Fri Apr 21 04:21:36 MSD 2006
Thomas Adam wrote:
> --- bob van der Poel <bvdp at xplornet.com> wrote:
>> Any light on how search engines like google work? I've recently moved my
>> web stuff to the free site provided by my new ISP. For some reason
>> google refuses to list any of my "really good stuff". And, doing some
> Probably then, the google site is listed in 'robots.txt' on the
> webserver as not being valid when sending out bots to spider your site
> (or a subdomain). It's a common practise to ban most harvesting bots,
> and to allow only a few.
I've actually checked that out. The ISP assures me that there are no
robots.txt files buggering things up. I should have mentioned that in my
original post, I guess.
> To me it's a moot point, and is slightly egotistical to want to see your
> site on a search engine. If that happens all well and good. Doubtless
> it might happen eventually.
Of course it is moot. Which is why I posted this message. Figured it
might of some interest to others. Sorry if you think I'm just being
egotistical or whatever.
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: bvdp at xplornet.com
More information about the TAG