robots.txt help

[aero_mansoor]

Tweet

I have seen some people use


Disallow: /google/

Disallow: /lycos/

Disallow: /images/

Disallow: /mirago/

Disallow: /looksmart/

Disallow: /thomson/

Disallow: /overture/

Disallow: /looksmart2/

Disallow: /yell/




in their robots.txt, can someone shed light as to the purpose?

Regards

Mansoor

[john-pp]

It basically just tells those robots (googlebot, lycosbot etc) to go away. But the problem is for it to work properly you would need to name the bot itself and the directory you want it to ignore eg the correct syntax for the robot file to do this would be:

User-Agent: spider or bot name
Disallow: the folder or directory you don't want it to index

However the format you're showing ie Disallow: /google/ is usually the syntax used for folders and directories not for user agents (spiders and bots) so it's a little confusing. Unless they have folders named google, lycos etc but as there's no user agent statement shown its hard to say but it looks like they are just folders that they don't want indexed.

[Paul Smith]

Anyone who has that in their robots.txt is snorting jasmin and hoping the bumble bee king will come and serve them honey on a platter every day, it's pointless.

The intention of course is to ban specific robots from crawling your site but in that format it is about useful as asking the man who runs the sock shop not to buy you socks for christmas.

If you don't want robots to index you put:

User-agent: *
Disallow: /

Otherwise do what normal people do and let the search engines crawl your site, its not a bad thing as you know.

[aero_mansoor]

I understand how to block specific bots, though issue really is what purpose does doing things like Disallow: /google/ give you? They are a fairly large website that i am competing against.

Intrestingly though the staright forward link analysers show only 2K links but in reality they have over 12K so somehow they are blocking off 10K links, through robots.txt and i ahve no idea how they ahve done that. (I know its 12K thorugh some other mouch more powerful services)

[nickynoodles]

I understand how to block specific bots, though issue really is what purpose does doing things like Disallow: /google/

Disallow: /google/

does nothing on it's own. This:

User-agent: *
Disallow: /google/

Would block all bots to the directory called /google/ on the site.

I have no idea why someone would want to do this. Assume incompetence, even though they are a large competitor. Paste the whole file if you like and I'll tell you if it has any practical purpose.