How to block search engine crawlers?

[rphillips08]

I check my log file every day or so and have noticed that for the last several weeks google, inktomi and other search engines have been simultaneously hammering at my database. I'm concerned not only that they are cataloging my website but that they may be degrading performance for regular people. I've posted "robots.txt" files all over the place but my assumption is that these are being bypassed since my web pages referenced by numerous links, many of which I don't control. Can an entry be put into the .php code that tells them to stay out?

[aefhrt562]

I have had the same problem and I resolved it by requiring users to logon to access my web site. I just set up a Guest user id and password and that stoped the crawlers.

[theKiwi]

I would just wonder how to "regular people" get to find out about your site if it isn't for the search engines?

My site too gets hit by these things, but whenever I've checked it from outside my house (the sites are running on my PowerMac G4 that I use every day) the response is fine.

For some order of comparison, between the TNG sites I run, the Apache log files have a total of 1.6 million lines in them starting from January 2004. Analogs of the sites eg

http://roger.lisaandroger.com/Report.html

indicate that the majority of them are Search Engines.

But it's these search engines which drive visitors, and even unknown relatives to my site, so I consider it a fair trade off.

Roger

[kpeirce]

According to the rules, which most web robots follow, they look for the robots.txt (lower case only), and this file must be in the root directory of the domain address. They ignore robots.txt files placed any where else.

So if your web address is www.example.com the only place they will look is at www.example.com/robots.txt. This creates problems for people who aren't hosting directly under the domain name. I have mine hosted in a separate directory, but because I own my own domain name and have full access to the server, I can place the robots.txt file in the top directory.

The only other option is to use META tags in the HTML headers, but this can be troublesome.

[mvigor]

You can use PHP code to determine if a given page is being requested by a web crawler...

$is_it_a_bot = getenv("REMOTE_HOST");

if (strstr($is_it_a_bot,"msnbot.msn.com") ||

strstr($is_it_a_bot,"googlebot.com")) {

// Here we'll allow Google and MSN search engines

// to have access to this page, so the pages

// will get included in their search results.

$logged_in = "Yes, it's a search engine bot.";

$logged_in_email = "robot@robots.com";

}

[CCarter]

Can you tell me where this code is placed? Is it part of the robot.txt file?

You can use PHP code to determine if a given page is being requested by a web crawler...

$is_it_a_bot = getenv("REMOTE_HOST");

if (strstr($is_it_a_bot,"msnbot.msn.com") ||

strstr($is_it_a_bot,"googlebot.com")) {

// Here we'll allow Google and MSN search engines

// to have access to this page, so the pages

// will get included in their search results.

$logged_in = "Yes, it's a search engine bot.";

$logged_in_email = "robot@robots.com";

}

Thank you,

-charles