javo Posted April 20, 2008 Report Share Posted April 20, 2008 Hi, I am new at this so excuse some silly questions. I relise the importance of search engines but when looking at my access log I get hammered all the time by several crawlers at the same time. I am sure not all of them is needed as it seems most people use google as a search tool. My site is also foreign thus will mainly not be used by english. ex is host36-15-dynamic.19-87-r.retail.telecomitalia.it. How can I search there ip adress?Any thoughts will be appretiated, Thanks. Quote Link to comment Share on other sites More sharing options...
Cheeseburger Posted April 20, 2008 Report Share Posted April 20, 2008 Hi, I am new at this so excuse some silly questions. I relise the importance of search engines but when looking at my access log I get hammered all the time by several crawlers at the same time. I am sure not all of them is needed as it seems most people use google as a search tool. My site is also foreign thus will mainly not be used by english. ex is host36-15-dynamic.19-87-r.retail.telecomitalia.it. How can I search there ip adress?Any thoughts will be appretiated, Thanks.Hi,you can exclude those crawlers from the log.Just go to your admin page Setup >> Configuration >> Log Settings >>Exclude Host Namesand fill in following string: turnitin.com,googlebot.com, search.live.com, phx.gbl, amazonaws.com, crawl.yahoo.net, cuill.comThis will prevent showing up all those crawlers in your log, but will allow being crawled and indexed by the crawlers Quote Link to comment Share on other sites More sharing options...
javo Posted April 21, 2008 Author Report Share Posted April 21, 2008 Hi,you can exclude those crawlers from the log.Just go to your admin page Setup >> Configuration >> Log Settings >>Exclude Host Namesand fill in following string: turnitin.com,googlebot.com, search.live.com, phx.gbl, amazonaws.com, crawl.yahoo.net, cuill.comThis will prevent showing up all those crawlers in your log, but will allow being crawled and indexed by the crawlersThanks, I see in my log file for ex. google's crawler is shown as accessed by crawl-66-249-70-144.googlebot.com. Do I need to insert the complete accessed by crawl-66-249-70-144.googlebot.com or just googlebot.com.Also is there any crawlwers we should be wary of? Quote Link to comment Share on other sites More sharing options...
arnold Posted April 21, 2008 Report Share Posted April 21, 2008 If I trust googlebot.com, and I do, I add only googlebot.com. Quote Link to comment Share on other sites More sharing options...
TNGUSER Posted April 22, 2008 Report Share Posted April 22, 2008 Thanks, I see in my log file for ex. google's crawler is shown as accessed by crawl-66-249-70-144.googlebot.com. Do I need to insert the complete accessed by crawl-66-249-70-144.googlebot.com or just googlebot.com.Also is there any crawlwers we should be wary of?Don't exclude Yahoo crawl. Their IP addresses vary beginning with "lj" but they are safe as Roger pointed out in another Topic.Carol Quote Link to comment Share on other sites More sharing options...
Patsy Posted April 30, 2008 Report Share Posted April 30, 2008 Would it help to include a robots.txt to allow and disallow? I don't want all my directories to be crawled just my index.php...........Patsy Quote Link to comment Share on other sites More sharing options...
rdmorss Posted April 30, 2008 Report Share Posted April 30, 2008 The original topic of this thread was "Should I block crawlers"...It depends on whether you want your site to appear in search engines. If a crawler is blocked, then virtually no listings for your site will appear in that crawler's search engine.Excluding a crawler from your log (using the controls in TNG's admin section->Log Settings) does not prevent the crawler from visiting your site. It just won't report those hits in the TNG log file. So even though the crawler won't appear in your log, it could still be using a lot of your bandwidth.If you want to block a crawler from accessing your site, use an .htaccess and/or robots.txt file. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.