Bageone Posted October 4, 2006 Report Share Posted October 4, 2006 HiI was able to remove googlebot from showing on my access logs but have had no luck in getting rid of the hundreds of inktomisearch.com addresses. Do all of these need to be added or is there some way to exclude them all simply. They are like roaches with different numbers in front.Jodywww.internetbaglady.com Quote Link to comment Share on other sites More sharing options...
Scotty Posted October 4, 2006 Report Share Posted October 4, 2006 HiI was able to remove googlebot from showing on my access logs but have had no luck in getting rid of the hundreds of inktomisearch.com addresses. Do all of these need to be added or is there some way to exclude them all simply. They are like roaches with different numbers in front.Jodywww.internetbaglady.comAccording to this web page 123apx.com/whoisonthe IP's of 74.6. and 72.30. will cover them for the most part. I found these by Googling "inktomisearch", you are'nt the only one getting hit by them every day.Scotty Quote Link to comment Share on other sites More sharing options...
Bageone Posted October 4, 2006 Author Report Share Posted October 4, 2006 Is there a way to type in a wild card for the IP address instead of entering each one separately? Quote Link to comment Share on other sites More sharing options...
Scotty Posted October 4, 2006 Report Share Posted October 4, 2006 Is there a way to type in a wild card for the IP address instead of entering each one separately?Those IP addresses are the first 2 octets of a full 4 octet ip range. 74.6. wiil cover 74.6.0.0 to 74.6.255.255. Same for the other number. That covers quite a range..... especially since that is the range that covers Inktomi.OrgName: Inktomi Corporation OrgID: INKTAddress: 701 First AveCity: SunnyvaleStateProv: CAPostalCode: 94089Country: USNetRange: 74.6.0.0 - 74.6.255.255 CIDR: 74.6.0.0/16 NetName: INKTOMI-BLK-6NetHandle: NET-74-6-0-0-1Parent: NET-74-0-0-0-0NetType: Direct AllocationNameServer: NS1.YAHOO.COMNameServer: NS2.YAHOO.COMNameServer: NS3.YAHOO.COMNameServer: NS4.YAHOO.COMNameServer: NS5.YAHOO.COMComment: RegDate: 2006-02-13Updated: 2006-02-13OrgName: Inktomi Corporation OrgID: INKTAddress: 701 First AveCity: SunnyvaleStateProv: CAPostalCode: 94089Country: USNetRange: 72.30.0.0 - 72.30.255.255 CIDR: 72.30.0.0/16 NetName: INKTOMI-BLK-5NetHandle: NET-72-30-0-0-1Parent: NET-72-0-0-0-0NetType: Direct AllocationNameServer: NS1.YAHOO.COMNameServer: NS2.YAHOO.COMNameServer: NS3.YAHOO.COMNameServer: NS4.YAHOO.COMNameServer: NS5.YAHOO.COMComment: RegDate: 2005-01-28Updated: 2005-10-19Just out of curiousity, how are you blocking IP's now?Scotty Quote Link to comment Share on other sites More sharing options...
Bageone Posted October 5, 2006 Author Report Share Posted October 5, 2006 Thanks Scotty,This is the text in my log settings admin filebot,crawler,google.bot,74.6.0.0 to 74.6.255.255,72.30.0.0 to 72.30.255.255 But for some reason they are still showing up on my live log. Arggh! Thanks for helping me with this. Have you been able to block them from showing? Would you mind posting what is in your log settings here?JodySorry this is the link to my log access file with those entries in the admin area;http://www.internetbaglady.com/genealogy/showlog.phpI just changed it to bot,crawler,google.bot,74.6.0.,72.30., and is still showing in log.. Oh woah is me! lol Quote Link to comment Share on other sites More sharing options...
theKiwi Posted October 5, 2006 Report Share Posted October 5, 2006 Those settings in the Admin won't delete existing entries, but the settings will stop new entries from those IP number ranges from being written to the log.Check your log tomorrow or the next day and you should find it as you expect it to be.Roger Quote Link to comment Share on other sites More sharing options...
Scotty Posted October 5, 2006 Report Share Posted October 5, 2006 OK, we are talking two different functions: You want to block the stuff showing up in the log.I want to block these bandwidth hogs from even getting in the door. My mistake for reading something else into your original message. According to the help file:Exclude Host NamesBefore making any log entry, TNG will check this list. If the host of the visitor responsible for the potential log entry is on the list, no log entry will be made. Host names should be separated by commas (no spaces) and can consist of entire host names, IP addresses, or portions of either. For example, "googlebot" will block "crawler4.googlebot.com".Using "inktomisearch" should block it showing up in the access log according to the above. BTW, inktomisearch.com is the Yahoo web crawler. HTHScotty Quote Link to comment Share on other sites More sharing options...
mrstucci Posted October 13, 2006 Report Share Posted October 13, 2006 I, too, have that ink.... on my website and would like to block them. I am a real dummy when it comes to any mods but is there somewhere in the TNG Admin that I can put in 'inktomisearch' and that will block them from my site? Could it be that simple? JudyOK, we are talking two different functions: You want to block the stuff showing up in the log.I want to block these bandwidth hogs from even getting in the door. My mistake for reading something else into your original message. According to the help file:Exclude Host NamesBefore making any log entry, TNG will check this list. If the host of the visitor responsible for the potential log entry is on the list, no log entry will be made. Host names should be separated by commas (no spaces) and can consist of entire host names, IP addresses, or portions of either. For example, "googlebot" will block "crawler4.googlebot.com".Using "inktomisearch" should block it showing up in the access log according to the above. BTW, inktomisearch.com is the Yahoo web crawler. HTHScotty Quote Link to comment Share on other sites More sharing options...
B McFadyen Posted October 13, 2006 Report Share Posted October 13, 2006 I, too, have that ink.... on my website and would like to block them. I am a real dummy when it comes to any mods but is there somewhere in the TNG Admin that I can put in 'inktomisearch' and that will block them from my site? Could it be that simple? JudyBlocking,Just to clear things up.Using the "Exclude Host Names" in the admin->setup will prevent the accesses from showing up in the log.It does not block access to the pages.If you really want to block these accesses because you are concerned about bandwidth, you have two choices:1. Your robots.txt file can be used to tell friendly well behaved bots (like the ones you are concerned about - yahoo and friends) which sections of your site you do not want indexed, or to simply leave you alone.2. If you want to be forceful about it, you can use your ".htaccess" controls to tell your server to not serve pages to specific botsFor my site, and many others, we have spent significant efforts to encourage these bots to index our sites, so closing the door seems a little counter productive, but to each their own.If you want the bots to index your site but you feel they are hammering your site, you should consider the careful tailoring of your robots.txt file and the TNG file tngrobots.php. There are several forum threads that discuss this topic.If the above suggestions seem a little like a foreign language, check some of the forum threads that have discussed these ideas, then pose a question or two if you still need some assistance. There are many forum members that will offer suggestions and directions.Brian Quote Link to comment Share on other sites More sharing options...
mrstucci Posted October 13, 2006 Report Share Posted October 13, 2006 Thanks Brian for the clarification, have a great weekend, Judy Quote Link to comment Share on other sites More sharing options...
Russellgs Posted October 19, 2006 Report Share Posted October 19, 2006 Blocking,Just to clear things up.Using the "Exclude Host Names" in the admin->setup will prevent the accesses from showing up in the log.It does not block access to the pages.If you really want to block these accesses because you are concerned about bandwidth, you have two choices:1. Your robots.txt file can be used to tell friendly well behaved bots (like the ones you are concerned about - yahoo and friends) which sections of your site you do not want indexed, or to simply leave you alone.2. If you want to be forceful about it, you can use your ".htaccess" controls to tell your server to not serve pages to specific botsFor my site, and many others, we have spent significant efforts to encourage these bots to index our sites, so closing the door seems a little counter productive, but to each their own.If you want the bots to index your site but you feel they are hammering your site, you should consider the careful tailoring of your robots.txt file and the TNG file tngrobots.php. There are several forum threads that discuss this topic.If the above suggestions seem a little like a foreign language, check some of the forum threads that have discussed these ideas, then pose a question or two if you still need some assistance. There are many forum members that will offer suggestions and directions.BrianHello,I just checked my access log which I have set to 600 and it was full of the following bot: 38.99.203.110 Panscient_Data_Services.demarc.cogentco.com here is a link to a blogger about this bot. My log was full from 8PM till 12 AM when I denied access to stop, it had scanned over 4496 pages and 110MB of bandwidth :x Quote Link to comment Share on other sites More sharing options...
arnold Posted October 19, 2006 Report Share Posted October 19, 2006 How did you stop it???I am unable to. None of the regular methods (IP Deny Mnager and .htaccess) work.A great big HELP!!!!! Quote Link to comment Share on other sites More sharing options...
Russellgs Posted October 21, 2006 Report Share Posted October 21, 2006 How did you stop it???I am unable to. None of the regular methods (IP Deny Mnager and .htaccess) work.A great big HELP!!!!!I did a deny from 38.99.203.110 in .htaccess and I put Disallow: User-agent: panscient in my robots.txt fileI haven't had any hit from them since. last entry in my access log was at 11:54 PM on the 18th denied access at 12 AM on the 19th.Russ Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.