Rush Posted December 17, 2007 Report Share Posted December 17, 2007 This might be one throw into your 'ban' list:Microsoft URL Control bot hit me up for over 2 gigs of bandwidth in one day... This bot pretty much harvests email address, so blocking it isn't such a bad thing I use .htaccess take care of that business as it will likely ignore a robots.txt file.On a side note, I also block libwww-perl, which cuts down on most of the URL hacks that are attempted.-----------htaccess snippet----------------------------SetEnvIfNoCase User-Agent "Microsoft URL Control" keep_outSetEnvIfNoCase User-Agent "libwww-perl" keep_out<Limit GET POST >order allow,denyallow from alldeny from env=keep_out</Limit>----------------------------------------------------------You can also test against the changes you made by using this siteThis will simulate any user agent to make sure its being blocked.Rush Quote Link to comment Share on other sites More sharing options...
Micheal Mathews Posted December 18, 2007 Report Share Posted December 18, 2007 This might be one throw into your 'ban' list:Microsoft URL Control bot hit me up for over 2 gigs of bandwidth in one day... This bot pretty much harvests email address, so blocking it isn't such a bad thing I use .htaccess take care of that business as it will likely ignore a robots.txt file.On a side note, I also block libwww-perl, which cuts down on most of the URL hacks that are attempted.-----------htaccess snippet----------------------------SetEnvIfNoCase User-Agent "Microsoft URL Control" keep_outSetEnvIfNoCase User-Agent "libwww-perl" keep_out<Limit GET POST >order allow,denyallow from alldeny from env=keep_out</Limit>----------------------------------------------------------You can also test against the changes you made by using this siteThis will simulate any user agent to make sure its being blocked.RushHello Rush -- thanks for the info on this. I'm having a tough time with some nasty ones as well.I'm not trying to hijack your thread, but this is probably a good time to tell others about another nasty bot they might want to block -- the MJ bot version 1.08This version of the MJ bot is actually a bad bot faking as the otherwise well behaved MJ version 1.2 bot.I caught this bot in my logs draining off about 1/2 gig of bandwidth and did go to the MJ site to find out what the heck they were all about. If anyone is finding this one, you might want to check out http://www.majestic12.co.uk/projects/dsearch/mj12bot.php where they tell you what is going on.In the meantime, this is the recommended way to block the fake bot in htaccess:SetEnvIfNoCase User-Agent "^MJ12bot(.*)1\.0\.8" bad_bot <LIMIT GET PUT> Order Allow,Deny Allow from all Deny from env=bad_bot </LIMIT> blocking i.p. is not going to work with this version of the bad MJ bot because they appear to have built quite a big botnet.Thanks Rush.MEM Quote Link to comment Share on other sites More sharing options...
Rush Posted December 18, 2007 Author Report Share Posted December 18, 2007 Good info too...Going to need to add that one into my .htaccess file also.Thanks!Rush Quote Link to comment Share on other sites More sharing options...
nimebe Posted December 18, 2007 Report Share Posted December 18, 2007 Is the .htaccess just a text file to create and put somewhere? If so, where should it be put?If not, how do I go about creating it?Thanks,Nick Quote Link to comment Share on other sites More sharing options...
Rush Posted December 19, 2007 Author Report Share Posted December 19, 2007 What you can do is fire up Notepad and save the file as: htaccess (no dot)Upload the htaccess to your web root directory and rename it .htaccessThe file will control the folder it is in and all sub-folders. So if its in your web root directory, it will cover the entire site.HTH,Rush Quote Link to comment Share on other sites More sharing options...
nimebe Posted December 19, 2007 Report Share Posted December 19, 2007 What you can do is fire up Notepad and save the file as: htaccess (no dot)Upload the htaccess to your web root directory and rename it .htaccessThe file will control the folder it is in and all sub-folders. So if its in your web root directory, it will cover the entire site.HTH,RushHere's what I did:SetEnvIfNoCase User-Agent "Microsoft URL Control" keep_outSetEnvIfNoCase User-Agent "libwww-perl" keep_outSetEnvIfNoCase User-Agent "^MJ12bot(.*)1\.0\.8" bad_bot <Limit GET POST >order allow,denyallow from alldeny from env=keep_outDeny from env=bad_bot </Limit>Does that work? Quote Link to comment Share on other sites More sharing options...
nimebe Posted December 19, 2007 Report Share Posted December 19, 2007 Changed to:SetEnvIfNoCase User-Agent "Microsoft URL Control" keep_outSetEnvIfNoCase User-Agent "libwww-perl" keep_outSetEnvIfNoCase User-Agent "^MJ12bot(.*)1\.0\.8" keep_out <Limit GET POST >order allow,denyallow from alldeny from env=keep_out</Limit>Using the site Rush provided above, ^MJ12bot(.*)1\.0\.8, is still able to see my site. Any ideas?This is a new realm to me. Thank you both for your help!Nick Quote Link to comment Share on other sites More sharing options...
nimebe Posted December 19, 2007 Report Share Posted December 19, 2007 I Found this Site that offered this code as being better for that bot: SetEnvIfNoCase User-Agent "^MJ12bot(.*)1\.0\.8" stay_out <Location /> Order Allow,Deny Deny from env=stay_out Allow from all </Location>I don't know if it's better or not, but this code worked for me. I also found: Site which has a good list of "bad bots" if you're interested.NickNevermind, that blocked everything from my site! The original code posted by Michael works, I used MJ12bot1.0.8 as the bot in the test and it works, but MJ12bot(.*)1\.0\.8 doesn't. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.