thewag Posted August 2, 2006 Report Share Posted August 2, 2006 Hey all, Wondering what experiances y'all have had with werelate.org and phx.gbl scanning your site. From what I can tell werelate seems to be legit, but I can't find any info (not even a whois) on phx.gbl.Thoughts? Quote Link to comment Share on other sites More sharing options...
Jodi Sweere Posted August 2, 2006 Report Share Posted August 2, 2006 Hey all, Wondering what experiances y'all have had with werelate.org and phx.gbl scanning your site. From what I can tell werelate seems to be legit, but I can't find any info (not even a whois) on phx.gbl.Thoughts?phx.gbl is the MSN searchbot.Look here: http://theapothecaryscat.com/trapdoor.php Quote Link to comment Share on other sites More sharing options...
Micheal Mathews Posted August 13, 2006 Report Share Posted August 13, 2006 phx.gbl is the MSN searchbot.Look here: http://theapothecaryscat.com/trapdoor.phpThat's a bit misleading in my opinion. Everybody and their dog, including the big boys, are in that list. The website you are referring to isn't implementing a robots.txt file that I can see -- so if I can't see it then robots can't see it either, which means that those bots probably followed the do not follow link and got captured. The purpose of trapbots is to get bad bots who do not read and follow robots.txt -- This seemingly is confirmed with a site:www.theapothecaryscat.com command in Google. Maybe other Google datacenters show different results, but I show two results and one supplemental, none of which are the home page and one link that goes to goes directly to a directory listing. YIKES!Kind of hard to get search results if you are locking out the major engines.Regards,Micheal Mathews Quote Link to comment Share on other sites More sharing options...
B McFadyen Posted August 14, 2006 Report Share Posted August 14, 2006 That's a bit misleading in my opinion. Everybody and their dog, including the big boys, are in that list. The website you are referring to isn't implementing a robots.txt file that I can see -- so if I can't see it then robots can't see it either, which means that those bots probably followed the do not follow link and got captured. Keep in mind that it is straight forward to present a different robots.txt file based upon the user agent using mod_rewrite. Just because you don't see a robots.txt file, doesn't neccessarily mean there isn't one.Brian Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.