wombmate Posted March 11, 2007 Report Share Posted March 11, 2007 Does anyone have an idea if this is a new crawler?crawler4.irl.cs.tamu.eduI noticed a number of entries in my log file beginning today and have traced it to the Texas A&M University Computer Science Department but can't find anything beyond that.Thanks in advance,DarleneSorenson-Robey.org Quote Link to comment Share on other sites More sharing options...
Rush Posted March 11, 2007 Report Share Posted March 11, 2007 As best I can tell, its a internet topology project, so it should be harvesting links, not data.That spider has been around my site a few times.The spider should also respect the robots.txt file, so you can block it if you want to. Rush Quote Link to comment Share on other sites More sharing options...
arnold Posted March 11, 2007 Report Share Posted March 11, 2007 I looked at your access log and saw the same entries. I think your going to Texas A&M University Computer Science Department shows that the crawler is coming from them. The crawler may be no more than a class project. If you search for the crawler on Google, you will see that it is hitting other websites also.Seems to me that it is probably harmless. But you can deny it access to your website or you can allow it in, but not have it appear on your access log. Or include it in a robots.txt file, as Rush suggested. Or pick up the phone and call the department head:Dr. Valerie E. TaylorPhone: 979-845-5820 Quote Link to comment Share on other sites More sharing options...
wombmate Posted March 11, 2007 Author Report Share Posted March 11, 2007 Thank you Rush and Arnold for the fast reply. I believe that I will move the entry to my botlog to keep things tidy. Sounds like an interesting project.Cheers!Darlene Quote Link to comment Share on other sites More sharing options...
Jodi Sweere Posted March 12, 2007 Report Share Posted March 12, 2007 Does anyone have an idea if this is a new crawler?crawler4.irl.cs.tamu.eduI noticed a number of entries in my log file beginning today and have traced it to the Texas A&M University Computer Science Department but can't find anything beyond that.Thanks in advance,DarleneSorenson-Robey.orgThat same crawler has been visiting me this afternoon... Quote Link to comment Share on other sites More sharing options...
mcentyre501 Posted March 15, 2007 Report Share Posted March 15, 2007 copied to new topic Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.