bayanne Posted December 29, 2006 Report Share Posted December 29, 2006 I am trying to reduce the amount of crawling by all the search engines as it is beginning to really push up the site usage to levels that the host does not like.I have version 6.0.3 installed but how do I make best use of tngrobots.php with a suitable robots.txt, ..... if that is how it's done.Certainly since I upgraded to version 6 there has been no lessening in the amount of crawling.Cheers Quote Link to comment Share on other sites More sharing options...
Rush Posted December 29, 2006 Report Share Posted December 29, 2006 If you use Google Sitemaps you can adjust the crawl rate to slow.If that doesn't slow it down enough, I suppose you could block Google for awhile or let it crawl for a few days at a time. This would be done through a htaccess file and if your not sure how I could post the instructions.Rush Quote Link to comment Share on other sites More sharing options...
Scotty Posted December 29, 2006 Report Share Posted December 29, 2006 I am trying to reduce the amount of crawling by all the search engines as it is beginning to really push up the site usage to levels that the host does not like.I have version 6.0.3 installed but how do I make best use of tngrobots.php with a suitable robots.txt, ..... if that is how it's done.Certainly since I upgraded to version 6 there has been no lessening in the amount of crawling.CheersMake a robots.txt file with e following:User-agent: GooglebotCrawl-delay: 10Crawl-delay is the number of days, change that to whatever you desire.Disallow: / will stop all crawling if you'd rather use that.Save it on the ROOT folder of you site. Not a subfolder as it will be ignored. And yes it works as I have stopped Yahoo. I had 3 days in row of over 500 hits from Yahoo before I got tired of them.Scotty Quote Link to comment Share on other sites More sharing options...
Rush Posted December 29, 2006 Report Share Posted December 29, 2006 Yahoo and MSN obey the crawl delay, Google doesn't.Controlling Google crawl delayRush Quote Link to comment Share on other sites More sharing options...
Scotty Posted December 30, 2006 Report Share Posted December 30, 2006 Yahoo and MSN obey the crawl delay, Google doesn't.Controlling Google crawl delayRushYup... I didn't dig deep enuff into Google earlier....cept my crawl rate link is not available... and robots.txt is on my root...more playing around is in order.Scotty Quote Link to comment Share on other sites More sharing options...
bayanne Posted December 30, 2006 Author Report Share Posted December 30, 2006 I had set up the following in a robots.txt file which I placed in the root domain:User-Agent: *Disallow: /I thought that this may stop all, but certainly it didn't stop google, unless there is a time dealy before it kicks in.Tony Quote Link to comment Share on other sites More sharing options...
Rush Posted December 30, 2006 Report Share Posted December 30, 2006 There could be a little bit of delay before Google re-looks at your robot.txt file.Rush Quote Link to comment Share on other sites More sharing options...
Gary Barton Posted February 21, 2007 Report Share Posted February 21, 2007 If you use Google Sitemaps you can adjust the crawl rate to slow.I, too, want to slow down the crawl rate for googlebot. Following these threads, I realize and went to Google Webmaster Tools to set up a Site Map. To do that Google wants to verify that I am the owner. To do that they want a line of code (verify) pasted in the first <HEAD> section before the <BODY> section. My problem is that in my index.php file I don't see an opening or closing <HEAD> statement. Interestingly, I can see it by going to the page and View Source, but not in the actual index.php file.Where is it? How can I add the code so that Gooogle can verify my ownership so that I can set the Delay to 10 or 15 days?thx,garywww.bartondatabase.info [template #4] Quote Link to comment Share on other sites More sharing options...
arnold Posted February 21, 2007 Report Share Posted February 21, 2007 I am trying to reduce the amount of crawling by all the search engines as it is beginning to really push up the site usage to levels that the host does not like.I have version 6.0.3 installed but how do I make best use of tngrobots.php with a suitable robots.txt, ..... if that is how it's done.Certainly since I upgraded to version 6 there has been no lessening in the amount of crawling.CheersI am going to put a different spin on this. Most of us die to have search engines hit our websites. That is the *only* way for us to be seen. By limiting the number of hits search engines are allowed, we end up limiting our visibility to those on the net.Of course, where I am going with this is to suggest that if you have the money and time to transfer your database to a new host, you may be better off doing that. I apologize if staying where you are is most important. Quote Link to comment Share on other sites More sharing options...
Gary Barton Posted February 21, 2007 Report Share Posted February 21, 2007 I'm getting alot of crawling activity from:c-24-9-167-77.hsd1.co.comcast.net.Does anybody recognize them? How can I block them? Is it the robots.txt file? If so, what is the syntax?thx,gary Quote Link to comment Share on other sites More sharing options...
theKiwi Posted February 21, 2007 Report Share Posted February 21, 2007 I, too, want to slow down the crawl rate for googlebot. Following these threads, I realize and went to Google Webmaster Tools to set up a Site Map. To do that Google wants to verify that I am the owner. To do that they want a line of code (verify) pasted in the first <HEAD> section before the <BODY> section. My problem is that in my index.php file I don't see an opening or closing <HEAD> statement. Interestingly, I can see it by going to the page and View Source, but not in the actual index.php file.When I have set up sitemaps for Google, I've been instructed to upload a file with a particular file name to my site and then tell Google that the file is there, and they look for it and verify it.That presumably was offered as an option for your site?RogerI'm getting alot of crawling activity from:c-24-9-167-77.hsd1.co.comcast.net.Does anybody recognize them? How can I block them? Is it the robots.txt file? If so, what is the syntax?This is an individual user who has a Comcast cable internet connection, in Colorado. Their IP address is 24.9.167.77If you want to block them you need to do that with a .htaccess file - since it's not a Search robot, robots.txt will not prevent their access.Roger Quote Link to comment Share on other sites More sharing options...
arnold Posted February 22, 2007 Report Share Posted February 22, 2007 An option to Roger's reply to use the .htaccess file is to go to your cPanelX, click on the IP Deny Manager icon, and enter the offending IP Address.You must be aware that then you enter a specific IP Address anyone using that IP Address is denied access. Generally there are many using that IP Address, but probably not accessing your website, so it is probably a safe move to make. When someone is denied access, they receive the default 403/Forbidden message. The words on this message are written by your host. I changed our 403 message to list which IP Addresses or Domain Names are being denied access, why, and provide my e-mail address so that those folks can contact me. Arnold Quote Link to comment Share on other sites More sharing options...
theKiwi Posted February 22, 2007 Report Share Posted February 22, 2007 You must be aware that then you enter a specific IP Address anyone using that IP Address is denied access. Generally there are many using that IP Address, but probably not accessing your website, so it is probably a safe move to make.Only one person, or more strictly 1 subscriber to a cable company or DSL provider is using that IP number at any one time, and even though these numbers issued by Cable providers and DSL providers are "dynamic" the reality is that a single subscriber retains the same IP number for days or weeks at a time.There is no way that different people in different parts of the country are using the same number at the same time.Where you might run into trouble is people who have internet access by dialup where they are assigned one of their ISP's IP numbers for the duration of their session, and when they're done, possibly another user will get the same number - I'm thinking of places like AOL who show in logs as coming from a proxy or cache.aol.com.In this case mentioned above though it's a single customer of ComCast somewhere in Colorado who is doing the visiting.Roger Quote Link to comment Share on other sites More sharing options...
arnold Posted February 22, 2007 Report Share Posted February 22, 2007 Roger,Thank you for your reply. I did not mean to give bad information.Are you saying that, in a city the size of Chicago where ComCast operates, a separate IP Address is assigned to each subscriber who may actually keep that IP Address for weeks at a time?Arnold Quote Link to comment Share on other sites More sharing options...
theKiwi Posted February 22, 2007 Report Share Posted February 22, 2007 Are you saying that, in a city the size of Chicago where ComCast operates, a separate IP Address is assigned to each subscriber who may actually keep that IP Address for weeks at a time?Yes, that is what I am saying.Comcast subscriber has a unique IP number assigned to them out of the blocks of IP numbers assigned to Comcast.As each SpeakEasy.net DSL customer across the country at any one time has a unique IP address (I have 8 static IPs from them on my DSL connection).For the address in questionwhois 24.9.167.77Comcast Cable Communications, Inc. EASTERNSHORE-1 (NET-24-0-0-0-1) 24.0.0.0 - 24.15.255.255Comcast Cable Communications COLORADO-10 (NET-24-8-0-0-1) 24.8.0.0 - 24.9.255.255which is telling me that Comcast Eastern shore has the block24.0.0.0 to 24.15.255.255 - 16 x 256 x 256 = 1,572,864 unique IP addresses. Of them 131,072 of them are in the block called Colorado-10.Roger Quote Link to comment Share on other sites More sharing options...
Rush Posted February 22, 2007 Report Share Posted February 22, 2007 There is no way that different people in different parts of the country are using the same number at the same timeUnless that IP is running a proxy - knowingly or unknowingly, but not very likely. Rush Quote Link to comment Share on other sites More sharing options...
arnold Posted February 22, 2007 Report Share Posted February 22, 2007 Yes, that is what I am saying.Comcast subscriber has a unique IP number assigned to them out of the blocks of IP numbers assigned to Comcast.As each SpeakEasy.net DSL customer across the country at any one time has a unique IP address (I have 8 static IPs from them on my DSL connection).This is where my confusion comes in. When I sign up with a host Internet provider, I am given generally a choice of static or dynamic IP Addresses. Static is more expensive. I had assumed that static meant that an IP Address is taken out of the block is "given" on a permanent basis to the user. I had assumed that dynamic meant an IP Address is "shared" by a changing, random group of users. You are saying, if I understand correctly, that each user is provided a separate IP Address, no matter if one pays the higher static rate or the lower dynamic rate. I have always selected the lower priced dynamic rate. What would the advantage be to me if I were to switch from a dynamic IP Address to a static IP Address? This may be the same question, but why does the host internet provider offer both?(I am sorry to have gotten this thread off-topic.)Many thanks. Quote Link to comment Share on other sites More sharing options...
theKiwi Posted February 22, 2007 Report Share Posted February 22, 2007 This is where my confusion comes in. When I sign up with a host Internet provider, I am given generally a choice of static or dynamic IP Addresses. Static is more expensive. I had assumed that static meant that an IP Address is taken out of the block is "given" on a permanent basis to the user. I had assumed that dynamic meant an IP Address is "shared" by a changing, random group of users. You are saying, if I understand correctly, that each user is provided a separate IP Address, no matter if one pays the higher static rate or the lower dynamic rate. I have always selected the lower priced dynamic rate. What would the advantage be to me if I were to switch from a dynamic IP Address to a static IP Address? This may be the same question, but why does the host internet provider offer both?With a static IP number you're paying for the certainty that you know what it is and that it won't change, and for the expectation that you would use it for the purpose to which it is best suited - ie having a server in your home connected full time online acting as a server.With a dynamic IP number, you're not told what the number will be, and there's no guarantee that it won't change. I think in reality it remains the same often for as long as the connection between the company and the modem/box in the home remain connected, but if either end gets "reset" then the number might change.Roger Quote Link to comment Share on other sites More sharing options...
ebelew Posted February 22, 2007 Report Share Posted February 22, 2007 Some of us have to have the same (static) IP address. That's the way we get into our home systems through programs like VNC and pcAnywhere. You can still do it, it's just harder to do.If you ever need to know your IP address assigned by your ISP.Go here http://www.ipaddressworld.comFolks spend a lot of time in school learning this.Eddy Quote Link to comment Share on other sites More sharing options...
arnold Posted February 23, 2007 Report Share Posted February 23, 2007 Roger and Eddy,Thank you. What you wrote is very helpful.Eddy, I went to your link and think that adding the IP Address box to our getperson.php pages would be great. That would let those who hammer on our websites be aware that "We see you and know who you are." I am serious and if anyone else is, and has good php programming skills, I would like to incorporate whatever you come up with.Thanks,Arnold Quote Link to comment Share on other sites More sharing options...
Rush Posted February 23, 2007 Report Share Posted February 23, 2007 Maybe I'm a bit skeptic, but anyone serious about data mining or scraping can easily defeat most defenses that can be written.Someone running through a proxy will mask who they actually are (or their IP).Banning a proxy IP doesn't really help either, because one can run multiple threads through hundreds of proxies. So basically it would look like single, double or triple hits coming from all different computers around the world. I'm not saying defenses shouldn't be implemented or designed. Most defenses put into place would help block about 90% of those people, but IMHO, it's the 10% that have the tools, skills, motivation and are driven by the bottom line that one needs to watch out for.Rush Quote Link to comment Share on other sites More sharing options...
arnold Posted February 23, 2007 Report Share Posted February 23, 2007 Banning a proxy IP doesn't really help either, because one can run multiple threads through hundreds of proxies. So basically it would look like single, double or triple hits coming from all different computers around the world.So, you are saying that even though I watch our access log like a hawk, all that anyone has to do to fool me and keep me from seeing a pattern of constant hits from one source, is to use many IP Addresses at once instead of just one? Quote Link to comment Share on other sites More sharing options...
Rush Posted February 23, 2007 Report Share Posted February 23, 2007 Yup...Which I suppose drives the question, is there a reason for someone to go through the hassle to do it on a genealogy site?Finding and utilizing numerous viable proxy servers can be a real pain sometimes and would consume a lot of time to 'steal' data from a genealogy site.The average person who just wants to grab the info for personal use probably wouldn't go through the trouble. Rush Quote Link to comment Share on other sites More sharing options...
arnold Posted February 23, 2007 Report Share Posted February 23, 2007 Which gets me back to my original request for someone write the code (and where to place it) to let us add the IP Address box to our getperson.php pages. I know html, but not php. Early on, I often locked up our website with my initial attempts to change Darrin's pages. We have had our genealogical information sold on e-bay. We once bought a CD which promised a great deal of Sprague information. The promise was valid, but it was all our work. Soooooo, if I can preclude folks from attempting to steal from us by no more than saying we are recording your IP Address on the getperson.php pages, then the developer of our database can sleep better. Quote Link to comment Share on other sites More sharing options...
theKiwi Posted February 23, 2007 Report Share Posted February 23, 2007 Which gets me back to my original request for someone write the code (and where to place it) to let us add the IP Address box to our getperson.php pages. I know html, but not php. Early on, I often locked up our website with my initial attempts to change Darrin's pages. We have had our genealogical information sold on e-bay. We once bought a CD which promised a great deal of Sprague information. The promise was valid, but it was all our work. Soooooo, if I can preclude folks from attempting to steal from us by no more than saying we are recording your IP Address on the getperson.php pages, then the developer of our database can sleep better.Just add whatever text you want to either of topmenu or footer.You don't need the code to get the IP address so you can then tell someone what their IP address is - just tell them that all page requests are logged surely?And they then will act accordingly - either to ignore your warning and carry on, or to heed your warning.If you want to stop them, you need to implement something like Brian's rip prevention, or take their IP addresses one at a time and ban them from your site as previously noted.Roger Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.