waterhead Posted November 25, 2005 Report Share Posted November 25, 2005 My site is getting hammered constantly by Google and my bandwidth limits are being exceeded as a result. I'd be grateful if someone can tell me how to stop this - or preferably limit it - I'd still like to be found by Google but it doesn't need umpteen GB every month to record my updates!Chris Quote Link to comment Share on other sites More sharing options...
arnold Posted November 25, 2005 Report Share Posted November 25, 2005 Use a google to find information on "robots.txt." You place specific text in robots.txt to preclude, for example, certain search engines from visiting your website. I assume that the file goes between the head tags.I am being purposefully vague as I looked at various websites to help you and thought some of what I read was contradictory. Others' replies here may be more helpful, but in the meantime try the above. Quote Link to comment Share on other sites More sharing options...
rdmorss Posted November 26, 2005 Report Share Posted November 26, 2005 How can I stop Google indexing?You can also add robots directives to your meta.html file:<meta name="robots" content="noindex,nofollow"> This tells all robots to go away and not index the current page or follow links to any other pages in your site. To index your homepage only, but not follow links to other parts of your site use this in your homepage <head> section:<meta name="robots" content="index,nofollow">Note that not all robots obey robots directives, whether in a robots.txt file or a meta tag. Quote Link to comment Share on other sites More sharing options...
KeithB Posted November 26, 2005 Report Share Posted November 26, 2005 HiI had a similar problem and tried using robots.txt. It didnt stop Google, so here is what I can offer.First, make a file called robots.txt and add this into it.User-agent: GooglebotDisallow: /Place this in the same folder (directory) as your main TNG public files.If that does not work, then you need to use the "hammer" approach. It requires that you understand a bit about IP adresses. This will work as long as Googlebot continues to use this IP address cluster.Create a file called .htaccess (nothing in front of the ".") and add this.deny from 66.249.66Place this in the same folder as your main TNG public files.Do not add the last set of numbers for a full IP addy. This will ban activity from the entire IP cluster. (66.249.66.0 to 66.249.66.255)Hope this helps. Quote Link to comment Share on other sites More sharing options...
waterhead Posted November 26, 2005 Author Report Share Posted November 26, 2005 Thanks very much for the replies. I had seen the stuff about nofollow not being respected by Google and had been rather discouraged so I shall try the approach suggested by KeithB.Chris Quote Link to comment Share on other sites More sharing options...
arnold Posted November 26, 2005 Report Share Posted November 26, 2005 Chris,I've been thinking a lot about your post. I, for one, pray for our website to be hammered constantly by Google. That is the only way that all our names will ever be accessible to those doing name searches via Google.I use 1and1.com http://order.1and1.com for the other websites I do. The price is minimal. Here are its two lowest rates:$2.99 per month250 GB monthly transfer volume $4.99 per month500 GB monthly transfer volume For most of us, these monthly transfer volumes far exceed our actual usage. As of now, the monthly bandwidth usage for The Sprague Project is 4.976 Gigabytes.My site is getting hammered constantly by Google and my bandwidth limits are being exceeded as a result. I'd be grateful if someone can tell me how to stop this - or preferably limit it - I'd still like to be found by Google but it doesn't need umpteen GB every month to record my updates! Quote Link to comment Share on other sites More sharing options...
rdmorss Posted November 27, 2005 Report Share Posted November 27, 2005 googlebot alone shouldn't cause you to exceed your bandwidth allowance - unless your allowance is extremely low. But as Arnold noted, bandwith allowance on virtual hosts these days is usually in the hundreds of gigabytes. Check your access logs. Are you delivering lots of huge image or video files on your site? Do you have a runaway script? Do you run a chat room?If you want people to find your site you really need to be indexed by Google.What's your URL? Quote Link to comment Share on other sites More sharing options...
waterhead Posted November 29, 2005 Author Report Share Posted November 29, 2005 Arnold, Robert,Thanks for your posts. My bandwidth allowance isn't nearly the 250 GB mentioned by Arnold - it is 15GB which until this month and last I had not come close to exceeding but for the last couple of months, Google has gone berserk. Below I show my stats for my web. The other robots are doing their usual thing.Fortunately it is nearly the end of the month and I should scrape by. I don't wish to change hosts, I have had too many bad experiences with other hosts, especially the cheap ones and I did a lot of research before picking my present one (HostDime) who has been magnificent with most service queries responded to in minutes rather than hours.I don't wish to shut off Google completely, I would just like to slow them down and am reluctant to shut them out with htaccess - the other suggestions don't seem to work - nor for MSNbot by the way.If need be, I shall probably upgrade my package - my site has put me in touch with so many nice people including relatives around the world I am now in the process of visiting. My Google presence is very good and searching on many surnames and particularly combinations of names, I come out at or near the top of the results list.BTW my site is http://chrisknight.info/genealogyFor anyone who wants more Google presence, the only thing I can suggest is to register your site at Cyndi's list - I did this way back in 2003 and got hit very soon after.ChrisRobots/Spiders visitors (Top 25) - Full list - Last visit 9 different robots* Hits Bandwidth Last visit Googlebot 234180 12.48 GB 28 Nov 2005 - 20:50 MSNBot 2620 67.31 MB 28 Nov 2005 - 04:05 Unknown robot (identified by 'robot') 400 6.93 MB 16 Nov 2005 - 11:16 WISENutbot 150 6.13 MB 28 Nov 2005 - 22:56 Inktomi Slurp 61 1.21 MB 28 Nov 2005 - 22:58 Unknown robot (identified by 'crawl') 17 1.70 MB 27 Nov 2005 - 13:35 AskJeeves 8 76.07 KB 27 Nov 2005 - 15:20 Scooter 3 62.44 KB 15 Nov 2005 - 20:00 Netcraft 1 0 27 Nov 2005 - 07:13 Quote Link to comment Share on other sites More sharing options...
waltreams Posted November 29, 2005 Report Share Posted November 29, 2005 Try this in your robots.txtUser-agent: googlebotCrawl-delay: 180The "180" is the delay is seconds. You can change this to whatever... Quote Link to comment Share on other sites More sharing options...
rdmorss Posted November 29, 2005 Report Share Posted November 29, 2005 Googlebot 234180 12.48 GB 28 Nov 2005 - 20:50 Incredible. Might be worth a try contacting google. Here's a webform from their site. I am interested to see if they answer and what they have to say about their 234,000 hits on your site this month (nearly 100 times more than the next highest robot). I wonder why they need to hit your site that much just to index it. Quote Link to comment Share on other sites More sharing options...
waltreams Posted November 29, 2005 Report Share Posted November 29, 2005 Also, there is some good info about disallows herehttp://forum.mamboserver.com/archive/index...hp/t-58771.html Quote Link to comment Share on other sites More sharing options...
waterhead Posted November 30, 2005 Author Report Share Posted November 30, 2005 I wonder why they need to hit your site that much just to index it.I can only conclude that it might have something to do with the nature of sites rendered by php. The size of my data is miniscule compared to the data transferred by Google but if you imagine the result of tracking every link from every other link as it were, then rendering these as pages, that just might do it. Without a lot of forensics with the web stats I am just guessing that Google does something like that. Of course my site is hardly unique in being served by php scripts so this notion may be total rubbish - if Google did that with all php sites there wouldn't be enough didk drives in the world to keep the data!Waltreams, thanks for the useful link.Chris Quote Link to comment Share on other sites More sharing options...
jtellup Posted December 17, 2005 Report Share Posted December 17, 2005 I had the same problem, and like you it just started recently.And I blocked their IP in htacess.Msn still spiders our site quite often, and I noticed that all the changes have shown up in google as they spider MSN. Quote Link to comment Share on other sites More sharing options...
rocksea Posted January 15, 2006 Report Share Posted January 15, 2006 And I blocked their IP in htacess.Msn still spiders our site quite often, and I noticed that all the changes have shown up in google as they spider MSN.haha! funny, Is it so?!! Quote Link to comment Share on other sites More sharing options...
waterhead Posted January 16, 2006 Author Report Share Posted January 16, 2006 I had to block Google today in htaccess - they had used 30GB of bandwidth this month!!!I am glad to say my web host reset my clock for me otherwise my site would have been down. This is a different site from the one I had trouble with earlier (I maintain them on separate servers so that if one goes down, I can use the other) so it shows that it was not site specific. I conclude that Google's crawlers are deficient in some major way if they waste this much effort on my site which doesn't change more than a few megs a month. Quote Link to comment Share on other sites More sharing options...
KeithB Posted January 17, 2006 Report Share Posted January 17, 2006 WH .........FYI,Googlebot is now using two IP strings. You may wish to add the second to your htaccess file.66.249.6566.249.66As I said before, do NOT fill in the last part of the string.These seem to hold them out thus far, but I have no doubt they will add more strings in the future.Regards Quote Link to comment Share on other sites More sharing options...
waterhead Posted January 18, 2006 Author Report Share Posted January 18, 2006 Keith,Thanks very much.Chris Quote Link to comment Share on other sites More sharing options...
KeithB Posted January 27, 2006 Report Share Posted January 27, 2006 Chris,Grrrrrrrrrrr,They have added a new string as well. Getting sneaky, googlebot is!!add this as welldeny from 66.249.72If they keep this up, I am goin' WABBIT huntin' !! Regards Quote Link to comment Share on other sites More sharing options...
janvorster Posted November 10, 2007 Report Share Posted November 10, 2007 I am so glad I found this thread as googlebot caused me to upgrade my package to accomodate them. It still was not enough and I had to pay some penalty fees for excessve taffic use caused by googlebot.I have placed .htaccess as follows in my root directory as well as sub directories. order allow,denydeny from 66.249.72deny from 66.249.65deny from 66.249.66allow from all In case they add another string please let us know.I don't understand much of the code as I am not a it specialist but I surely hope this is going to do the job. Quote Link to comment Share on other sites More sharing options...
arnold Posted November 10, 2007 Report Share Posted November 10, 2007 If you really want to prohibit google from hitting your website, enter the following in .htaccess:deny from googlebot.comHowever, most of us WANT Google to index our websites as that is what brings visitors to us. Quote Link to comment Share on other sites More sharing options...
janvorster Posted November 15, 2007 Report Share Posted November 15, 2007 If you really want to prohibit google from hitting your website, enter the following in .htaccess:deny from googlebot.comHowever, most of us WANT Google to index our websites as that is what brings visitors to us.I guess this will take care of there changing ip adress ... Quote Link to comment Share on other sites More sharing options...
arnold Posted November 15, 2007 Report Share Posted November 15, 2007 I guess this will take care of there changing ip adress ...Maybe. In the last few days, Google started hitting our website as google.com, not just googlebot.com. Please see below.Thu 15 Nov 2007 04:35:29 AM Individual info for Edgar M. Sprague (I115297) accessed by py-out-f136.google.com.Thu 15 Nov 2007 04:31:24 AM Individual info for Sybil Sprague (I142404) accessed by py-out-f136.google.com.Thu 15 Nov 2007 04:24:12 AM Individual info for Richard B. Sprague (I115295) accessed by py-out-f136.google.com.Thu 15 Nov 2007 04:14:55 AM Individual info for Mary Carrington (I42565) accessed by ip72-193-31-6.lv.lv.cox.net.Thu 15 Nov 2007 04:07:51 AM Individual info for Richard B. Sprague (I115295) accessed by py-out-f136.google.com.Thu 15 Nov 2007 04:07:01 AM Pedigree for Richard B. Sprague (I115295) 4 Generations accessed by py-out-f136.google.com.Thu 15 Nov 2007 04:05:54 AM Tree: accessed by 059148232033.ctinets.com.Thu 15 Nov 2007 04:05:41 AM Search Results for Last Name contains Sprague AND First Name contains RICHARD B. accessed by py-out-f136.google.com.Thu 15 Nov 2007 03:55:33 AM Surname List accessed by py-out-f136.google.com.Thu 15 Nov 2007 03:53:07 AM Family History Library (R19) accessed by py-out-f136.google.com.Thu 15 Nov 2007 03:52:17 AM Repositories accessed by py-out-f136.google.com. Quote Link to comment Share on other sites More sharing options...
janvorster Posted November 15, 2007 Report Share Posted November 15, 2007 Maybe. In the last few days, Google started hitting our website as google.com, not just googlebot.com. Please see below.Thu 15 Nov 2007 04:35:29 AM Individual info for Edgar M. Sprague (I115297) accessed by py-out-f136.google.com.This is a big concern for me. How can I stop them before they put my traffic in the red again. I have a feeling the py-out-f136 before google.com is a variable which they will change all the time. Quote Link to comment Share on other sites More sharing options...
arnold Posted November 15, 2007 Report Share Posted November 15, 2007 The simple solution is to add deny from google.com in the same place you added deny from googlebot.com.However, and this is a big, big, however, by denying access to google you are keeping your website from being indexed by the most popular search engine today. In your message, you wrote, "How can I stop them before they put my traffic in the red again." What do you mean by that? Quote Link to comment Share on other sites More sharing options...
janvorster Posted November 16, 2007 Report Share Posted November 16, 2007 The simple solution is to add deny from google.com in the same place you added deny from googlebot.com.However, and this is a big, big, however, by denying access to google you are keeping your website from being indexed by the most popular search engine today. In your message, you wrote, "How can I stop them before they put my traffic in the red again." What do you mean by that?Google was responsible for 2.15 gig not viewed pages while viewed pages was 500meg last month. Intresting most hits was from google search engine. Links from an Internet Search Engine - Full list- Google 689 689 - Yahoo 3 3 - Unknown search engines 2 2 - Tiscali 1 1 Which support your comment regarding using google to get people to visit your page. My situation however is different that my research is mainly on one surname and my domain has been on the internet for more than 10 years. My research goes back to 1721 and that is as far as the genealogy goes in my country. Most of the researchers already know my domain name and is listed on a few genealogy sites.As far as going in the red I had to pay significant penalties because my given trafic usage (2.5gig) was overshot only by google indexing every page. Of course I would like to get as most traffic as possible and the option is to get a better service provider with more traffic allocation but changing, can only mean trouble if I want to keep the same domain name. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.