Drop in # of Indexed Pages

[thewag]

2. Present a differently-formatted page to the google spider. Leave off the navigation and all fanciness, pretty much leaving only the unique content for that person.

Hmm . . interesting idea. If your running php header and footer that would be real simple. Might have to give that a try when I get more time. Anyone with more time have thoughts?

[sjwinslow]

I checked my getperson.php pages and I only add 5 links that repeat across all the pages (Home, Search, Surnames, Contact and FAQ) and non of them show up in the sitemap common words. However, the getperson template has typically about 25 words that show on each page, 9 of which show in the sitemap common words. I don't believe if I remove the 5 of mine that I'll have much effect on how Google percieves the simularities of the pages.

Here's my disclaimer: Again this is all speculation about Google discriminating against pages with similarities. I'm only making observations and don't want to cause any concerns when there is no proof. I do however, appreciate every one's participation in this thread. We may never find out if there is even a problem that we can effect but I appreciate all the suggestions and good ideas.

Steve

[Jpaprocki]

Steve, have you checked how your site is doing with other search engines? I realize google gets the most traffic, but lots of times the higher you are in goodle the better you do in others and the better you do in others the better you do in google.

Other sites of mine have shown up in paid only inclusion sites, just because I was ranked within the top 20 in google. I don't mean they showed up under google results, it showed under their results.

[sjwinslow]

Jpaprocki,

My site's listing on the search engines are nothing to brag about on any of them:

Engine...........Site.........Inurl

Google...........306...........46

Yahoo..............41............0

Msn.................93............0

Altavista............0............0

Note: Site is any of my pages indexed and Inurl is the number of getperson pages showing.

You'll notice that the number of pages in Google are still dropping. Soon I won't have anything to worry about as I won't have any pages showing in the indexes.

Steve

[sjwinslow]

I'm trying a new strategy and I would like your opinion. Remember that I have Adsense on my site so this strategy might not be appropriate for those who don't. I have been recently reading that one of the reasons that forums don't index very well is that there are so many links within the forum that connect to the same information. This is seen as duplicate pages by the search bots. It seems to me that a similar situation exists for TNG. Maybe all these extra links are the reason that Google is seeing our pages as duplicates and not indexing our pages even though it is crawling them like mad.

I have changed both my robots.txt and the tngrobots.php. The changes to the robots.txt restrict any crawling bot to three pages: surname-all.php, search.php and getperson.php. The tngrobots.php file sets all TNG to "nofollow" except for surname-all.php and search.php that are allowed follow but "noindex".

Here is the reasoning: The goal in this effort is to keep the number of duplicate links as small as possible. All getperson pages can be found by doing a surname-all and a followup search. There is no reason to index the intermediate steps (surname-all and search) when what I want indexed is the getperson pages. There is also no reason to follow any links from the getperson pages since all getperson pages can be found through the surname-all/search process. I allow indexing on all pages (although most should not be accessible by the crawling bots) to allow the Adsense bot to index any page that a person lands on.

If Google is preventing our getperson pages from being indexed because of the similarity of the pages (as discussed in previous posts) then this setup won't help. But if the number of duplicate links was the problem then this should significantly reduce them.

Let me know what you think,

Steve

[theKiwi]

To counter this, try this Google search for Mungawera - a mis-spelled place name in my genealogy file. The only hits on it are to my site. It occurs once in one person's record in my database.

http://www.google.com/search?q=mungawera&filter=0

Google has 8 results for that. So repetition seemingly isn't hurting in this instance.

Roger

[sjwinslow]

Very interesting. Did you also notice that the last time your TNG pages were indexed, looks like in mid-2005 (That is based on the cache dates on some of the results returned by the search you provided) You might try searching for some new person you have added to your site. It would be a toss up if they are indexed. The problems we are seeing have to do mostly with new sites (ones placed in service in the last 6 months). Google seems to have changed the criteria on what they index and they grandfathered sites before that time that have made few changes.

Honestly, I'll be surprised if the new robots.txt and tngrobots.php changes I've made will make a difference. But if what Google requires now is for every one of my 10,000 getperson pages to be completely different before it will show it in their index... It will be a Loooong time before I can get that accomplished.

Steve

[theKiwi]

Well whatever is going on has hit me too now - last week my site http://roger.lisaandroger.com had something like 26,000 results in a Google search for

site:roger.lisaandroger.com

today ZERO!!! Same for my wife's page at lisa.lisaandroger.com

and the Clan Moffat Society site I run which last week had over 45,000 results now has just over 500 from site:genealogy.clanmoffat.org

Hmmmmmm - they must have had a huge clean out - yet the search for Mungawera works on their site. And searching Google for

Moffat Genealogy

my Clan Moffat Society site is the first hit.

Roger

[sjwinslow]

theKiwi,

I saw that when I was doing the other checks on your site pages and was going to comment on it but I thought it maybe because you use subdomains... but I guess not. Good luck, I hope your pages come back. It seems like some of the older sites do reclaim many of their pages when they get caught in what ever is going on.

Steve

[Rush]

I got two pages left... Down from 494...

Yup all two of them are still there

Rush

[thewag]

Ok, I just got this error:

URL unreachable /robots.txt unreachable

Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your robots.txt file was unreachable. To make sure we didn't crawl any pages listed in that file, we postponed our crawl. When this happens, we return to your site later and crawl it once we can reach your robots.txt file. Note that this is different from a 404 response when looking for a robots.txt file. If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl.

Interesting.

[sjwinslow]

Roger,

Do you know who provided the error? Google? Server? Some other bot?

Steve

[sjwinslow]

After making changes to both the robots.txt file and the meta tags the search bots seem to be behaving themselves better than I've ever seen. To build the robots.txt file I took a simple walk through my site similar to the way a bot would and noted all the possible links they might take. This made my new file much more concise, rather than just throwing all the TNG filenames into the large list like I had before. I needed to do this because the robots.txt file I had prior was so large that I couldn't add some of the disallows I needed for my forum. (Google give errors if the robots.txt file is over 2K)

With a much cleaner robots.txt file the meta tags were set to support it. I ended up setting all TNG files to nofollow & noindex, except the surname-all.php and search that were set to just noindex and the getperson.php that was set to just nofollow. I confirmed that the noindex did not prevent the Adsense bot from reviewing my pages to support the ads.

Before I always saw the bots wandering into pedigree and descendant areas of my site even with those pages disallowed in the old robots.txt file. During the last three days of continual crawling, I haven't found one doing a mis-step yet.

This has not increased the number of pages included in the Google index significantly yet. It seems to have stopped the decline, but more time will be needed to know for sure.

Steve

[waltreams]

Wow. Great idea to arrange the files like that. You'll be an SEO consultant before long!!

Actually this thread has been a goldmine of info for search optimization.

Out of curiosity, how is your site performing on ask.com? I think a lot of people are realizing that google is all about the $$ and will be migrating. Ask is positioning itself to fill the void.

~wr

[steve30000]

Well, I had dropped down to zero pages indexed for most of last week. Showing up again this weekend with 350 pages but they are all old pages from June 2005.

And now I have the strange feeling that the googlebot is following me...

Sun 07 May 2006 01:18:02 PM Individual info for Pierre "Sieur de LaPierre" Roberge (I149) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:18:02 PM Individual info for Pierre "Sieur de LaPierre" Roberge (I149) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:57 PM Individual info for Ambroise Roberge (I162) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:57 PM Individual info for Ambroise Roberge (I162) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:54 PM Search Results for Last Name equals ROBERGE AND tree equals Hoover Family accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:54 PM Search Results for Last Name equals ROBERGE AND tree equals Hoover Family accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:42 PM Individual info for Basil L. Wagoner (I824) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:42 PM Individual info for Basil L. Wagoner (I824) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:40 PM Pedigree for Mary Anne Wagoner (I469) 4 Generations accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:40 PM Pedigree for Mary Anne Wagoner (I469) 4 Generations accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:36 PM Individual info for Mary Anne Wagoner (I469) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:36 PM Individual info for Mary Anne Wagoner (I469) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:34 PM Individual info for Ross McPherson Thompson (I458) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:34 PM Individual info for Ross McPherson Thompson (I458) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

Sun 07 May 2006 01:17:32 PM Individual info for Mary Alice Thompson (I3429) accessed by crawl-66-249-72-43.googlebot.com.

Sun 07 May 2006 01:17:32 PM Individual info for Mary Alice Thompson (I3429) accessed by 69-162-47-196.stcgpa.adelphia.net (User: Admin).

[theKiwi]

Well whatever is going on has hit me too now - last week my site http://roger.lisaandroger.com had something like 26,000 results in a Google search for

site:roger.lisaandroger.com

today ZERO!!! Same for my wife's page at lisa.lisaandroger.com

and the Clan Moffat Society site I run which last week had over 45,000 results now has just over 500 from site:genealogy.clanmoffat.org

Hmmmmmm - they must have had a huge clean out - yet the search for Mungawera works on their site. And searching Google for

Moffat Genealogy

my Clan Moffat Society site is the first hit.

Roger

Google must have been in the midst of a large update last week which took a while to put together?

I checked tonight, and

site:genealogy.clanmoffat.org = 224,000 (two weeks ago ~45,000, last week 500 or so)

site:roger.lisaandroger.com = 19,700 (two weeks ago ~26,000, last week zero)

I'm BACK!!!!!

Cheers

Roger

[arnold]

With a much cleaner robots.txt file the meta tags were set to support it. I ended up setting all TNG files to nofollow & noindex, except the surname-all.php and search that were set to just noindex and the getperson.php that was set to just nofollow. I confirmed that the noindex did not prevent the Adsense bot from reviewing my pages to support the ads.

Would you mind sharing with us what you entered for the robots.txt file and for the meta tags?

[sjwinslow]

Roger,

I hope your pages are really back. I checked mine last night and saw ~15,800 on my site. But when I did a closer look by clicking on the 10th page Google presented, I found that only about 120 were really indexed (most of those are marked supplemental). The estimated number that Google supplied was just way off. now this morning I have less than 100 pages indexed and only 4 getperson page remain. Its a good thing I have a number of loyal returning users or my site would be getting no visits at all.

arnold,

The bots seem to be following the robots.txt/meta instructions better than they have before. I was always having trouble with them indexing stuff that were in areas I had restricted. How the vast majority are looking in the right places and they seem to be getting better all the time. I only found one time in the last several days that they tried to index a timeline they shouldn't have.

You can see my robots.txt file by clicking on http://winslowtree.com/robots.txt

and here is a copy of the tngrobots.php

<?php
if( !$cms[support] )
    $tngscript = basename( $SCRIPT_NAME, ".php" );
else
    $tngscript = $file;

//No index only
$NOI = "<meta name=\"robots\" content=\"noindex\">\n";

//No follow only
$NOF = "<meta name=\"robots\" content=\"nofollow\">\n";

//No index AND no follow
$NOINOF = "<meta name=\"robots\" content=\"noindex,nofollow\">\n";

//each "case" is the name of the script file without the ".php" at the end
switch( $tngscript ) {
    // no follow
    case "addnewacct":
    case "ahnentafel":
    case "anniversaries":
    case "browsetrees":
    case "changelanguage":
    case "descend":
    case "desctracker":
    case "gedform":
    case "login":
    case "newacctform":
    case "pedigree":
    case "pedigreetext":
    case "places-all":
    case "places-oneletter":
    case "places":
    case "placesearch":
    case "places100":
    case "register":
    case "relateform":
    case "relationship":
    case "searchform":
    case "sendlogin":
    case "showlog":
    case "suggest":
    case "timeline2":
    case "ultraped":
    case "whatsnew":
    case "browsedocs":
    case "browseheadstones":
    case "browsenotes":
    case "browsephotos":
    case "browserepos":
    case "browsesources":
    case "browsetrees-old":
    case "extrastree":
    case "reports":
    case "showreport":
    case "surnames100":
    case "cemeteries":
    case "familygroup":
    case "headstones":
    case "showheadstone":
    case "showmap":
    case "showphoto":
    case "showrepo":
    case "showsource":
    case "showtree":
    case "surnames":
    case "surnames-oneletter":
        $flags[norobots] = $NOINOF;
        break;

    //allow follow but no indexing
    case "surnames-all":
    case "search":
                            $flags[norobots] = $NOI;
        break;

    //all pages not named get full indexing as well
    //no pages come in here unless they include genlib.php
    default:
        $flags[norobots] = "";
        break;
}
?>

Well, I had dropped down to zero pages indexed for most of last week. Showing up again this weekend with 350 pages but they are all old pages from June 2005.

And now I have the strange feeling that the googlebot is following me...

Steve30000

I believe that is because you have Adsense on your site. The Adsense bot takes a quick look at your site (if it doesn't have information already stored about the page) and tries to determine what the context of the page is so it can display the correct ads.

Steve

[sjwinslow]

Out of curiosity, how is your site performing on ask.com? I think a lot of people are realizing that google is all about the $$ and will be migrating. Ask is positioning itself to fill the void.

~wr

Walt,

Not well at all. (For that matter I wouldn't say that my site is doing good in any of the indexes) I know I'm being a little impatient as my site has only been in the indexes since mid Feb. and many others have been out there many months longer with only a home page indexed. I also read in the Web Master forums that it takes over a year to get a new site to index in the search engines.

As I mentioned earlier, its a good thing that I have several loyal return visitors or it would be just me looking at the site (me and the Google bot scanning a hundred pages per hour, that is )

My hope is to use this time as a learning period. I already have discovered a number of things that not only work for the search engines but it works well for my users. Also some of the same tracking metrics I use to monitor how well the AdSense is working has also told me how I need to improve my forum navigation and what interests my users in my newsletter. I've also found a few mistakes I made that I can pass along to others so they can sidestep them. I'm still kicking myself for changing the folder name where the TNG files reside. I thought the word genealogy was too long and too hard too spell, so I changed it to "tree". That little mistake sends totally the wrong message to the search engines. The search engines now think my site has something to do about gardening and tree pruning. My site has very specific content and a very specific audience. The more I can align it to that target the easier it will be for people to find the site. So now I'm gearing up to move the files back to the genealogy folder.

Steve

[sjwinslow]

During the last several weeks that I have been tracking the loss of pages displayed in the Google index my site has been experiencing, I have been using Darrin's personal genealogy pages as a benchmark. I felt that his site was probably one of the oldest TNG sites and was somewhat close in size to mine. In the past his site typically has shown about 17,000 getperson pages with a little over 14,000 individuals in his database. Today when I checked getperson pages on Google, it now has only 768, a loss of over 16,000 pages in just a few days.

It is looking like no site is immune to the restructuring that Google is going through.

Steve

[sjwinslow]

It's been quite a ride with Google through the month of May. However it looks like adding the surnames to my home page has pulled the site out of the darkness. Here are the page counts for the total of pages indexed on Google, the number of getperson pages and the number of find.php pages. The find.php page is a modified search.php to show name, birth date/location and spouse for all people by surname that I added on May 23th. Also on the 23th the home page was modified to include all of the surnames on my site with links to the appropriate find.php page. These changes were made because I notice with the recent "Big Daddy" Google release, Google was only indexing to the 3rd level of my site. This is what was causing the loose of all my getperson pages (they were at 4th level). The new flatter navigation allowed all people in my database base to be indexed at the 2nd level using the find.php pages and placed getperson pages at the 3rd level.

Date_____Total___getperson___Find

10-May____87________4

11-May___100________3

12-May____98________3

15-May___158_______13

16-May___157_______15

17-May___166_______15

18-May___172_______18

22-May___139_______10

23-May___151________9________0

24-May___154________9________0

25-May___874________9______663

26-May___924________7______783

30-May___955_______14______785

31-May_10500_______14______991

Note that the number of getperson pages have not increased significantly since adding the new find.php. About 80% of my find.php pages are indexed after about a week even though there has been no improvement in the number of getperson pages indexed. I doubt that all of the 10500 total pages shown for my site are real searchable pages but after several tests it does look like most of them are.

Steve

[sjwinslow]

A number of people have asked me about the process of adding the Surnames to the home page to increase the number of pages indexed in Google. I will try to summarize it here as the comments on the subject got scattered between this post, the Adsense post and the mailing list.

This discovery came from three months of trial and error and observing the reaction of the Google index to the changes I made. About Jan. of 2006 Google made a significant change to the architecture of the system it uses to index pages. It is believed this is the most complex algorithm any search engine has ever placed in service. There is a lot of speculation on the web as to what you can do to help get your pages indexed, however, I can’t comment on most of them. The information presented here is based on testing on my TNG site.

First, my site is a combination of html pages a PHPBB forum and TNG. At the time of writing I would estimate that there are about 150 html and forum pages and the TNG portion of the site has about 10,500 individuals.

During the testing phase I found that the following things had no discernible effect on the indexing of my site. These included duplicate content, page errors, age of site (due to the short test period there could be long term effects though), static versus dynamic content or the amount of content on a page.

The things I found that did have an immediate effect on page indexing were; Page Rank (PR), inbound links and page depth. These items need a little more clarification. Page Rank or PR, like most things Google creates, has little known about what determines it. It seems to be closely related to the number and quality of inbound links pointed to the page. The PR tool that Google supplies gives a gross indication of the relative value of PR but Google seems to keep another internal number that it never shares with people. An example of how to improve your page’s PR would be to have several sites that have higher PR than your page, link to it. Over a period of time Google will notice these links and raise your page’s PR. There is a whole list of items that are speculated that may also effect a page’s PR, but that is beyond the scope here. However, PR is important in that if you have no or low PR, Google will not index beyond you home page many times.

Page depth, seems to be the one item that became very sensitive in the Jan. changes that Google made. Page depth is the number of clicks a search engine must connect through to reach any page on your site from the home page. On my site my desire was to get all of the getperson.php pages indexed because they contained the detailed information on each of the individuals. These pages were located on the forth level of my site (or three clicks away from the home page).

I found that my site was very well indexed on Google for the first three levels of my site. However, the forth level only had a hand full of pages out of the 10,000 I desired. This observation that the depth a page was away from the home page is what determined if it was or was not to be indexed, was the motivation to include all the surnames on my home page.

This new set up involved two major changes to my site’s navigation. First, every surname was placed on the home page and linked to a new summary directory I called find.php. The find.php page is a modified search.php page that shows all the individual names, birth dates, locations and souse for each individual belonging to that surname. You can see how this is set up on my site http://www.winslowtree.com by scrolling down the home page and clicking on one of the surnames listed there.

The reason the find.php page was developed was to remove the extraneous Person ID, Tree Name and place links. It also enhanced the titles and meta tags to make each page unique by adding the surname to the titles and descriptions. Additionally, the find.php page defaults to displaying 600 rows per page rather than the normal 50 rows. Note: this goes against Google's recommendations of having only 100 links per page.

With all of the surnames on the home page and each of the surname linked to a find.php page that shows a summery of every individual for that surname, every person on my site can be reached in just one click from the home page (or level two). The detailed information contained in the getperson.php can be reached from the links on the find.php pages, putting the getperson.php pages on the third level.

The results achieved from this test are still preliminary. The modifications to my site were made on the night of May 23rd and the Google bot started indexing the find.php pages on May 25th. Below is the latest information on the number of pages indexed in Google for total pages, getperson.php pages and find.php pages.

_Date_____Total__getperson_____find

15-May_____158_____13__________0

16-May_____157_____15__________0

17-May_____166_____15__________0

18-May_____172_____18__________0

22-May_____139_____10__________0

23-May_____151______9__________0

24-May_____154______9__________0

25-May_____874______9________663

26-May_____924______7________783

30-May_____955_____14________785

31-May___10500_____14________991

1-Jun_____13800_____14_______1000

2-Jun_____16400_____40______15400

5-Jun_____20900_____83______17700

6-Jun_____21200_____83______17900

7-Jun_____21300_____83______17900

8-Jun_____23300_____89______21500

I’m not a master coder and can not help you much if you decide to implement this approach on your site. I obtained the surnames by capturing the display of the surname-all.php page and cut and pasted the html to my home page (my home page is html) and tweaked it a little so it would link to the find.php page. I have attached the find.php page I created to this post so you can download it if you like. You will need to be sure and change the hard-coded $find_url to point to your site. I had to hard coded this since I didn't want to make major modifications to TNG. Also the links from your surnames need to be in the format:

http://www.yourdomain.com/your TNGfolder/find.php?mylastname=YourSurname&lnqualify=equals&showspouse=yes

(The items underlined need to be set for your site and each surname.)

This will display all the individuals for your surname with name, birth, location and spouse.

You can find additional notes at this post:

http://www.tngforum.us/index.php?s=&showto...findpost&p=5342

There is no way to be sure this approach will continue to work with Google. Google is continually updating their indexing algorithm. However, I found several benefits to having all of the surnames listed on my home page. These benefits included increased traffic from Yahoo and MSN and the number of page bounces on my home page was reduced. The additional traffic from Yahoo and MSN is believed to be from people searching for other surnames other than the primary surname of my site. Both of these search engines have few of my pages listed other than the home page. The reduction in page bounces is believed to be that people landing on my home page have many more options for find people they are interested in and don’t leave the site immediately.

Steve

UPDATE: An enhanced version of the find.php file was uploaded 8/10/06 with the following changes.

- The meta title and description was changed from [surname Family] to [surname] to reduce duplication.

- Defaults changed so if only the surname is included, name, birth date/location and spouse are displayed.

- The Name displayed changed from reversed to normal format for better search engine recognition.

- If no surname is included, the [nosurname] individuals are displayed.

find.php

[theKiwi]

I’m not a master coder and can not help you much if you decide to implement this approach on your site. I obtained the surnames by capturing the display of the surname-all.php page and cut and pasted the html to my home page (my home page is html) and tweaked it a little so it would link to the find.php page. I have attached the find.php page I created to this post so you can download it if you like. You will need to be sure and change the hard-coded $find_url to point to your site. I had to hard coded this since I didn't want to make major modifications to TNG.

So one question I have is from the above paragraph....

It seems that the list of surnames on the index.html page is static? So you have to recreate it each time you update the database?

Is it possible to have this dynamic or impossible?

Thanks

Roger

[arnold]

Your question certainly applies to our website where we attempt to update on a monthly basis and we have about 17,000 individuals with the surname of Sprague. With each update we add around 2,000 individuals, a fair number named Sprague.

Making it dynamic would save our bacon.

[sjwinslow]

So one question I have is from the above paragraph....

It seems that the list of surnames on the index.html page is static? So you have to recreate it each time you update the database?

Is it possible to have this dynamic or impossible?

Thanks

Roger

Roger,

On my site it is static. That is the reason I continually caution people to think about the ramifications before they implement this solution.

However, if someone would take the surnames-all.php script and make a few modifications and create a new file, this new file could be made to work with the find.php instead of the search.php. This new file could be added to a php home page with an include statement and you would have a total dynamic solution.

Steve