Drop in # of Indexed Pages

[Ed Barnard]

Steve,

Thanks for the tip!

I'm not excluding anyone from looking at the navigation portions. Rather, I'm placing all that at the end of the html, so that the unique content is as close to the top of the page as possible. For example, look at the page source for:

http://www.ewbarnard.com/ancestry/getperso...30&tree=barnard

The javascript and navigation material comes AFTER the tables of info about my grandfather. To ease the formatting, I moved the navigation stuff *above* the person's name rather than below. I insert blank space above the name, and then that gets overlaid with the navigation as the page finishes loading.

You can see my style sheet here:

http://www.ewbarnard.com/ancestry/genstyle.css

I discovered that "position: absolute" in a <div id=> statement is actually relative to the enclosing div. So if the enclosing div begins 500 px down the page, the inner div starts counting 500 down the page.

That's why I use Darrin's $flags[more] feature - worked out beautifully. By putting the deferred html in $flags[more], I can spew it out AFTER closing all <div> ... </div> sections. Then I enclose the nav stuff in fresh <div> tags, which does absolute positioning from the top of the page.

In short, I write out the page with the unique content, and then go back to the top to write the logo, byline, navigation, tabs, and so on. I'm only doing this with getperson, since that should be the only thing the spiders see. A real human, of course, doesn't care!

My theory is that the closer the unique content is to the top of the page, the better.

If you view source on my domain root http://www.ewbarnard.com/ you'll see the same thing. I do the blog stuff then the surnames then the sidebar and heading/logo at the top. That's where I learned the trick - from somebody's WordPress blog template.

[sjwinslow]

Ed,

Very interesting. I've heard other people talking about the <div> technique but I had never understood how it was accomplished. Thanks for the info. It makes sense.

I'd be very interested to hear if you can quantify that the search engines place your pages in their index more frequently and give you better position because of this technique.

I see so many TNG pages that have been moved to supplemental it would be great to find something that works consistently. I was confident that the find.php pages were going to be indexed because of their uniqueness and thier content. But it appears that Google has a problem with so many links on a page (this is just a guess, I really don't have any proof as to why the find.php pages were removed for the Google index). My find pages continued to gain in number until a couple of days ago. Then within a couple of days they were all removed. I'm now waiting to see if this was some mistake on Googles part or if they are going to stay that way.

I'm not confident that we will ever be able to keep getperson.php pages in the index because of the duplicity between spouses and the high number of labels that are similar on every page. I think a different format will be needed to have long-term success.

One thing I have observed that seemed to work for one site's getperson pages was a lot of anatation on thier photos. Since the photos only showed that were associated with an individual and these photos contained a lot of text explaination, Google kept them indexed. Unfortunately, the vast majority of people on my site don't have photos and coming up with unique notes for each person will take for ever. I continue to look for some method that will allow our pages to be indexed without rebuilding the entire genealogy database.

Steve

[Ed Barnard]

Steve,

What, exactly, do you use as your google measure(s)? Best to compare apples to apples!

site:ewbarnard.com inurl:tng

yields 81 results at present. Wierdly enough most of them seem to be direct ancestors or siblings thereof. About 8000 persons in the database. Perhaps more, or more unique annotation? Who knows.

This weekend I cut over from ewbarnard.com/tng/ to ewbarnard.com/ancestry/ and

site:ewbarnard.com inurl:ancestry

gives zero results.

How do you tell a page is supplemental?

I still have 4000+ static html pages showing in site:ewbarnard.com - and of course would like to see the tng pages appearing!

Ed

[sjwinslow]

Ed,

Be careful using the site:ewbarnard.com inurl:tng because you may have a few tngprint pages that are duplicates, (all though your site is in better shape than most.) It would be best to add the slash at the end e.g., inurl:tng/

This is going to seem weird but this is the search string to identify supplementals for you old tng setup

site:ewbarnard.com inurl:tng/ -inutl:www (it also shows 81 but the two searches can be different)

No one know why adding the -inurl:www brings up the supplementals but it does. You can tell that they are supplementals because the word supplemental will be shown in green following your URL on each listing displayed.

Steve

P.S. Take a look at site:ewbarnard.com -inurl:www and see how many are there (2,340) and for

site:winslowtree.com -inurl:www there are 14,500 supplementals while site:winslowtree.com returns 19,000.

[Ed Barnard]

I'll try to track things, and have put up the page:

http://www.ewbarnard.com/supplemental.txt

for counts of pages in the index. We'll see!

Thanks again Steve and everyone,

Ed

[theKiwi]

P.S. Take a look at site:ewbarnard.com -inurl:www and see how many are there (2,340) and for

site:winslowtree.com -inurl:www there are 14,500 supplementals while site:winslowtree.com returns 19,000.

And what is even stranger is that my site is

http://roger.lisaandroger.com/

site:roger.lisaandroger.com -inurl:www = 16,600

site:roger.lisaandroger.com = 19,700

now having www as part of my site's address simply isn't possible, yet the numbers are different.

Roger

[sjwinslow]

Ed,

I was taking a look at your site and noticed that you still have quite a few static pages listed in Google's index you created with your Legacy software. You could significantly increase the traffic and page rank on your TNG pages by redirecting the traffic and page rank from the static pages using a 301 redirect via mod-rewrite.

If you are using Apache server software with the ability to do mod-rewrite you could route each of those static pages to your new TNG dynamic pages with a few simple rewrite commands. Since you kept the same ID number in your static pages and your TNG pages, each static page in your old pedigree directory could be redirected to the correct getperson.php page in TNG. Anyone linking to one of the old static pages would be routed to the TNG page seamlessly. Also any links referring to your old pages would also add to your link count for the new pages. Of course this is assuming you want to static page to eventually go away.

I'm not any good at mod-rewrites but there are people on this forum who are. I'm sure they would help if your interested.

Just a thought.

Steve

[Ed Barnard]

Thanks, Steve, good idea. In fact I was planning on finding that discussion about canonical url - I make no distinction between www.ewbarnard.com and ewbarnard.com. Maybe in the adsense discussion?

I do have apache and do know how to rewrite, so I'm cool there

However, I did renumber all my people before making the leap from "old" tng install to "new" tng install. That way I can more consistently ensure the person id's don't change on me in the future. So I can't do a direct conversion.

If I do a 301 redirect to, say, a fresh surname page, since it's a redirect, it shouldn't count against me as duplicate should it? Assuming not, how about make a "welcome" page that lets you navigate around the site, and...

Come to think of it, I could add the surname index to the bottom of the TNG welcome page at ewbarnard.com/ancestry/.

Thanks, Steve. This might be a good way to use those 4000+ cached pages which are, as you note, intended to be replaced by the TNG pages.

[sjwinslow]

For what its worth, here is a recent observation and my theory as to why Google reacted they way it did in removing all of my find.php pages.

Several weeks ago I developed the find.php page as a modified search.php page and used it to display all the people on my site listed by surname. My goal was to create a page that would be seen by Google to have much less duplicate content than the getperson.php page. To make this work on my site I placed a list of all surnames contained in my database on my homepage with links to the appropriate find.php page. This allowed both visitors and bots a short hop from the surname link on my homepage to the find.php page that ultimately linked to the getperson.php page.

Once implemented, Google very quickly crawled all of the find.php pages and they started showing up in Google's index. It took Google about three weeks to get about 2/3s of the 1600 surnames (and find.php pages) indexed. But rather than continuing to index the remaining 1/3, Google hesitated for about two weeks at the 2/3s mark.

I had always been concerned that the content of the getperson.php pages have significant similarity between pages and especially between spouses, preventing their indexing in Google. Since Google had stalled in indexing the find.php pages, I felt the cause was probably the getperson.php pages and not the find.php pages since each of the find pages was unique. I then blocked all indexing of the getperson.php pages, thinking that removing these duplicate getperson pages from Google's view would allow better indexing of the find pages.

Over the course of 2 weeks after blocking the getperson.php pages Google slowly removed all of the find.php pages from the index.

In an act of desperation, a couple of days ago I re-enabled indexing of the getperson pages and today 60 of the find.php pages are back in the Google index. I had made no changes to the indexing of the find.php pages either when they started being removed from the index or now that they are starting to come back (only to the getperson.php pages).

I'm not fully confident in my explanation as why Google is putting my find.php pages back in the index just because a subordinate page is now allowed to be scanned. However, here is my best guess as to why the getperson.php pages must be indexable for the Find pages to show in the listings.

We know Google weights the value of links highly in its determination to index a page. The find.php pages' content is a very high percentage of links and the anchor text associated with those links. If those links point to pages that can not be indexed then Google completely devalues both the links and the anchor text. Once Google does this to a find.php page there is almost no information of value remaining on the find.php page and Google removes it from the index.

This may have something to do with why Google only indexed about 2/3s of my find.php pages originally.

We know that the getperson.php pages appear to show high values of duplication to other getperson.php pages. Once Google determines a getperson page is a duplicate to another getperson page, then it not only devalues the getperson page but also the links that point to it. Once a find.php page is determined to contain mostly links to duplicate getperson pages, it too is devalued and removed from the index. If this theory is correct, we may have no choice but to change the format of the getperson.php page to make it more unique.

It will be interesting to see if the number of find.php pages continue to increase to the original value or if some new aberration takes its place.

Steve

[Ed Barnard]

Steve,

Might there be a possible benefit to (pardon the sexist suggestion here...) not indexing married women? Put a noindex,nofollow on the getperson page, and nofollow on the link off the find page? On the find page, perhaps even make it unlinked text?

Generally speaking, when I have information relating to the family unit, I put it in the husband's notes. So in my case I would give preference to indexing husband rather than wife.

Just thinking out loud here... :)

[sjwinslow]

Ed,

I have definitely thought about removing the spouse link on the find.php page. Since the spouse can be found as soon as the person goes to the getperson page. I initially removed most of the links that existed on search.php when creating the find.php page from the search template. I think with the latest observations that the spouse link should also be removed but leave the spouse name (just no link to their getperson.php page). However, there are many other links that cross reference the spouse and the positive effect of this may be very minor.

Before going to the trouble of trying to patch up the getperson.php page with nofollows to married women, I think there are more rudimentry problems with the structure of the getperson.php page.

It is very unfortunate that the getperson.php page is the keystone to genealogy information in our databases. The spouse similarities is just the most blatant problem of duplication we are discussing at the moment. If that were removed, then we would still be faced with all the repetitive labeling for such words as birth, died, sex, mother, father, etc. Even when comparing completely non-related individuals I can get as high as 80% similarity between them. When I look at Google's sitemap page analysis of my site, it indicates that 5 of the top 10 words Google identifies with my entire site are labels that can be found on the getperson.php page.

My hope was that the find.php page could summarize the genealogy information and present the most frequently searched componets to the search engines without duplication. And even though I believe the page has done that, its linkage to the getperson.php pages now seems to be its undoing.

It would be interesting if someone with good java skills could create code that would allow visitors to link from the find.php pages to appropriate getperson.php pages but keep the link hidden from the bots. If this could be accomplished then the find.php page would appear as a destination page to the bots and not a linkage page. Then blocking the getperson.php page from the bots would not have an adverse effect on the find.php page. Unfortunately, I know almost nothing of java script and it would take me a long time to develop such a page to be tested.

Steve

[Ed Barnard]

A quick look implies the ancestry.com trees are suffering too. Maybe the place to look at is getperson rather than find.php. Maybe present a different version to google which contains just the facts, no labeling?

[sjwinslow]

Ed,

I made a proposal to Darrin a month or so ago and he said he would look into it but I can understand that it's not one of his highest priorities. The proposal was a replacement that presented the basic genealogy information in a fairly standard graphical layout that makes the relationships intuitive to the viewer without labels. The extended information could be labeled with a single headline like Notes:, Sources:, Events:, etc. with the pertinent information under the headline. I think these few labels could be tolerated since they would only be used when needed and each implies additional text that would make the page more unique.

I think this format would retain the user experience while reducing the potential of duplication significantly.

I've been learning more php coding since I made the proposal. I may get to the point where I tackle a different format but the information presented in the getperson.php pages will be quite a challenge for someone with my abilities. In the mean time the find.php page is my best hope in the short term.

Steve

[Ed Barnard]

Hmmm... still thinking out loud... how about this? Surname list goes to find.php, and find.php links to a different getperson, e.g., gethusband.php. gethusband.php is a stripped down getperson.php just for google.

Meanwhile, use mod_rewrite to catch HUMANS and present them with the real getperson.php page. I assume mod_rewrite can detect that the user agent is NOT the google spider, and change gethusband.php to getperson.php?

The goal, then, is to get the find.php pages and the gethusband.php pages indexed. People would be searching on actual names, dates, places. They wouldn't be searching for the word 'surname', or the word 'name prefix'.

Another possibility would be to mangle getperson.php to use "is a spider" or some such flag, and if it's a spider, not produce navigation or formatting text.

[sjwinslow]

Ed,

Interesting thoughts. I especially like the last one. The tngprint of the getperson.php is already a page that contains no navigation and few links. And to think I've been going to extra lengths to try to block this page. It might be interesting to modify the find.php page to point to the print pages and then make the print pages an index but nofollow page.

I'm not sure I want to step into the realm of cloaking just to get my genealogy site to index.

Steve

[Ed Barnard]

(smacks palm to forehead) Duh! Great idea, Steve! And now, to quote the sidekick in the original Indiana Jones movie, "Snakes. Very dangerous. You go first."

Actually, I still have 0 ewbarnard.com/ancestry/ pages indexed, so I don't have too many results to watch :)

[sjwinslow]

Well I'm not sure I have too much to loose. Of the getperson.php pages indexed for my site (all of them currently supplemental) 5,510 are with tngprint and 10,300 are without (everyone of the 5,510 I have sampled are also included in the 10,300 [duplicates]). I would like to give my site at least a week to settle down after just allowing the getperson pages to be crawled again. After that I'm game. It's not like I make a living from this site and it might be different if I had all my getperson pages in the normal Google index. But with all of them setting in supplemental and my traffic setting at a barely 100 unique visitors per day, things could be worse but not by too much.

I also doubt this is the answer. Even with the stripped down version of the getperson page, we will still have the spouse duplicate problem and the label similarities. It might be enough to push a few of them out of the duplicate supplemental world but I'm not optimistic.

It does give me a thought that I should look at how the tngprint function works and maybe a new function could be built that condenses the information even further and would remove the labels (and maybe displayed only when Google bot was detected Hmmm...) That still leaves the duplicate spouse issue.

It always comes back to a new type of page is needed with few labels and all of the nuclear family displayed on the same page. Otherwise, Google is going to keep throwing our pages out as it does now.

Steve

[Ed Barnard]

I made a page:

http://www.ewbarnard.com/ancestry/layout.html

which shows how I have the page laid out with div tags. If you've struggled with css layout (as I have been), this might make sense, and help as a concrete example.

Also, today I found this tutorial helpful:

http://www.yourhtmlsource.com/stylesheets/csslayout.html

To keep on topic, remember my objective in messing with the page layout is to get the page's unique content at the top of the page, and everything else pushed to the bottom of the page. (Bottom of the page, meaning at the end of the html source.)

[sjwinslow]

Ed,

I would like to sincerely thank you for posting your information on DIV and CSS. It finally got me off of dead center and I converted all of my static html pages over to CSS and off of table structure. I really don't know if placing the content first increases the value of a page to the search engines but CSS certainly added a level of consistency (I'm not totally there yet) that my site lacked.

On another Note:

During my site revamp I also added several mod-rewrites that cleans up duplicate paths to my pages. Right now Google has dumped all of my find.php pages but Yahoo has picked up and indexed over 500 of them, so it's a wash. I think Google dumped them because of the duplicate pages it was seeing with the tngprint, print friendly pages. Hopefully, Google will figure out that the duplicate pages are now gone (due to the mod-rewrite) and place them back in the index.

I'm still waiting for Google to clean up the supplemental pages for my site. Google shows about 39,000 supplemental pages for my site at the time of this posting. If everything was cleaned up I would expect less than 12,000 pages if the site was fully indexed.

If I had it to do all over to do again, I wish I had blocked every bot from my site until I could get everything configured to keep the duplicate pages to a minimum. All of our TNG sites are being killed in the indexes by duplicate content.

Steve

[sjwinslow]

Finally some good news!

Google has just added 1093 find.php pages to its index for my site. That's about 75% of the total find pages available for my site. I believe the recent mod-rewrites included last month to remove all of the print friendly screens is the resolution. These duplicate pages (mostly caused by the tngprint function) have caused the removal of the find pages in Aug and for almost all of my getperson.php pages to go supplemental for the last several months.

The changes last month included a small modification to the find.php page that made the default to display the spouse and use the default tree. This allowed a much smaller URL that only includes the surname as the parameter. The mod-rewrites were then included to only allowed Google to use the new URL format and blocked the parameter tngprint=. This limited Google to only one possible URL for each find page and removed the duplicate filter had placed on so many of these pages.

The next steps are to ensure this find pages stay in the main index and see if I can get the getperson pages remove from the supplemental status.

Steve

[Russellgs]

Finally some good news!

Google has just added 1093 find.php pages to its index for my site. That's about 75% of the total find pages available for my site. I believe the recent mod-rewrites included last month to remove all of the print friendly screens is the resolution. These duplicate pages (mostly caused by the tngprint function) have caused the removal of the find pages in Aug and for almost all of my getperson.php pages to go supplemental for the last several months.

The changes last month included a small modification to the find.php page that made the default to display the spouse and use the default tree. This allowed a much smaller URL that only includes the surname as the parameter. The mod-rewrites were then included to only allowed Google to use the new URL format and blocked the parameter tngprint=. This limited Google to only one possible URL for each find page and removed the duplicate filter had placed on so many of these pages.

Steve,

What changes did you make? I have over 20,000 people in my database and can't seem to get google to index more than 50 pages . I have tried to implement your suggestions but nada I only saw one time where the index was over 1000 pages and that was a couple of days after I installed the find.php pages which was about the 15 of August.

HELPPPP

today it is now 70

[Barry]

[moved to a new thread]

[sjwinslow]

Russell,

I think it takes several things to get a large number of pages listed and keep them listed. First I managed to get several inbound links. Second, as a site ages Google tends to turst it more and it makes it easier to get and keep pages listed. Third, the hardest thing to accomplish was to limit Google to just the most important pages and make sure there was a single URL that would access them. It seems it is very easy for Google to turn TNG pages to supplementals when it finds multiple paths to the same page.

Steve