For all you 'Googlemaniacs'

[Rush]

I worked on a little page to make searching Google for your TNG pages a little easier.

This will display the total number indexed for certian pages.

The script is a little clunky right now and a few things need tweaked up and added.

When you bring it up the first time, it runs a empty search, but just type in you TNG website: www.mytngsite.com and hit the search button.

Here's the page

Rush

[sjwinslow]

Wow Rush it responds very quickly. What commands did you use? I noticed I get a different response when I include the www.

Steve

[Rush]

I used (in order)

site:

site: inurl:getperson.php

site: inurl:familygroup.php

link:

I just got tried of checking them all manually, so I thought...what the heck

Rush

[Russellgs]

I used (in order)

site:

site: inurl:getperson.php

site: inurl:familygroup.php

link:

I just got tried of checking them all manually, so I thought...what the heck

Rush

Wow, kinda neat but how can I get mine to be more then just 4 pages indexed????

I put in rgtrong-genes.com and it says only 4 for site, 2 for getperson.php, and 0 for familygroup.php

Russ

[tmaguirre]

Wow, kinda neat but how can I get mine to be more then just 4 pages indexed????

I put in rgtrong-genes.com and it says only 4 for site, 2 for getperson.php, and 0 for familygroup.php

Russ

Have you submitted a sitemap to Google? That seemed to really help my site, along with Russ' idea about google notebook -- what I did to experiment was pick a couple of "way back" relatives and in TNG display the descendants page, and then I copied that into google notebook, and within a day or two all the people listed as descendants started showing up in the Google searches. Seems Google found the links in the Notebook and indexed them that way?

[sjwinslow]

I just finished reading a very interesting article on page rank theory and how to apply it to your site. Here is the link: http://netsavy.net/Graphics/PageRank.pdf The part I was most interested in was how much site reviews could help your page rank.

The article is 21 pages long and you may not want to wade through all of it. But the jest of the article is that each of your pages listed in the index has some level of starting page rank. Based on the navigational structure of your site you can focus your internal page rank and that received from other sites to get the most out of what you have.

A hierarchical navigation structure will tend to focus page rank on the homepage, where and interlinked structure will tend to spread that rank more evenly. This may seem obvious but if you're like me you have done little with your site navigation to focus the page rank your site currently has to the pages you most want listed in the indexes. (If your able to get hardly any pages listed at all )

There are also suggestions on how to set up a site map (not Google's sitemap.xml) that will enhance the page rank of all the pages it points to while keeping the page rank low on the site map its self. Here the suggestion was to break up your site map in to functional groups and create pages for each group. While each of the site map group pages will have links to the functional areas, be sure to include links back to your most important pages too. This will ensure that your important pages will receive an additional share of the site maps page rank.

I know many of us belong to the TNG webring. But there was an interesting section on doing site reviews. Once a site review is created and indexed the page will have a page rank. For us, I see an number of benefits by reviewing each others sites to increase everyone's page rank.

First, there are obviously a lot of very creative and informative TNG sites that deserve recognition. Second, I think a site that contained a section that objectively reviewed other sites, adds a level of credibility and authority to the reviewing site while acknowledging the attributes of the site being reviewed. Finally, the creation of a review will produce an increase in rank for both the reviewing and the site being reviewed. Not bad for a couple of paragraphs and a link.

Anyone interested in swapping site reviews, please contact me via email. sjwinslow@gmail.com

Steve

[Rush]

My Google Notebook links are dropping out of the index.

I can't say I'm suprised, but a little disappointed. At least the Notebook stays out there and is index. So that can be a weird sort of way to get to the site.

I would be more than happy to share any site reviews, but unfortunately, my writing composition stinks.

Though, rank in listing doesn't concern me to much, I would just like to actually get indexed and stay there

My backlinks haven't changed in months, for the good or the bad...

On a side note:

Oddly enough, my Altavista / Yahoo links went from about 60 to 39,900 in one night. Hmmm....

Rush

[sjwinslow]

Rush,

Let's make this easy. Since you know your site, how about we just trade reviews. I'll write up one on my site and send it you and you can do the same. We can expound on the areas that we feel places our sites in the best light. I think we can keep the size at about 2 to 3 paragraphs (larger if you need). Once I get your info I create an area for reviews and maybe we can get this started easily.

The reason I'm interested is I only have 16 links showing in Google. Several of them are from either the web ring or other TNG users. Yahoo shows several hundred back links for my site but Google seems to value other genealogy sites much higher than any other links I've obtained. Related links may be the only way we can get Google to index our pages and keep them indexed. I've had similar problems with some of my pages showing in the index for a few weeks and then slowly disappearing. I really think it's because I just don't have the page rank to support the pages in Google. We have the perfect opportunity here to help get the TNG community indexed.

Steve

Rush,

On a side note, you may need to remove the ".php" from you inurl:gerperson.php and inurl:familygroup.php to get an accurate reading now. It seems Google is messing around with their search commands again.

Steve

[Rush]

Sounds good to me!

I think I have tried just about everything else. I know Google loves content, content, content...

Maybe content with relevent links (deep links) could help.

Google just did a backlinks update, but well, nothing improved here

I can write something up and have it over, say tomorrow or Thursday.

You can send yours to genealogy@bythedrop.com

Are still see a steady decrease in indexed pages?

Rush

[sjwinslow]

My site has been on a slight increase in the number of find.php pages lately (but I'm not optimistic that they will stay) However, the 14,500 getperson pages that were all supplemental have been dropping at the same time. From my stats it's been difficult to tell if people can even find my site from supplemental pages so I don't know if all these pages have been doing be any good. I know when I try to do a search for something unique on one of my supplemental pages, Google rarely returns the page.

Google can return people located on find.php pages that are indexed. So I would rather have them in the index that the supplemental getperson pages. I'm just concerned that something will happen that I'll loose both sets of pages.

Steve

[thewag]

LOL, I find this interesting....if I do genealogy.wagners.homeip.net...I get 26 pages, 26 getperson, 26 familygroup. If I do wagners.homeip.net I get 181 pages, 179 getperson, 181 family groups. Searching through...I don't see 179 getpersons or 181 family groups. However, I see a bunch, BUNCH, BUNCH of seriously outdated links. Maillisting I haven't hosted for over 2 years....software that hasn't run in over 1. And the cache of the few pages I do have are oh! so! out of date by three or four version ago.

I really should ditch google.

Ben

PS, Rush...I don't think there is an error in your code..think google is **insert choice word**. What a shame.

[Rush]

I think Google ditched me Again...

I know you don't think its the code, but I just had to make sure, so...

I checked your results against a straight Google search and they do come pretty close. About a difference of 2, which could be a result of Google data centers (DC).

I had pages that didn't exist for years in Google also. I blocked them using htaccess so they would kick out a valid 404 error in the header and get removed from Google. Also requested the pages be removed through Google.

To my suprise, they disappeared in a few days...then came back about a month later

So let me see if I can figure this out. Pages that don't exist for years and cause 404 errors get drop out and get re-indexed, pages that are current, updated and externally linked drop out of the index in a few weeks.

Yup, makes total sense...

Rush

[thewag]

So let me see if I can figure this out. Pages that don't exist for years and cause 404 errors get drop out and get re-indexed, pages that are current, updated and externally linked drop out of the index in a few weeks.

Yup, makes total sense...

Hi, my name is Darel, this is my brother Darel, and my other brother Darel.

Yup.

[Micheal Mathews]

Hey Steve:

The article is 21 pages long and you may not want to wade through all of it. But the jest of the article is that each of your pages listed in the index has some level of starting page rank. Based on the navigational structure of your site you can focus your internal page rank and that received from other sites to get the most out of what you have.

With all due respect to all of the hard work and education you are providing to other TNG webmasters, I have to step in here and say that there is way too much emphasis on webmasters going for Page Rank versus building a useful site full of content for users, which is what Google wants -- content. When webmasters start building for Page Rank, then they are getting close to stepping into the manipulation arena.

Also, I really don't think that the highly controversial Jill Whalen (whom I happen to agree with on one point -- that content is king) or any other SEO "expert" can even begin to know how Google determines Page Rank, especially from a document that is almost five years old. There have been umpteen revisions to Google's algo since November of 2001. One very important thing that TNG webmasters have to realize is that genealogy sites ARE NOT ecommerce sites, and so what works for ecommerce, in all reality, is most likely not going to work for us. Keywords and keyphrases for generic genealogy sites? Ouch. I concede that arena to MyFamilyInc and instead focus on the searchers who want to know about George Michael Eller.

BigDaddy speaks:

Give visitors the information they're looking for

Provide high-quality content on your pages, especially your homepage. This is the single most important thing to do. If your pages contain useful information, their content will attract many visitors and entice webmasters to link to your site. In creating a helpful, information-rich site, write pages that clearly and accurately describe your topic. Think about the words users would type to find your pages and include those words on your site.

Also, links are good -- if they are natural:

Keep in mind that our algorithms can distinguish natural links from unnatural links. Natural links to your site develop as part of the dynamic nature of the web when other sites find your content valuable and think it would be helpful for their visitors. Unnatural links to your site are placed there specifically to make your site look more popular to search engines. Some of these types of links (such as link schemes and doorway pages) are covered in our webmaster guidelines.

TNG webmasters -- forget Page Rank -- get rid of the stupid Google tool bar with the green indicator that seems to consume webmasters and focus instead on building a great content rich site. Instead of having mostly empty pages with names and dates, get some source material, research notes and event data into your primary pages. Write historys, essays, etc. and so on. Make them visible to your visitors.

Did I mention that content is king?

Regards,

Micheal Mathews

[sjwinslow]

Hey Steve:

... I have to step in here and say that there is way too much emphasis on webmasters going for Page Rank versus building a useful site full of content for users, which is what Google wants -- content. When webmasters start building for Page Rank, then they are getting close to stepping into the manipulation arena...

Regards,

Micheal Mathews

If I gave the impression that content wasn't one of the more important aspects of getting pages listed, it was unintentional. I realize that I have literally 1000s of pages that need additional content to be considered valuable to the search engines as well as visitors. However, while I'm working on obtaining additional sources, notes and other text that is meaningful to recording an ancestor, I also understand many people's discouragement with Google not listing the many individuals, represented by getperson.php pages, that do have significant documentation.

If Google only discriminated against the pages with low content then the focus could be much simpler. Other things like duplicate content and obtaining inbound links (which relates closely with the "real page rank" not the blue tool bar) must be considered as well. It would seem that Google has stepped up the importance of the "real page rank" since not only does it control your position in the listings but it also has an effect on the depth of crawl and the number of pages that are listed for a site.

You can have the best content in the world but if you don't have the "real page rank", a trusted site, no duplicates, valuable inbound links etc., etc., etc. those pages will never see the light of day, regardless of their content.

I'm at a point that I know that I have several hundred getperson.php pages that have content that should be considered adequate to be listed on their own merit. If I can't get those pages to show in the index (or at least get them so people can find them), me adding new relevant content to an additional 50 pages over the next year will have little effect. I believe, I, like most people doing genealogy, will continue to build their database as they find the documents and do the research that supports additional material. I also believe there are a significant number of people who put their research on the web to share with other people doing research. I've only posted the information I've learned about search engines on this forum because I doubt that I could help very many people with their genealogy research. But maybe I can save them some time in getting their TNG website listed so other people can share their research.

Please don't take the above as an attack or even a rant (Many times a good discussion of ideas may sound emotional in writing when, in reality it is only the limitation of the medium).

Steve

[Micheal Mathews]

Please don't take the above as an attack or even a rant (Many times a good discussion of ideas may sound emotional in writing when, in reality it is only the limitation of the medium).

Steve

Hi Steve:

No problem at all, and I agree about exchanges getting taken out of context. I will say it again, you have been of great benefit to the TNG webmasters getting their Google sorted out.

I am reloading my static site again and will relegate the TNG site to secondary status as a choice to my users. I can not say for certain, but I am a bit suspicious that my switch to TNG and a dynamic site came just before the BigDaddy rollout that relegated everything to supplemental. Now, the trick will be to keep all the search engines out of the dynamic side so we don't get the duplicate penalty. Unfortunately I will probably take a hit for a major site overhaul and get the non-existant sandbox, but at this point I can't see it mattering -- unless a user enters a word for word search term my results are no longer showing. It will be interesting to see if static pages can hit the index in a timely manner without being supplemental.

Great stuff Steve, thanks!

Micheal Mathews

[sjwinslow]

Micheal,

I'll be very interested to hear how your static page works out (be sure and keep us posted). I would really like to see a real live comparison with real stats. I've been on the fence on this one (as well as both sides). Currently, I'm leaning toward the side of, "dynamic links don't matter if there not too long" but there are a few caveats. Looking over my logs I've noticed with a dynamic site it is very easy to have multiple URLs point to the same content. Since I only have one tree I have seen the following in my logs where Google has been looking for the following URLs.

http://winslowtree.com/tree/getperson.php?personID=I1
http://winslowtree.com/tree/getperson.php?personID=I1&tngprint=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow
http://winslowtree.com/tree/getperson.php?personID=I1&spouse=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&tngprint=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&spouse=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&tngprint=yes&spouse=yes

They all point to the same content; and we wonder why our pages go supplemental :roll:

[Micheal Mathews]

Micheal,

I'll be very interested to hear how your static page works out (be sure and keep us posted). I would really like to see a real live comparison with real stats. I've been on the fence on this one (as well as both sides). Currently, I'm leaning toward the side of, "dynamic links don't matter if there not too long" but there are a few caveats. Looking over my logs I've noticed with a dynamic site it is very easy to have multiple URLs point to the same content. Since I only have one tree I have seen the following in my logs where Google has been looking for the following URLs.

http://winslowtree.com/tree/getperson.php?personID=I1
http://winslowtree.com/tree/getperson.php?personID=I1&tngprint=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow
http://winslowtree.com/tree/getperson.php?personID=I1&spouse=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&tngprint=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&spouse=yes
http://winslowtree.com/tree/getperson.php?personID=I1&tree=Winslow&tngprint=yes&spouse=yes

They all point to the same content; and we wonder why our pages go supplemental

Steve:

I was reading on the SEO sites that that dynamic sites with an ID string in the URI may complicate indexing. The emphasis on the word may. I'm not up to speed yet, but still researching on whether the personID=[string] would be not so good when it also gets to the point that it includes the &tree=Winslow&tngpring=yes&spouse=yes?

I'm hoping to have the static site back up tomorrow -- now I am debating on how to handle the dynamic site -- through a subdomain or folder.

Regards,

Micheal

[sjwinslow]

Micheal,

If your dynamic site has any pages in the index (real or supplemental), Google may see these as duplicate content to your static site even if the site no longer exists. I believe even if you don't have any pages indexed but G is crawling your pages, these un-indexed pages can be use to determine if duplicates exist.

I've recently noticed that G is continually trying to crawl pages that have been blocked with both robots.txt and tngbots.php. So G knows those pages were once out there and knows their content even though they are not in the index. Also, I had a full sitemap.xml file and now even though the sitemap.xml has been removed, I still receive errors in G's sitemap tools that I have pages listed in my sitemap that are being blocked by the robots.txt file. There are also URLs listed as being in my sitemap.xml as being blocked that I never included in my sitmap.xml.

My point is that it may be difficult for you to put up a static site and have it effectively indexed if you recently have had a dynamic site, using the same genealogy information, being crawled.

My old domain (it was free and I was using their sub-domain) shut down in Jan of this year, G still has over 4,500 TNG type pages listed in its index from that site. I had to do a mod-rewrite (the site came back after being out of service for over a month and after I moved my site) to redirect those pages to my new domain and to get G from seeing those pages as duplicates of my current site.

I have even gotten very conservative about making organizational changes to my current site. I recently added a number of new html content pages and didn't think through their organization very well. I was working on reorganizing it last night when the thought came to me that G may already have those pages indexed and would see me moving exiting files to new folders as potential duplicate content. Sure enough, the new pages were already in the index, so I just left them where they were. I'll do a major reorganization one day and just take the hit for it, but for now I'll learn to think through my organization design before I start adding pages.

Steve

[Micheal Mathews]

Hi Steve:

Micheal,

If your dynamic site has any pages in the index (real or supplemental), Google may see these as duplicate content to your static site even if the site no longer exists. I believe even if you don't have any pages indexed but G is crawling your pages, these un-indexed pages can be use to determine if duplicates exist.

Today, I have nothing in the index. Zip. Nada. Zilch. It's all gone. I checked with Google Sitemaps -- nothing. Says right up front - No pages from your site are listed in the index. Ouch for me. True enough, Google is crawling on a daily basis - so I don't know the effect of it all. Right now though, I have decided to bite the bullet and put back the static site and offer the TNG site as an alternative to site visitors. I know -- it's going to make me take a penalty, so for the next 9 months to a year while I am in that in that non-existent penalty box, I will write up allot of genealogy and family history content that has been on my to-do list for three years or so. My only saving grace through this episode is that I have a great bunch of researchers who know how to actually find the site and between them we can at least keep it active until it all gets sorted out.

While I am at it, I am going to put up a genealogy blog discussing the frustration of search engines and genealogy. Kind of a diary of "how my ancestors have been lost twice now" or some such thing. (Anybody get the idea that I am going for some extra content there?)

Let's explore that genealogy site review more that you had Steve. I think that's a worthwhile endeavor.

I'm going to stick around the TNG forums and will be happy to share ideas as we go forward. I just hate to see an entire group of genealogists having such a tough time right now.

Regards,

Micheal Mathews

Steve Winslow continued:

I've recently noticed that G is continually trying to crawl pages that have been blocked with both robots.txt and tngbots.php. So G knows those pages were once out there and knows their content even though they are not in the index. Also, I had a full sitemap.xml file and now even though the sitemap.xml has been removed, I still receive errors in G's sitemap tools that I have pages listed in my sitemap that are being blocked by the robots.txt file. There are also URLs listed as being in my sitemap.xml as being blocked that I never included in my sitmap.xml.

My point is that it may be difficult for you to put up a static site and have it effectively indexed if you recently have had a dynamic site, using the same genealogy information, being crawled.

My old domain (it was free and I was using their sub-domain) shut down in Jan of this year, G still has over 4,500 TNG type pages listed in its index from that site. I had to do a mod-rewrite (the site came back after being out of service for over a month and after I moved my site) to redirect those pages to my new domain and to get G from seeing those pages as duplicates of my current site.

I have even gotten very conservative about making organizational changes to my current site. I recently added a number of new html content pages and didn't think through their organization very well. I was working on reorganizing it last night when the thought came to me that G may already have those pages indexed and would see me moving exiting files to new folders as potential duplicate content. Sure enough, the new pages were already in the index, so I just left them where they were. I'll do a major reorganization one day and just take the hit for it, but for now I'll learn to think through my organization design before I start adding pages.

Steve

<edit> Micheal edited orignial reply for stupid spelling ... </edit>

[sjwinslow]

Micheal,

Well my site took a major hit in the number of pages indexed today too. What's really sad is that the 500+ non-supplemental find.php pages have all but 4 been dropped again. This could be due to a lot of work I've been doing with mod-rewrite this past week to clean up all of the multiple URLs pointing to the same content.

G has been crawling my site real hard since I made those changes so maybe it's just digesting the new information (or maybe G just didn't like what I did and will never list those pages again ). On top of the mod-rewrite changes I've also revamped all of my html pages from using a table structure to a lot more CSS. The pages look more consistent now and will be easier to maintain in the future but it may also cause a hiccup in my listings.

The site:domain.com command usually shows about 19,000 pages for my site (almost all of the supplemental) but tonight I noticed some of the data centers are showing 29,000 pages. G is obviously continually making updates and changes. I don't think there is any way we can stay ahead of them. I think you have the right approach. Get your site in the best possible shape and just build content.

Steve

P.S. The site reviews are going well. G seems to love these type of pages. I only have two completed and posted but plan to do more. If you like I'll be happy to do one for your site. On vacation through the 28th but after that I'll be able to spend some time on reviews.

[Micheal Mathews]

Micheal,

Well my site took a major hit in the number of pages indexed today too. What's really sad is that the 500+ non-supplemental find.php pages have all but 4 been dropped again. This could be due to a lot of work I've been doing with mod-rewrite this past week to clean up all of the multiple URLs pointing to the same content.

Hi Steve:

Enjoy your vacation, maybe you will come home to 19,000 indexed pages.

I'm dead in the water right now, as I mentioned before. Nothing. My home page is indexed as supplemental. That is how much of a mess this is for me. What really is the pits though is that my old domain that has not been active for almost 6 months now still shows 3900 results and will allow following of the URLs from the SERPS. Never mind the fact that some of the supplemental pages go back to January of 2005!

I have some advice for us: set a <META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE"> in our meta.html files. This will tell the search engines that they should not cache our pages and instead page requests should be sent to website server.

G has been crawling my site real hard since I made those changes so maybe it's just digesting the new information (or maybe G just didn't like what I did and will never list those pages again ). On top of the mod-rewrite changes I've also revamped all of my html pages from using a table structure to a lot more CSS. The pages look more consistent now and will be easier to maintain in the future but it may also cause a hiccup in my listings.

If it wasn't for Google, I wouldn't be using 4.3 gigs of bandwidth. Now if I could only get 4.3 gigs of results!

The site:domain.com command usually shows about 19,000 pages for my site (almost all of the supplemental) but tonight I noticed some of the data centers are showing 29,000 pages. G is obviously continually making updates and changes. I don't think there is any way we can stay ahead of them. I think you have the right approach. Get your site in the best possible shape and just build content.

I'm starting to think that those tools are just plain unreliable at this point. What the heck do I know though, I have only one supplemental home page listed - but on Tuesdays and Fridays it doesn't even show.

Steve, have you considered emailing Google and asking them to review your site and see if they can tell you why they are not indexing? I have tried, no replies. I'll try to find Matt Cutts blog post again about what address to write to ask for a review. Probably won't get a response, but at this time, anything is worth a shot.

Talk to you when you get back.

Regards,

Micheal

Steve

P.S. The site reviews are going well. G seems to love these type of pages. I only have two completed and posted but plan to do more. If you like I'll be happy to do one for your site. On vacation through the 28th but after that I'll be able to spend some time on reviews.

[Jerry]

Well, I went from 30 pages yesterday to 3300 today. That is the first time I have been over 100 EVER.

Jerry Milo Johnson

http://www.voiceinverse.com/family/

[Micheal Mathews]

Well, I went from 30 pages yesterday to 3300 today. That is the first time I have been over 100 EVER.

Jerry Milo Johnson

http://www.voiceinverse.com/family/

Jerry:

Nice site you have. Have long have you been up and running? Is that a template, or have you developed that yourself?

You've got some page rank showing Let's hope your pages hold and you continue to grow.

Best to you,

Micheal

[Jerry]

Jerry:

Nice site you have. Have long have you been up and running? Is that a template, or have you developed that yourself?

You've got some page rank showing Let's hope your pages hold and you continue to grow.

Best to you,

Micheal

The domain has been live for about 3 years.

The family folder has been live since about April 1

The TNG site was added late April.

I opened the site to search engines mid June.

I started letting people put the link on other sites in mid-late June.

So, all in all, I am pretty happy.

Jerry Milo Johnson