Want to get Your Genealogy Site Listed with the Search Engines?

[sjwinslow]

I’ve noticed several people using the find.php pages and still having no luck getting any significant number of their site pages listed in Google. Below is a list of items that need to be reviewed before your efforts in adding a find.php page will have any benefit. I don’t believe there is any quick fix that will get your pages indexed these days. The approach presented below just makes sure you don’t have errors on your site, you have quality links pointing to your site and when pages do get indexed they are the ones that have information on them where people can find your site. If you already have pages that are supplemental due to duplicate content, you will need to fix this problem before you will be effective in getting more pages in the active index.

1. Make sure your site can be crawled by the bots. E.g., If you require people to login with a password to view your TNG site, then the bots won’t be able to index it. Robot.txt errors, submit forms and java links can also block bots from crawling your site.

2. Make sure your site does not have html errors in the code the bots see. You can use this site ( validator.w3.org/detailed.html ) to review your pages and it will provide you a list of any errors it finds in the html code it sees. Remember, your site may be coded in PHP but the bots only see the html code that is produced by the PHP scripting.

3. Get other genealogy websites to link to your site. Make sure you are listed (these are not all genealogy related but they are must haves) in Yahoo.com, MSN.com, DMOZ.com, CyndisList.com, mycinnamontoast.com. There are many others and you’ll want to get as many as you can. If the Google bot is not crawling your site every day it will be a very long time before you get any more than your homepage and a handful listed. More suggestions here for possible places to obtain links - http://www.tngforum.us/index.php?showtopic=1216

4. Do not allow your site to produce PHPSESSID=xxxxxxx responses to the bots. If your site is allowing session ids to be created, it will create a new session id every time a bot visits a page. Google will see these session ids in the page’s URL and think you have different URLs pointing to the duplicate content. Google will many times either remove these duplicate pages or move them to supplemental results where they have very low rating. You can find out how to stop session ids here. http://www.tngforum.us/index.php?s=&showto...findpost&p=3303

5. The Google bot requires unique content for a page to get/stay indexed. Many of our TNG pages look very similar and many also contain a low percentage of unique content to the labels and navigation links on each page. Think about it, if you view two different people using a getperson.php page, there may be little difference in the content of each page. As genealogist we know each entry is unique and very important but the search engines look at our pages in a very cold light of how much one page is different from another. Make sure your titles, descriptions and keyword meta tags are all different. Here is a suggestion to add a little more uniqueness to each of these http://www.tngforum.us/index.php?s=&showto...findpost&p=2893

6. At this time I’m only recommending a very simple sitemap.xml be submitted to Google that only includes your homepage. A number of errors are being reported recently with both the tool and the free tools that create the sitemap but it is a good way to get your site noticed and it provides some elementary tools to diagnose your site.

7. Make sure you have a restrictive robots.txt file. A new site is not trusted by the major indexes these days. With so many spammer sites gaming the search engines, the major indexes have placed barriers to keep these sites that crop up every few weeks from getting in the index. This also keeps many of our new legitimate genealogy sites from being well indexed. The way to build trust is the passage of time and additional quality links as mentioned in step 3. Once Google starts indexing a few of your pages you want to make sure the most important pages are the ones listed. TNG provides many ways to present the same information. Allowing the bots to crawl all the TNG pages causes two problems. First, you will most likely get low value pages listed in lieu of high value pages. Second, the bots can use so much of your website’s bandwidth that your hosting company may shut down your site. Only allow the bots to crawl the pages you consider the highest priority. Remember, you may only get 1% of your pages listed for a year or more.

8. In release 5 you will need to prevent the bot from crawling the print friendly pages. Google will see the print friendly pages as a duplicate for almost every page on your site and keep them from being indexed correctly. For instructions on this modification see this posting http://www.tngforum.us/index.php?s=&showto...findpost&p=5787 (Note: Darrin indicates that this is corrected in release 6)

9. Update the tngrobots.php file to work in conjunction with you robots.txt file. Since it seems Google will only index so deep and a limited number of pages on new sites, its best to limit the crawling to those pages that will most likely be found by searchers.

10. Once you have control of the items above, consider using the find.php file as the primary file provided to the search engines. The reason I recommend this file is because it summarizes the ancestor information to just their name, birth date and location and their spouse’s name per line. It also has a very low quantity of lables and only one link per line to the getperson.php page of the person. Its primary difference from the search page is that it is only used to locate surnames and it only contains the pertinent information for which a potential visitor would search. I would also suggest that initially only the top 25 surnames be placed on the homepage that would link to the find.php pages. Use the simplest URL to link from the surname to the find.php page. (e.g., If you don’t need to denote the tree on your site then don’t include it in the URL. See if something as simple as .http://YourURL.com/find.php?mylastname=SurnameHere will work on your site.) For more informaton on find.php implementation see this link http://www.tngforum.us/index.php?s=&showto...findpost&p=5437

11. Check the major indexes periodically (I do it weekly) to make sure nothing is being indexed that shouldn’t. New sites seem to only get a limited quantity of pages listed (Many think the quantity of pages you are allowed listed is related to you sites page rank). Make sure the ones you intended to be listed are being listed and no others. Also keep a close eye on supplemental listings. These are a symptom that you have either duplicate pages or low content pages being indexed and then moved to supplemental.

[Micheal Mathews]

Hi Steve:

As always, you provide some great advice for TNG webmasters.

6. At this time I’m only recommending a very simple sitemap.xml be submitted to Google that only includes your homepage. A number of errors are being reported recently with both the tool and the free tools that create the sitemap but it is a good way to get your site noticed and it provides some elementary tools to diagnose your site.

I know that it's controversial between SEO experts and webmasters, but I maintain a position that not only Google, but the other major search engines don't quite have all the bugs worked out with crawling dynamic pages, and Google seemingly confirms this:

Make your site easily accessible

Build your site with a logical link structure. Every page should be reachable from at least one static text link.

I take that to mean that if you have an html sitemap page linking off your front page, you are more than likely going to have improved crawling results. It's a heck of a lot of work, so individual webmasters will have to determine the ROI value for their efforts.

I also see issues in the Google Sitemaps tools, chiefly that they appear to lag about two weeks between submission and updated crawl. This concerns me a bit, so I have started taking the approach of adding five updated names a day to the database, and by adding some robot tracking code to the getperson.php that emails me when google visits I can get a better feel for the crawling activity. This all may wash out when Google finally deems the site worthy and releases it from that non-existent sandbox.

In the end though, I feel it comes down to two of your other points that you mentioned, which most webmasters are overlooking: provide valuable user content to your pages; and get as many reputable and associated links as you can.

Thanks for all the advice.

Micheal

[Rush]

On a side note, building a static html 'site map' link page isn't too bad. It took about 10 minutes.

I ran a query from the tng_people database, exported the results into Excel and used a simple formula to convert the text over to links.

I haven't put up the page yet as I've been a little busy with theming on the site.

Rush

[sjwinslow]

Micheal,

Thanks for the kind words.

Steve

[Micheal Mathews]

On a side note, building a static html 'site map' link page isn't too bad. It took about 10 minutes.

I ran a query from the tng_people database, exported the results into Excel and used a simple formula to convert the text over to links.

I haven't put up the page yet as I've been a little busy with theming on the site.

Rush

Hi Rush --

Do you mind posting a link to the page when you get it done? I'd be interested in seeing how it looks and works. I'd also be interested in knowing how quickly engines find it and crawl it.

As a side note, Google in one sentence asks webmasters to submit XML sitemaps, and in another says build a static link page ... aye aye aye ...

Regards,

Micheal Mathews

[Rush]

Not a problem...

It will take me a few minutes to tweak it up. I think I'm done trying to destroy the 'Net with my other coding projects for a little while now

I suppose I could have created a static html link page generated out PHP (not to hard) but I was curious to see how this would work anyways.

Rush

[Rush]

Here it is...

Whole bunch of names

I'm not exactly sure why I did this awhile ago... messing around I guess.

Don't know if it's even of slightest bit of value, but if it proves to be the least bit effective I might just write the php script and url_rewrite to display it as a static html page that will constantly stay current.

I'm going to link up to in my footer tonight.

Rush

[tmaguirre]

Rush, has your "Whole Bunch of Names" had any effect, positive or negative?

Here it is...

Whole bunch of names

I'm not exactly sure why I did this awhile ago... messing around I guess.

Don't know if it's even of slightest bit of value, but if it proves to be the least bit effective I might just write the php script and url_rewrite to display it as a static html page that will constantly stay current.

I'm going to link up to in my footer tonight.

Rush

[Rush]

Looking at my Google cache, it doesn't seem that google has picked up on it yet.

I wrote up a little script that will pull the names out of your database and display them in an HTML format.

Then one can just copy the page source and save it as an HTML file.

Not sure if it has any real value, but it's here if anyone wants it.

Rush

[Micheal Mathews]

Looking at my Google cache, it doesn't seem that google has picked up on it yet.

I wrote up a little script that will pull the names out of your database and display them in an HTML format.

Then one can just copy the page source and save it as an HTML file.

Not sure if it has any real value, but it's here if anyone wants it.

Rush

Hey Rush:

I think the script has value -- it does indeed pull out the names from the database and build links to the individual pages. I took the output a step further and seperated out the names into a alphabet index of A, B, C, D, etc., that included major names: A | B-Bot | Bott | C-Chan | Channell | D etc., them saved the output into a pure html file. It took some extra work, but the end results will be human consumption friendly rather than just a mass of names on a page that would be robot friendly. What I am getting from my research and reading of the big G is that they want nautural and human friendly links that flow organicaly - so I think you are off to a good start on this.

I will definately use it on the dynamic side of my "new" genealogy site.

Thanks for helping others!

Regards,

Micheal Mathews

[Rush]

Good idea Micheal!

If I get a few minutes, I might tweak up the script to do that automatically...

Rush