"What are Google's design and technical guidelines?"

[arnold]

Below the row of equal signs is information from Google's Webmaster Help Center. Please note the problematic constraint which pertains directly to TNG: "If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few."

It appears to me that dynamic pages produced by TNG are being ignored by Google on purpose. This may explain why our web pages are not being indexed by Google.

Please note: In order for this message to be accepted by the TNG Forum, I had to misspell a word in Google's technical guidelines below. Please substitute the letter "y" for the character "*" in L*nx.

====================================

Webmaster Help Center

What are Google's design and technical guidelines?

Design and content guidelines

* Make a site with a clear hierarchy and text links. Every page should be reachable from at least one static text link.

* Offer a site map to your users with links that point to the important parts of your site. If the site map is larger than 100 or so links, you may want to break the site map into separate pages.

* Create a useful, information-rich site, and write pages that clearly and accurately describe your content.

* Think about the words users would type to find your pages, and make sure that your site actually includes those words within it.

* Try to use text instead of images to display important names, content, or links. The Google crawler doesn't recognize text contained in images.

* Make sure that your TITLE and ALT tags are descriptive and accurate.

* Check for broken links and correct HTML.

* If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.

* Keep the links on a given page to a reasonable number (fewer than 100).

Technical guidelines

* Use a text browser such as L*nx to examine your site, because most search engine spiders see your site much as L*nx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

* Allow search bots to crawl your sites without session IDs or arguments that track their path through the site. These techniques are useful for tracking individual user behavior, but the access pattern of bots is entirely different. Using these techniques may result in incomplete indexing of your site, as bots may not be able to eliminate URLs that look different but actually point to the same page.

* Make sure your web server supports the If-Modified-Since HTTP header. This feature allows your web server to tell Google whether your content has changed since we last crawled your site. Supporting this feature saves you bandwidth and overhead.

* Make use of the robots.txt file on your web server. This file tells crawlers which directories can or cannot be crawled. Make sure it's current for your site so that you don't accidentally block the Googlebot crawler. Visit http://www.robotstxt.org/wc/faq.html to learn how to instruct robots when they visit your site.

* If your company buys a content management system, make sure that the system can export your content so that search engine spiders can crawl your site.

* Don't use "&id=" as a parameter in your URLs, as we don't include these pages in our index.

[rdmorss]

This has been discussed: see this topic

[sjwinslow]

arnold,

I too thought at first that Google was not managing dynamic pages well and not indexing them. However, recent tests have change my opinion. First, my site has both html and dynamic pages and the dynamic pages are from both TNG and PHPBB.

I first though there was a problem with dynamic pages because I could not get the getperson.php pages from TNG indexed nor could I get the replies of my forum indexed but every one of my html pages was indexed.

The forum on my site gave me the first clue. Google was not indexing the postings and replies but it did index the various forum titles and topic titles just fine. If Google couldn't handle dynamic pages then why would it index these directories that were also dynamic pages.

The conclusion I came to was that it is not the dynamic nature of the page that Google was having problems with but Google was not indexing below the third level on my site. This explained why all my html pages were indexed. They were all on third level or above.

To test this theory, I placed all of my surnames contained on my site on my home page and created a new PHP page called find.php that displayed the name, birth date, location and spouse for each of the surnames on my home page to link to. This allowed the search engine to locate a summary of all the people on my site just one click off the home page.

I implemented this change on the night of May 23, when I had only 150 total pages of my site indexed. One week later over 10,000 pages were indexed. See the chart below that shows the changes in pages indexed on my site for the total, getperson pages and the new find.php pages.

Date______Total__getperson___find

23-May_____151_____9_________0

24-May_____154_____9_________0

25-May_____874_____9_______ 663

26-May_____924_____7_______783

30-May_____955____14_______785

31-May___10500____14_______991

1-Jun_____13800____14______1000

2-Jun_____16400____40_____15400

5-Jun_____20900____83_____17700

Google also has guidelines that there should not be more than 100 links per page. As you can imagine I have exceeded that guideline significantly. This could be a problem in the future for this approach.

This approach will not work for everyone. It becomes very cumbersome to keep a flat navigation with 100's of thousands pages or where you have a few surnames that contain thousands of individuals. Also your site must have some level of page rank to get Google to index any farther than the home page.

But the key to getting pages indexed seems to be to get a higher page rank and a flatter navigation structure not dynamic versus static pages.

Steve

[arnold]

Steve,

I looked at your website and could not find where you had placed a link to find.php. And also could not see the source code for find.php. Would you be willing to share these "secrets"???? I would like to implement your idea.

Robert,

A short while back, Google's index for our website showed around 30,000 web pages. After Google's having visited our new TNG-based website for eight months, this was not particularly impressive as we have around 249,000 web pages to index. I know that Google's bot had seen all of our web pages many, many, many times over based on the TNG log. Then the number of web pages indexed by Google dropped to 11,000. Then down to 9,000. And now it is at 718.

The TNG Features web page continues to state that search engines index TNG-based websites. Having our pages thoroughly indexed was of prime consideration for us before we switched to TNG. I discussed this several times with Darrin.

It was out of strong frustration that I started a new topic placing Google's stated indexing policy in post #1. We have 249,000 individuals on our website, 180,00 of them are deceased, and thus searchable. The main search engine used today, Google, lists only 718 of them. As a result, we are dead in the water.

[sjwinslow]

arnold,

You can find the link to the find.php file here in the forum post as well as several comments later in the thread on what was modified and what you would need to change.

To find the surname links, they are on my home page, but you have to scroll down to find them.

Since you have so many people on your site you may need to adapt this approach to the size of your site. I had problems getting Google to index pages that had more than ~1000 links on it. The surname Winslow on my site has about 1800 individuals. When I set the find.php page to display all of them on one page I never did get Google to index that surname. I backed off the number of rows displayed to 600 and that seemed to fix the problem. But now I need to go back to my home page and add additional links for Winslow1, Winslow2 and Winslow3 to cover all the pages.

If you are not interested in the structure on your home page being useful to people it might work better for your size site to just display names by page. This would mean adding links to your homepage like page1, page1, page3... then have the link from page1 link to the first 600 people in your database and page2 link to the second 600 people in your database. You would keep doing this until all people could be found. This would give you the most people displayed on the fewest pages. If you are interested in this method and need more information on how it might work, let me know.

Steve

[arnold]

Steve,

Thank you (1) for the link to download find.php and (2) where to find the link to that file on your main page. I do not know how I missed the latter.

I like your idea as to page1, page2, page3, etc. The only rub is that we have 16,742 deceased individuals with the Sprague surname. It would be nice to be able to automate the page production as steve3000 does with his sitemaps.

[sjwinslow]

arnold,

I'm sure there are better ways, but just off the top of my head here's how I would take a shot at automating the page1, page2, page3... idea. I would start with excel and let the drag feature create all the Page1 ,Page2, etc. with the the correct hyperlink for the second page (the first page is unique but the second through whatever only has the page number different). This process will place the same link address on every link but at least most of the work of link name and link address would be created. Then you would have to go in and change the page number on pages 3 through however many pages you have.

It's still quite a bit of work but remember the find.php page is currently set to default at 600 rows per page so the number of pages wouldn't be too large.

I'm sure there are better ways out there and maybe someone will jump in and suggest one.

Steve

[steve30000]

I would suggest:

1) Take your surnames-all.php, rename it and strip all the extra information like icons and menus so it is just producing the surnames.

2) When it is calculating how many surnames there are, put a small routine in that would create a new page for every 600 individuals with that surname.

3) Adjust the links it is putting on the page to the find.php pages.

4) Use an "include" statement to add that file into what ever page you want.

[mvigor]

I would suggest:

1) Take your surnames-all.php, rename it and strip all the extra information like icons and menus so it is just producing the surnames.

2) When it is calculating how many surnames there are, put a small routine in that would create a new page for every 600 individuals with that surname.

3) Adjust the links it is putting on the page to the find.php pages.

4) Use an "include" statement to add that file into what ever page you want.

Great minds think alike! Before reading this post, I did JUST THAT this morning.

Vigor Family Genealogy Front Page