Google is only very slowly indexing TNG pages

[PeterC66]

I have successfully used the Create Site Map Mod for quite a few years and it gave Google what it needed so that everyone in my database was indexed. Earlier this year I noticed that Google was no longer indexing most of the getperson pages on my site, even though they are in the tng sitemap.  There seems to be no problem with Google indexing all the WordPress pages.

I spent several weeks before the summer trying to find out the cause using Google's Console documentation etc, and trying various possible solutions but with little seeming success.  I suspect it is something to do with my "crawl budget".

On recently starting to look at the issue again I note that Google has now indexed about 100 people (out of over 13000), so it is going up - but very slowly.

The Console says of those not indexed:

• Discovered – currently not indexed (ie not crawled yet) 13,196 
• Crawled - currently not indexed 68

When I enter

site:https://www.hcnhistory.org.uk/ “getperson”

into Google I get 80 hits, which is also more than a few months ago (I am not sure how this 80 relates to the 100 figure from the Console, but it is not a major issue).

Does anyone have any experience of tackling this Google indexing issue, or suggestions?
 

[Katryne]

Hello Peter,

It seems that you have tried everything to get your site indexed (site map, SEO plugin and so on), but when I searchGoogle  in site:www.hcnhistory.org.uk, I get only 458 results altogether.

When I look into the source of your home page, I find :

Citation

<meta name="robots" content="max-image-preview:large" />

And that's all for the robots.text file. It seems to come from the SEO plugin.

To be found, a site should at least provide a robots.txt file expressly authorizing the indexing of your site and the tracking of links on it. Ideally it should also include the url of your sitemap

See there.

https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

[PeterC66]

Hi Katryne, Thanks for looking into my issue. 

I do not understand a lot about it, but I do have a robots.txt file that specifies, among other things:

# Sitemaps (but Yoast disagrees putting them here)
# This one is dynamic, generated by All in One SEO, an index containing 9 sitemaps for post, page, attachment, presentation, slide, post-archive, post_tag, slide-page, element_category
Sitemap: https://www.hcnhistory.org.uk/sitemap.xml
# This one is static, generated by TNG mod after each GEDCOM load (the index points to just? tngsitemap1.xml which has all the data)
#   such as http://hwnhistory/tng/getperson.php?personID=P1&tree=hwn
Sitemap: https://www.hcnhistory.org.uk/tngsitemapindex.xml 

And when I look at Google's Console I can see the sitemaps and the robots.txt file, and that it does not prevent indexing of the getperson.php files. I think this is confirmed because there are 100 such pages indexed.

If you would be willing to look further I would be happy to give you private access to my site.

[PeterC66]

Update: Amazingly, the same day I posted my issue here Google seems to have relented, and today (17 Oct 2022) I see that 2.36 K of my expected 13 K + person pages are indexed - and it says that most were last crawled on 14/10/22.  Hopefully, the rest will appear in the index in the coming weeks.  I do not remember doing anything to the site in the weeks leading up to this event (except using Google Console to see what was going on), so I cannot shed any light on it.  Using 

site:https://www.hcnhistory.org.uk/ “getperson”

 now has About 1,860 results, whereas last week it had About 65 results. If the improvement continues then I am delighted.