Jump to content
TNG Community
manofmull

No Follow Meta Tag

Recommended Posts

manofmull

Exactly where in my template index.php should I insert the following?

I currently have it right at the top of the template index.php

<meta name="robots" content="index, nofollow">

If not in the template index.php should it go in meta.php?

For good bots, will this allow indexing of the homepage and nothing else?

Share this post


Link to post
Share on other sites
Rob Severijns

Hi manofmull,

 

I'm no security specialist but I'm not sure why you would want to include this in the index.php. 

My robot.txt is placed in the rootfolder of my tree

It contains the following:

User-agent: *
Disallow: /

This doesn't allow any indexing at all, which is exactly what I want since my site is private and for family members only.

Good bots will follow the rule, bad bots might ignore it.

More info on the robot.txt file can be found here:

https://support.google.com/webmasters/answer/6062596?hl=us

It also shows you how to ban bad bots and allow good bots.

Next to the robot.txt I use the recaptcha  and Bot-Trap mod to try to keep bad bots out. The Bot-Trap mod still works with TNG 12.3

So far I didn't recieve any spam messages yet.

There is also a possibillity the edit the .htaccess file with a list of bad bots to prevent access and indexing of your site.

More info on that topic can be found here:

https://www.seoblog.com/block-bots-spiders-htaccess/

Not sure if this is of any help but I hope it will help you a little further is the right direction.

 

Kind regards,

 

Rob

Share this post


Link to post
Share on other sites
theKiwi
On 9/11/2020 at 5:50 PM, manofmull said:

Exactly where in my template index.php should I insert the following?

I currently have it right at the top of the template index.php

<meta name="robots" content="index, nofollow">

If not in the template index.php should it go in meta.php?

For good bots, will this allow indexing of the homepage and nothing else?

This functionality is already built in to TNG. Take a look at the file tngrobots.php.

It is set to allow some pages to be indexed, and others not.

You could alter it so that all pages except index.php get 

			$flags['norobots'] = $NOINOF;

Roger

Share this post


Link to post
Share on other sites
manofmull

Rob

I have Bot-trap installed as well as RIP prevention and robots.txt

 

Roger

I'll have a look at tngrobots.

I only want my homepage indexed, nothing else.

My site requires logging in so that adds a good level of security/privacy

Share this post


Link to post
Share on other sites
Ken Roy

Michael,

As far as I know, if your site requires login, then the bots cannot index your site, so there would be no sense or benefits on updating the tngrobots.php nor the robots.txt, since they cannot login and follow any links

Share this post


Link to post
Share on other sites
Rob Severijns

Ken, Roger, Manofmull,

 

I wasn't aware of the existance of a tngrobots.php and what it does.
Meanwhile I found the wiki about the tngrobots.php and the wiki containing info on securing tng data which both are very useful.
I see the default setting in tngrobots.php is to index and follow.
This is not what I want since my robot.txt file says:

User-agent: *
Disallow: /

So my question is "Are the settings in the tngrobots.php file overriding the robot.txt file settings in the root?"
If so, I need to change the tngrobots.php and if not I can leave things as they are.
 

Kind regards,

Rob

Share this post


Link to post
Share on other sites
manofmull

Rob

Good question. If truth be told, I'm more confused than you are.

I have no idea how to edit the file Roger pointed out.

I want what Rob wants except I don't mind the homepage being indexed.

Share this post


Link to post
Share on other sites
Rob Severijns

Manofmull,

 

I can understand the confusion. I feel the same.

Regarding TNG and security it seems  there's more to it than meets the eye.

There is little mentioning regarding security and the options when installing TNG.

I guess the default settings in TNG are good enough for most users or maybe most users aren't aware of the risks.

GDPR requires me to do my best to keep the information of people secured and access restricted.

Trying to comply as much as possible via SSL, reCaptcha, Bot-Trap, robot.txt and user restrictions.

If the tngrobots.php overrides all of the above and makes indexing and following links possible I would be very disapointed.

Wondering what the answer to my question is.

 

Rob

Share this post


Link to post
Share on other sites
Ken Roy

Rob,

There are several articles in the TNG Security category on the TNG Wiki on how to protect your web site.

Share this post


Link to post
Share on other sites
Rob Severijns

Hi Ken,

 

I saw that and added the link to that wiki page in my previous message too.

Just not sure if the tngrobots.php overrides the settings in robot.txt

 

Rob

Share this post


Link to post
Share on other sites
theKiwi
5 hours ago, Rob Severijns said:

Ken, Roger, Manofmull,

 

I wasn't aware of the existance of a tngrobots.php and what it does.
Meanwhile I found the wiki about the tngrobots.php and the wiki containing info on securing tng data which both are very useful.
I see the default setting in tngrobots.php is to index and follow.
This is not what I want since my robot.txt file says:


User-agent: *
Disallow: /

So my question is "Are the settings in the tngrobots.php file overriding the robot.txt file settings in the root?"
If so, I need to change the tngrobots.php and if not I can leave things as they are.
 

Kind regards,

Rob

 

No - at least for good robots, the robot.txt file should have precedence since the robot will read it before it reads any of the pages.

Other notes - if your site requires log in to see anything, then the robots can only get to the index.php page and will get stopped at the login page by any request for any other page.

Roger

Share this post


Link to post
Share on other sites
theKiwi
2 minutes ago, Rob Severijns said:

Hi Ken,

 

I saw that and added the link to that wiki page in my previous message too.

Just not sure if the tngrobots.php overrides the settings in robot.txt

 

Rob

In your case it would. In my case, where robots.txt says to index almost everything (except I do have a list of excluded files) it won't - the tngrobots.php file will have the control.

Roger

Share this post


Link to post
Share on other sites
Rob Severijns

Roger,

 

Thx, this is what I was hoping for.

Hope Manofmull will find a solution for his question too.

 

Rob

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×