the bot that won't go away

[Deni]

For more than 2 and a half hours, the same 'bot has been traversing my TNG site and I can't seem to get rid of it.

The identifier (domain? IP address? What exactly is this?) is 189-041-102-156.xd-dynamic.ctbcnetsuper.com.br

I have tried adding that whole thing to my .htaccess file. No change.

I've tried just .xd-dynamic.ctbcnetsuper.com.br. No change.

I've tried .br (because there's really no one from Brazil who should be accessing my site). No change.

When it gives a referring page, it's always http://fillbest.com, which is a Canadian pharmacy company.

Part of it is annoyance to have a bot on my site when I have specifically stated that I don't want them. The other part is that it masks information about real people who come to the site so that I can see what people are looking at. It tells me whether there's any point to having some of the pages I've created, if nobody reads them.

Is there any way to get rid of this pest?

---------------------------------------

ETA: It finally stopped. This bot stayed on my site for well over 3 hours, access one page or another every 4 or fewer seconds. Maybe it won't come back, but is there any way to make sure it doesn't?

[tmcguire]

Hello -

The identifier (domain? IP address? What exactly is this?) is 189-041-102-156.xd-dynamic.ctbcnetsuper.com.br

This is a Host Name, the IP address is most likely 189.41.102.156 .

If you do a whois on the IP address via:

http://ws.arin.net/whois/?queryinput=189.41.102.156

or

http://www.dnsstuff.com/tools/whois.ch?ip=189.41.102.156

you will find more information about the IP. The BR indicates Brazil as the location of the IP host.

Since I am unsure how you have your .htaccess file setup, I can only suggest how you would block this IP if it were via my .htaccess file:

# ------------------------------------------------------------
# Deny access by IP address to sites who have broken the rules
#
Order Allow,Deny
Allow from All

# Insert Deny from statements below -

# Block the IP
Deny from 189.41.102.156

# or Block the IP Range
Deny from 189.0.0.0/8

The above is how I would do it in my .htacces file. You can decide if you want to block just one IP or if you want to block the range. All of your other 'Deny from' statements should be grouped with this one.

Once you have edited your .htaccess file, don't forget to upload it to your root public directory and test accessing your website. If access fails, you have made a mistake in the .htaccess... back out the change and start over. Make sure your .htaccess file permissions are at least '644'

Regards,

T. McGuire

[Deni]

Thank you much. I'll use that one.

I thought about posting my .htaccess file, but it's 3K in size these days. That was one of the things that was so confusing. Adding the host name in the past has worked. I've even stopped bots right in the middle of accessing before.

Maybe this will do it, though.

Thanks again.

[Deni]

I must be completely unclear on a concept here. Maybe someone can help me.

I've got some bot going through my site again, in spite of the instructions in my robots.txt file. There have been 600 entries in the past 15 minutes or less that look pretty much like this:

Sat 31 May 2008 07:56:31 AM Pedigree for Henry Everett Worthington (I110710) 5 Generations accessed by 88.pool85-54-57.dynamic.orange.es.

Here's a part of my .htaccess file:

####   Stop libWWW hackers ####
SetEnvIfNoCase User-Agent "^.*libwww-perl" block_bad_bots
SetEnvIfNoCase User-Agent "^.*psycheclone" block_bad_bots

<Limit GET>
order deny,allow
deny from 88.pool85-54-57.dynamic.orange.es
...
allow from all
</Limit>

(The ... is about 100 other lines of other bots that I've collected over time.) In the past, when I have added the "from" to my .htaccess file, even if the bot was in the midst of going through my site as this one is right now, it suddenly stopped. This one has not. I even changed the .htaccess file line to

deny from .es

because it's unlikely that any real person from Spain would be interested in my site. No good. They're still using my bandwidth at an incredible rate.

What am I doing wrong?

[Scotty]

I must be completely unclear on a concept here. Maybe someone can help me.

I've got some bot going through my site again, in spite of the instructions in my robots.txt file. There have been 600 entries in the past 15 minutes or less that look pretty much like this:

Sat 31 May 2008 07:56:31 AM Pedigree for Henry Everett Worthington (I110710) 5 Generations accessed by 88.pool85-54-57.dynamic.orange.es.

Here's a part of my .htaccess file:

####   Stop libWWW hackers ####
SetEnvIfNoCase User-Agent "^.*libwww-perl" block_bad_bots
SetEnvIfNoCase User-Agent "^.*psycheclone" block_bad_bots

<Limit GET>
order deny,allow
deny from 88.pool85-54-57.dynamic.orange.es
...
allow from all
</Limit>

(The ... is about 100 other lines of other bots that I've collected over time.) In the past, when I have added the "from" to my .htaccess file, even if the bot was in the midst of going through my site as this one is right now, it suddenly stopped. This one has not. I even changed the .htaccess file line to

deny from .es

because it's unlikely that any real person from Spain would be interested in my site. No good. They're still using my bandwidth at an incredible rate.

What am I doing wrong?

BotsvsBrowsers

And you can check to see if your .htaccess entries really work. Since the denys and bad_bots are not working from what you are seeing now. You can modify the settings and test the results until you get that 403 error you are going after. And for the bad bots you have listed I had to remove the Wildcard and Quotes symbols. Don't know if that will solve your problem.

And don't give me credit for finding that site, Think Rush is the one who originally found it.

Scotty

[Deni]

And for the bad bots you have listed I had to remove the Wildcard and Quotes symbols

I removed them and they still didn't work. It looked to me like the page you directed me to was to block certain browsers, not domains.

I did find this:

<?php
$block = array("xxx.xxx.xxx.xxx", "yy.yy.y.yyy");

if (in_array ($_SERVER['REMOTE_ADDR'], $block)) {
    header("Location: http://google.com/");
    exit();
}
?>

I was wondering if I could change $_SERVER['REMOTE_ADDR'] to $_SERVER['REMOTE_HOST'] and put all of my bad bots into the array. Then add this to the begin.php file and use that to redirect bad bots away from my site.

[Scotty]

I removed them and they still didn't work. It looked to me like the page you directed me to was to block certain browsers, not domains.

I did find this:

<?php
$block = array("xxx.xxx.xxx.xxx", "yy.yy.y.yyy");

if (in_array ($_SERVER['REMOTE_ADDR'], $block)) {
    header("Location: http://google.com/");
    exit();
}
?>

I was wondering if I could change $_SERVER['REMOTE_ADDR'] to $_SERVER['REMOTE_HOST'] and put all of my bad bots into the array. Then add this to the begin.php file and use that to redirect bad bots away from my site.

I just tried some of the names you have and none of them are getting blocked.

Are you making changes online or offline and then uploading the file?

Why not rename the file to htaccess.txt and add it as an attachment to this thread. And we can take a look at the file.

And you are using GoDaddy for a host?

And if you are doubting that site try .es and http://www.dsdarling.com. Then try libwww and then try libwww-perl and see the results.

Scotty

[Deni]

Are you making changes online or offline and then uploading the file?

Offline and uploading.

And you are using GoDaddy for a host?

No. It's a new, small company called TinyPortalHosting.

File is attached.

I see what you mean about that site. I'll check with my host and if they can figure out what's up. Thanks for all your help.

htaccess.txt

[Scotty]

Offline and uploading.

No. It's a new, small company called TinyPortalHosting.

File is attached.

I see what you mean about that site. I'll check with my host and if they can figure out what's up. Thanks for all your help.

Change this line from:

order deny,allow

to

order allow,deny

and see if that works. Thats how mine is set.

And I'm not sure about those two lines of

<Limit GET>

</limit>

I don't use them.

I'm no expert, I just fiddle until I get things to work.

Scotty

[Deni]

I understand. Thanks!

I've also sent in a question to my server admin.

[Scotty]

I understand. Thanks!

I've also sent in a question to my server admin.

I looked up my question that I sent to ICDSOFT support about blocking ip's since they don't use an IP Deny Manager:

Hello,

You can block the IP address(es) via .htaccess file, located in your main directory. The file (note the dot at the beginning of the file name) would need to have the following content:

order allow,deny

deny from 123.456.789.12

deny from 456.789.012.34

allow from all

as you change the numbers with the corresponding IP addresses. You can also block subnets or hostnames. For example, if you want to block the whole 64.124.85.xxx IP range, you can use the following deny statement:

deny from 64.124.85.

This would deny the access for all IP addresses starting with 64.124.85.

Best Regards,

Support

So it must be right!! Or it seems to work.

Scotty

[tmcguire]

Hello -

I looked up my question that I sent to ICDSOFT support about blocking ip's since they don't use an IP Deny Manager:

So it must be right!! Or it seems to work.

Yes. it is right; BUT the order statement must be:

Order Allow,Deny

NOT

Order Deny,Allow

FWIW, I usually follow this statement with the

Allow from All

This will keep it from getting lost as additional 'Deny from' statements are added to the .htaccess file. It also makes it easier to see not only what the ORDER is but what the ALLOW is with a single glance.

I also agree with removing '<Limit GET>' and '</Limit>'. The may be changing the rules of the Allow statement.

May I offer an additional change. Replace:

#### Stop libWWW hackers ####

SetEnvIfNoCase User-Agent "^.*libwww-perl" block_bad_bots

SetEnvIfNoCase User-Agent "^.*psycheclone" block_bad_bots

With:

# Start of URL rewriting code

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} ^libwww-perl [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^psycheclone [NC,OR]

RewriteRule ^.*$ - [F,L]

I think you will have better results with blocking these two agents.

For documentation on these statements, see: http://www.javascriptkit.com/howto/htaccess14.shtml

and http://www.javascriptkit.com/howto/htaccess13.shtml

Regards,

T. McGuire

[Deni]

Thank you.

I know that

Order Deny,Allow

worked in the past, but I've changed it. I've also made the other change. I am still curious about the addition of the php code that I posted. Since all of the pages start with

include("begin.php");

if I added that code to redirect "bad" bots to google, would it work? Would the

    exit();

command mean that it would stop executing the commands and send them to google?

ETA: Must have been the deny, allow thing, because the testing page works now. Sorry for all the problem.

I'd still like to know about the php thing, though, if anybody knows.

I thought it worked after switching the allow,deny thing, but it didn't. Changing the other code caused an error on the site.

I'm back to where I was. It doesn't matter whether it's allow,deny or deny,allow. They have the same result.

[tmcguire]

I thought it worked after switching the allow,deny thing, but it didn't. Changing the other code caused an error on the site.

I'm back to where I was. It doesn't matter whether it's allow,deny or deny,allow. They have the same result.

It is important to have the:

Order Allow,Deny

Allow from All

If you change this to 'Order Deny,Allow' you have negated what the 'Deny from' statements are to do. This is a boolean process. Most likely you have badly coded 'Deny from' statements.

I will get back to you.

Regards,

T. McGuire

[Deni]

Hang on.

At the moment, I have

order allow,deny

deny from 88.pool85-54-57.dynamic.orange.es

deny from 203.162.2.133

allow from all

(there are more "deny from" lines as well)

Before I had

order deny,allow

deny from 88.pool85-54-57.dynamic.orange.es

deny from 203.162.2.133

allow from all

Which makes a whole lot more logical sense, since the list has first "deny" lines and then an "allow" line.

Most likely you have badly coded 'Deny from' statements.

What would be correctly-coded "Deny from" statements? I took all of the statements directly from the TNG log.

I just went through every possible permutation,

order deny,allow and order allow,deny

allow from all under the "order" line and allow from all at the bottom

I used an IP address that I have in the file and tries out that BotBrowser site. No matter what I did, the site was accessible.

[tmcguire]

Hello -

I found the problem. Inadvertantly I created it. The last 'RewriteCond' should be without the ',OR'

RewriteCond %{HTTP_USER_AGENT} ^psycheclone [NC]

I have attached an .htaccess file with all of the changes suggested to your .htaccess. I checked this .htaccess on my website.

The only other change I can suggest is to use IP addresses instead of what you have, i.e. convert all of the 'Deny from' to use IP addresses. Take your IP addresses from your website raw access logs instead of the TNG access logs.... or convert what TNG displays to IP addresses.

Hope this helps.

Regards,

T. McGuire

htaccess.txt

[Deni]

It must be a problem with my server. Using your .htaccess file still does not deny access based on an IP address.

I'm going to investigate the php option further. I understand php and it makes sense to me.

[tmcguire]

It must be a problem with my server. Using your .htaccess file still does not deny access based on an IP address.

How do you know? I know of no way of assuming another IP address (anonimzers do not count). The only thing that you can test is user agent, and that works. If you want to use your own IP, feel free.... just don't release your FTP client until you have restored / uploaded the .htaccess file without your IP address in it (this I have successfully tested many times).

Go ahead and investigate the php method. This is not the correct way. The correct way is via .htaccess.

BTW, you are uploading .htaccess, i.e. dot htaccess, and not htaccess.txt right?

Regards,

T. McGuire

[arnold]

BTW, you are uploading .htaccess, i.e. dot htaccess, and not htaccess.txt right?

T. McGuire is correct in mentioning the file's correct name (with a dot/period and no file extension).

Please note that some FTP programs, by default, do not "see" files that be begin with a dot/period. This can be tricky territory.

[Deni]

The only thing that you can test is user agent, and that works.

I tested a number of user agents from my .htaccess file in the BotvsBrowsers page and every one of them was still able to access my site.

I haven't tried my own IP address. I should probably do that.

Go ahead and investigate the php method. This is not the correct way. The correct way is via .htaccess.

If the preferred method doesn't work, I try another method.

BTW, you are uploading .htaccess, i.e. dot htaccess, and not htaccess.txt right?

Well, I uploaded your htaccess.txt and then changed the name of the file. I didn't want to overwrite my .htaccess file with yours until I knew that yours was better than mine. I haven't found that to be the case as yet.

Please note that some FTP programs, by default, do not "see" files that be begin with a dot/period. This can be tricky territory.

My FTP program "sees" .htaccess just fine.

The main point is about all this is that my file, with whatever order the "order deny,allow" line was originally worked just fine until I changed servers, which indicates to me that maybe -- just maybe -- it's a server issue. I thought at first that it was some clever bot that was able to circumvent the .htaccess file. It does not appear that such is the case. The only other possibility is the server.

I thank everyone for their attempts at help. I think we can close this now.