showlog.php and search engine robots

[arnold]

I posted this earlier under Installation & Configuration <showlog.php and search engine robots>, but it should also appear here:

It has become impossible to see which IP Addresses are visiting our website due to the overwhelming number of visits by various search engine robots, especially Google.com's. I wrote to Darrin about this problem to see if he would add a way to keep the visits by robots from cluttering up the log file. He wrote back stating that that feature is already included. Here is what he wrote:

Just go to the log and copy a portion of the logged domain name or IP address (ie, "google.bot"), then go to Admin/Setup/Log Settings and paste that into the field for host names to exclude. You can separate multiple entries with commas. That should still allow those crawlers to visit your site, but their visits will no longer be logged.

After implementing Darrin's "fix," our log file now looks like this:

The Sprague Project Log File

The hundreds and hundreds of visits by various robots no longer appear.

Thank you, Darrin, as always.

[TNGUSER]

I posted this earlier under Installation & Configuration <showlog.php and search engine robots>, but it should also appear here:

It has become impossible to see which IP Addresses are visiting our website due to the overwhelming number of visits by various search engine robots, especially Google.com's. I wrote to Darrin about this problem to see if he would add a way to keep the visits by robots from cluttering up the log file. He wrote back stating that that feature is already included. Here is what he wrote:

After implementing Darrin's "fix," our log file now looks like this:

The Sprague Project Log File

The hundreds and hundreds of visits by various robots no longer appear.

Thank you, Darrin, as always.

65-101-142-132.tukw.qwest.net accesses my site daily....hundreds of records at a time. I did a search for tukw.qwest.net and what I get is tukw.net which appears to be an adult site or at least it says it has adult content. If it's not a legitimate search engine, how can I stop them from accessing my site?

Thanks for your help.

Carol

[Rush]

This looks like a Qwest ISP subscriber.

What they might being doing is participating in the MJ-12 search engine project, but you would have to look at the server log files to see what agent they are using.

If your host has cPanel, you can block them using the 'IP Deny Manager'. Another way is to edit the htaccess file on your server directly:

order allow,deny

deny from 65-101-142-132.tukw.qwest.net

allow from all

Rush

[TNGUSER]

This looks like a Qwest ISP subscriber.

What they might being doing is participating in the MJ-12 search engine project, but you would have to look at the server log files to see what agent they are using.

If your host has cPanel, you can block them using the 'IP Deny Manager'. Another way is to edit the htaccess file on your server directly:

order allow,deny

deny from 65-101-142-132.tukw.qwest.net

allow from all

Rush

I'll give that a look-see. Thanks Rush.

Carol

[arnold]

I did a WHOIS search on 65.101.142.132 and came up with what you see below. Note that you can call by phone or send e-mail to the abuse folks at Qwest Communications Corporation.

======================

WHOIS Search Results

Your WHOIS Search Results

65.101.142.132

Record Type: IP Address

OrgName: Qwest Communications Corporation

OrgID: QCC-22

Address: 1801 California Street

City: Denver

StateProv: CO

PostalCode: 80202

Country: US

NetRange: 65.100.0.0 - 65.103.255.255

CIDR: 65.100.0.0/14

NetName: QWEST-INET-115

NetHandle: NET-65-100-0-0-1

Parent: NET-65-0-0-0-0

NetType: Direct Allocation

NameServer: AUTHNS1.MPLS.QWEST.NET

NameServer: AUTHNS2.DNVR.QWEST.NET

NameServer: AUTHNS3.STTL.QWEST.NET

Comment:

RegDate: 2001-01-03

Updated: 2006-02-09

OrgAbuseHandle: QIA2-ARIN

OrgAbuseName: Qwest Abuse

OrgAbusePhone: +1-877-886-6515

OrgAbuseEmail: abuse@qwest.net

OrgNOCHandle: QIN-ARIN

OrgNOCName: Qwest IP NOC

OrgNOCPhone: +1-877-886-6515

OrgNOCEmail: support@qwestip.net

OrgTechHandle: QIA-ARIN

OrgTechName: Qwest IP Admin

OrgTechPhone: +1-877-886-6515

OrgTechEmail: ipadmin@qwest.com

[TNGUSER]

I did a WHOIS search on 65.101.142.132 and came up with what you see below. Note that you can call by phone or send e-mail to the abuse folks at Qwest Communications Corporation.

======================

WHOIS Search Results

Your WHOIS Search Results

65.101.142.132

Record Type: IP Address

OrgName: Qwest Communications Corporation

OrgID: QCC-22

Address: 1801 California Street

City: Denver

StateProv: CO

PostalCode: 80202

Country: US

NetRange: 65.100.0.0 - 65.103.255.255

CIDR: 65.100.0.0/14

NetName: QWEST-INET-115

NetHandle: NET-65-100-0-0-1

Parent: NET-65-0-0-0-0

NetType: Direct Allocation

NameServer: AUTHNS1.MPLS.QWEST.NET

NameServer: AUTHNS2.DNVR.QWEST.NET

NameServer: AUTHNS3.STTL.QWEST.NET

Comment:

RegDate: 2001-01-03

Updated: 2006-02-09

OrgAbuseHandle: QIA2-ARIN

OrgAbuseName: Qwest Abuse

OrgAbusePhone: +1-877-886-6515

OrgAbuseEmail: abuse@qwest.net

OrgNOCHandle: QIN-ARIN

OrgNOCName: Qwest IP NOC

OrgNOCPhone: +1-877-886-6515

OrgNOCEmail: support@qwestip.net

OrgTechHandle: QIA-ARIN

OrgTechName: Qwest IP Admin

OrgTechPhone: +1-877-886-6515

OrgTechEmail: ipadmin@qwest.com

Thank you Arnold.

Any indication between you and Rush that this is an adult website? I don't mind legitimate search engines but if this is an X rated site and they are hackers or spammers lying in wait then I will do as Rush suggested plus call them as you mentioned.

They are really hammering my site and have been for a long time. I know next to nothing about this sort of thing so any additional info between the two of you would be greatly appreciated.

Carol

[arnold]

We are having similar hits. See below.

Sun 28 Oct 2007 04:28:52 PM Descendancy for Sarah Baker (I261888) accessed by 70-58-85-101.tukw.qwest.net.

Sun 28 Oct 2007 04:28:51 PM Descendancy for Sarah Baker (I261888) accessed by 70-58-85-101.tukw.qwest.net.

Sun 28 Oct 2007 04:28:44 PM Individual info for Sarah Baker (I261888) accessed by 70-58-85-101.tukw.qwest.net.

Sun 28 Oct 2007 04:28:40 PM Pedigree for Sarah Baker (I261888) 4 Generations accessed by 70-58-85-101.tukw.qwest.net.

I went to CPanelX on my website, as Rush mentioned. I then clicked on

File Manager

Public_html

.htaccess

then clicked on Edit File in upper right-hand corner

added these words: deny from tukw.qwest.net

and clicked on Save

You should notice, if this works, that tukw.qwest.net is denied access to your website and should disappear from your log file.

If we have lost you in terms of CPanelX, please advise.

[Rush]

Carol,

It's not an X rated site, but rather a Quest costumer. Much like someone using Cox Cable, Comcast, or AOL.

In a brief explanation, major ISPs will break down their customers into different areas for their servers.

You could have westtexas.cox.net, newyork.aol.com, or in this case tukw.qwest.net visiting you site.

So, tukw (dot) net is not the same as tukw.qwest.net and as Arnold pointed out earlier, it resolves back to qwest.net.

I suspect this is mostly harmless, but one way to be sure is to look at your server logs. If you have cPanel, you can download the access log.

This is located right on your main cPanel page in 'Raw Access Logs'. If you want, you can download the file from there and email it over to me: genealogy (at) bythedrop (dot) com.

I can see what user-agent they are using then most likely block the agent itself if you don't wish them to visiting your site.

Rush

[Scotty]

TUKW = Tukwila a small town south of Seattle.

I'm retired Qwest so I remember the lingo.... but I dunno what they are doing!

I've gotten them too.... but 22 hits doesn't interest me.

TUKW refers to the local Point of Presence (POP) for the Qwest.net node.

STTL is Seattle .... PORT is Portland, OR .... don't remember the others been 5 years since I've seen a list or had to think about them ...

Scotty in SPKN

[arnold]

RE: Tossing the baby out with the bath water.

That may be what I did last night when I denied access to tukw.qwest.net.

My preference is always to deny access to a specific IP Address which denies access to only one person or, at worst, to not very many. However, with the ISP/host we are discussing, all of you were reporting different IP Addresses ending with tukw.qwest.net. Sooooo, I made the decision to deny access to the ISP/host instead of just a single IP Address. (I make this decision to deny access sooner rather than later as a few years ago all of our genealogical information appeared on CDs being sold on e-bay.)

You can probably see where I am going. A wonderful contributor to our website wrote and asked why she was suddenly denied access to our website last night.

It is a gut call how soon you deny access and how broad the denial is. Since we were badly burned once, I act quickly.

[arnold]

Carol,

There is more.

I have access to an error log produced by my ISP/host which provides information (1) when someone is looking for a file which does not exist, (2) when someone is denied access, and (3) other things that I am probably unaware of.

It is (2) above which is of concern to us. This error log is of importance because it shows if our IP Deny Manager is working. You will see from below that it is working. Last night I denied access to tukw.qwest.net. And this morning when I looked at the error log, IP Address 70.58.85.101 is being denied access to our website. IP Address 70.58.85.101 is used by tukw.qwest.net.

It is my understanding that access to this error log has to be set up by your ISP/host. You must be aware that this log continues to accumulate and accumulate all the access problems your visitors experience. In our case, if I am not careful, the file approaches 1MB in size.

Hope this helps,

Arnold

[sun Oct 28 18:51:13 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_new.gif

[sun Oct 28 18:51:13 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_rpt.gif

[sun Oct 28 18:51:14 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_stats.gif

[sun Oct 28 18:51:14 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_note.gif

[sun Oct 28 18:51:14 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_src.gif

[sun Oct 28 18:51:15 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_admin.gif

[sun Oct 28 18:51:15 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_contact.gif

[sun Oct 28 18:51:15 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/tng/tng_repo.gif

[sun Oct 28 18:51:15 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/images/top.gif

[sun Oct 28 18:51:22 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/genealogy/search.php

[sun Oct 28 18:53:08 2007] [error] [client 70.58.85.101] client denied by server configuration: /home/csdb/public_html/genealogy/search.php

[TNGUSER]

RE: Tossing the baby out with the bath water.

That may be what I did last night when I denied access to tukw.qwest.net.

My preference is always to deny access to a specific IP Address which denies access to only one person or, at worst, to not very many. However, with the ISP/host we are discussing, all of you were reporting different IP Addresses ending with tukw.qwest.net. Sooooo, I made the decision to deny access to the ISP/host instead of just a single IP Address. (I make this decision to deny access sooner rather than later as a few years ago all of our genealogical information appeared on CDs being sold on e-bay.)

You can probably see where I am going. A wonderful contributor to our website wrote and asked why she was suddenly denied access to our website last night.

It is a gut call how soon you deny access and how broad the denial is. Since we were badly burned once, I act quickly.

I'll give Rush's suggestion a try. It sounds like you did the same thing. I'd rather be safe than sorry too.

[TNGUSER]

TUKW = Tukwila a small town south of Seattle.

I'm retired Qwest so I remember the lingo.... but I dunno what they are doing!

I've gotten them too.... but 22 hits doesn't interest me.

TUKW refers to the local Point of Presence (POP) for the Qwest.net node.

STTL is Seattle .... PORT is Portland, OR .... don't remember the others been 5 years since I've seen a list or had to think about them ...

Scotty in SPKN

My list in showlog of this .net company accessing my site goes on and on. Rush wrote in his reply that he didn't think it was an adult site. I'd have to go back and look at my internet history but if they are a host provider then maybe what I clicked on was an adult site that uses them as their host? I don't know much about this kind of thing.

Carol,

It's not an X rated site, but rather a Quest costumer. Much like someone using Cox Cable, Comcast, or AOL.

In a brief explanation, major ISPs will break down their customers into different areas for their servers.

You could have westtexas.cox.net, newyork.aol.com, or in this case tukw.qwest.net visiting you site.

So, tukw (dot) net is not the same as tukw.qwest.net and as Arnold pointed out earlier, it resolves back to qwest.net.

I suspect this is mostly harmless, but one way to be sure is to look at your server logs. If you have cPanel, you can download the access log.

This is located right on your main cPanel page in 'Raw Access Logs'. If you want, you can download the file from there and email it over to me: genealogy (at) bythedrop (dot) com.

I can see what user-agent they are using then most likely block the agent itself if you don't wish them to visiting your site.

Rush

That's what I'm thinking too....that this company is the host provider for the link that I clicked on when I got the adult site.

I don't know if I have cPanel or what that is but I'll look. I use ICDSoft as my host provider. I don't like to mess around too much with the SQL database for fear I'm going to mess things up. If I can find it I'll send it to you. Like Scotty, if it was just a few files and they are a legitimate company then I probably wouldn't care but I have my showlog set to view 500 and often the entire screen is filled with their IP address.

[arnold]

I don't know if I have cPanel or what that is but I'll look. I use ICDSoft as my host provider.

TNGUSER,

It does not look as though you have access to CPanelX. You are provided something close to it, per what your host says at http://www.icdsoft.com/features.php/82. If this link does not work, here is what it states:

ONLINE CONTROL PANEL

Our in-house developed online Control Panel allows you to manage every feature of your hosting account. Following are some (not all) of the options available to you:

- Update your contact information.

- Upload and edit files online.

- Create and delete subdomains.

- Park more domains to your account.

- Check detailed site statistics.

- Create and manage mail accounts.

- Create and manage FTP users.

- Create and manage MySQL databases.

- Install various scripts in your account.

- Enable/disable the FrontPage server extensions.

- Protect parts of your web site.

It is easy to remember how to access your online Control Panel - just add "/cp" after your domain name, such as http://www.example.com/cp.

I don't like to mess around too much with the SQL database for fear I'm going to mess things up.

Going into CPanelX will not cause you problems. You cannot access your SQL database from there without taking an additional step or two. Actually it is good for the beginner to frequent the CPanelX just to learn what is there and what you are able to do as webmaster. Yes, as always, be careful while learning anything new.

[TNGUSER]

TNGUSER,

It does not look as though you have access to CPanelX. You are provided something close to it, per what your host says at http://www.icdsoft.com/features.php/82. If this link does not work, here is what it states:

ONLINE CONTROL PANEL

Our in-house developed online Control Panel allows you to manage every feature of your hosting account. Following are some (not all) of the options available to you:

- Update your contact information.

- Upload and edit files online.

- Create and delete subdomains.

- Park more domains to your account.

- Check detailed site statistics.

- Create and manage mail accounts.

- Create and manage FTP users.

- Create and manage MySQL databases.

- Install various scripts in your account.

- Enable/disable the FrontPage server extensions.

- Protect parts of your web site.

It is easy to remember how to access your online Control Panel - just add "/cp" after your domain name, such as http://www.example.com/cp.

Going into CPanelX will not cause you problems. You cannot access your SQL database from there without taking an additional step or two. Actually it is good for the beginner to frequent the CPanelX just to learn what is there and what you are able to do as webmaster. Yes, as always, be careful while learning anything new.

I'm a little embarrassed...."cpanel" stands for Control Panel. Yes, when I do the /cp it takes me to my log in page for ICDSoft. I looked at all the files in Control Panel and also looked at the SQL data base. I didn't find anything close to what Rush is asking for.

[arnold]

Not to be embarrassed. We are talking about quite different things. Rush and I talk about CPanelX. I suppose it is patented, copyrighted, and who knows what else. What your ISP/host offers is some in-house control panel which is similar to CPanelX. Unfortunately for you, many of us use ISPs/hosts which use CPanelX. I say "unfortunately" because we all know various parts of CPanelX well and can give good guidance on it. We are probably not familiar with in-house models.

I did a search on the Internet to let you see what CPanelX looks like. Go to this link for a looksee. Please note the IP Deny Manager in the lower right-hand corner. You click on that icon and off you go to denying access. No fuss, no muss.

[TNGUSER]

Not to be embarrassed. We are talking about quite different things. Rush and I talk about CPanelX. I suppose it is patented, copyrighted, and who knows what else. What your ISP/host offers is some in-house control panel which is similar to CPanelX. Unfortunately for you, many of us use ISPs/hosts which use CPanelX. I say "unfortunately" because we all know various parts of CPanelX well and can give good guidance on it. We are probably not familiar with in-house models.

I did a search on the Internet to let you see what CPanelX looks like. Go to this link for a looksee. Please note the IP Deny Manager in the lower right-hand corner. You click on that icon and off you go to denying access. No fuss, no muss.

Thanks Arnold. I'll get in touch with ICDSoft Maybe there's a way to deny an IP address and I'm just not finding what I need in my Control Panel to do that. Rush mentioned an htaccess file. I don't have that either. Is that something I can create and upload or is that a file that is available only thru the host provider?

This looks like a Qwest ISP subscriber.

What they might being doing is participating in the MJ-12 search engine project, but you would have to look at the server log files to see what agent they are using.

If your host has cPanel, you can block them using the 'IP Deny Manager'. Another way is to edit the htaccess file on your server directly:

order allow,deny

deny from 65-101-142-132.tukw.qwest.net

allow from all

Rush

Rush, Arnold has replied too. I'll check with ICDSoft for a way to deny that particular address. I also asked him about the htaccess file. I don't see that file in my Control Panel File Manager. I'm curious about the MJ-12 project...what is that?

[arnold]

At times, the file named .htaccess (note preceding dot) is better than IP Deny Manager. We did NOT need to overwhelm you with all this info at the beginning.

To get to .htaccess in CPanelX, click on the File Manager icon, then

click on public_html

click on .htacess

click on Edit File in upper right-hand corner

make your change

click on Save

I'm curious about the MJ-12 project

I know nothing about it. You might want to do an Internet search on it and report back.

[TNGUSER]

This looks like a Qwest ISP subscriber.

What they might being doing is participating in the MJ-12 search engine project, but you would have to look at the server log files to see what agent they are using.

If your host has cPanel, you can block them using the 'IP Deny Manager'. Another way is to edit the htaccess file on your server directly:

order allow,deny

deny from 65-101-142-132.tukw.qwest.net

allow from all

Rush

I couldn't access the cPanel that Arnold gave me the link to without making them my host provider. I contacted my provider ICDSoft Support and they told me how to create an .htaccess file. I'll add the order allow, deny etc. etc. that you posted above. Do I save the file in the root directory or does it go in Admin or someplace else? My last question is how do I deny access to other IP addresses....is this the proper format.....

order allow,deny

deny from 65-101-142-132.tukw.qwest.net

deny from 77.77.77.77

allow from all

Thanks for all your help Rush, Scotty and Arnold too.

[arnold]

I couldn't access the cPanel that Arnold gave me the link to without making them my host provider.

That is odd. I do not use them. I just looked and looked on the Internet until I found an example of one and included it earlier. I searched on "CPanelX".

I would probably deny using 65-101-142-132. Then keep and eye on your access log. If more excessive hits come in with a similar IP Address, then you can block a group of IP Addresses. I never remember the correct text for that, so I have to once again do a search on the Internet.

The worst thing you can do is to block a Domain Name, such as tukw.qwest.net. (Well, that is not a Domain Name, but close enough for what we are discussing.) That is what I did (see earlier posts) and the consequence is that you end up denyiing a whole group of folks all at one time when it is your intention to weed out only one person. Do this kind of denial only when you think you may have a major problem and need to bring out the big webmaster hammer.

[Scotty]

I'll check with ICDSoft for a way to deny that particular address. I also asked him about the htaccess file. I don't see that file in my Control Panel File Manager.

ICDsoft does not have a deny manager. But tech support will show you hows it done.

Scotty

[TNGUSER]

That is odd. I do not use them. I just looked and looked on the Internet until I found an example of one and included it earlier. I searched on "CPanelX".

I would probably deny using 65-101-142-132. Then keep and eye on your access log. If more excessive hits come in with a similar IP Address, then you can block a group of IP Addresses. I never remember the correct text for that, so I have to once again do a search on the Internet.

The worst thing you can do is to block a Domain Name, such as tukw.qwest.net. (Well, that is not a Domain Name, but close enough for what we are discussing.) That is what I did (see earlier posts) and the consequence is that you end up denyiing a whole group of folks all at one time when it is your intention to weed out only one person. Do this kind of denial only when you think you may have a major problem and need to bring out the big webmaster hammer.

Just for the heck of it I checked my showlog and you're right...some legitimate users have tukw.qwest.net with their IP address. So, if I understand what you're saying just deny the 65-101-142-132 and don't add tukw.quest.net as that would block everyone affiliated with that domain or whatever it is.

Thanks Arnold. I had asked where I put the .htaccess file and how to add other addresses if necessary. Can you answer that? I think I was replying to Rush in my example as to the proper format for additional IP addresses.

ICDsoft does not have a deny manager. But tech support will show you hows it done.

Scotty

They did Scotty. Their tech support is excellent and they gave me step by step instructions. In ICDSoft I just have to create a file called .htaccess then add in that file the deny, allow etc. I didn't ask them where that file goes or how to add other IP addresses if it becomes necessary. I'm hoping Rush or Arnold will know or if you have time to answer that would be great.

Thanks.

[arnold]

Just for the heck of it I checked my showlog and you're right...some legitimate users have tukw.qwest.net with their IP address. So, if I understand what you're saying just deny the 65-101-142-132 and don't add tukw.quest.net as that would block everyone affiliated with that domain or whatever it is.

Denying only the IP Address 65.101.142.132 (note use of dots, not dashes; do NOT use dashes) will deny but one user (yes, maybe more users if it is, and it probably is, a dynamic IP Address) is the way to go. If you continue to be hit by other users of tukw.quest.net, get back to us with examples from your access log.

Thanks Arnold. I had asked where I put the .htaccess file and how to add other addresses if necessary. Can you answer that?

Read post # 18 in this thread. Please note that it applies to CPanelX. My guess is that the file should be placed in the directory/folder public_html and should be named

.htaccess This is edited. I misspelled the file name with only one c. mea culpa.

Remember to include the leading dot in its name and do not use upper case.

[TNGUSER]

Denying only the IP Address 65.101.142.132 (note use of dots, not dashes; do NOT use dashes) will deny but one user (yes, maybe more users if it is, and it probably is, a dynamic IP Address) is the way to go. If you continue to be hit by other users of tukw.quest.net, get back to us with examples from your access log.

Read post # 18 in this thread. Please note that it applies to CPanelX. My guess is that the file should be placed in the directory/folder public_html and should be named

.htacess

Remember to include the leading dot in its name and do not use upper case.

I wrote back to my ICDSoft tech support. The file is .htaccess (two c's as opposed to one) and it goes in my root directory. The correct format for adding other IP addresses is........

order allow,deny

deny from 33.33.33.33

deny from 77.77.77.77

deny from (IP address)

allow from all

Thanks Arnold, Rush and Scotty for all your help. I will definitely follow up in this forum if this tukw.qwest.net continues to be a problem. They access every day and it's hundreds of files. I can't imagine what they could be looking for if this is not a search engine like google. I didn't get any hits on search results in the first few pages of what the "M12 Project" is relating to search engines or searching web sites in general.

Carol

[arnold]

Thanks for catching my typo. I always, but *not* this time, cut-and-paste to prevent this kind of goof.

If you posted your URL, I missed it. If you do not mind, I think that some of us would like to see your access log.