DOJ Indexing Site

[Micheal Mathews]

While it's not the end of the genealogy world or maybe not even a topic for the forums, I would like to know if anyone else has noticed from your access logs and/or seperate site statistics that the United States Department of Justice is indexing your genealogy site(s)? I just hate the thought of my ancesetors data being pulled into a government database for mining.

I keep seeing repeated activity from the DOJ, and they appear to be taking large amounts of data in stages.

We'll see if their spiders follow Robots.txt protocols and knock it off.

[arnold]

1. Would you mind showing us the DOJ entries you are finding in your log?

2. You can easily deny an IP Address, or even an ISP, from hitting your website. If your ISP provides you with cPanelX, go to IP Deny Manager and deny access to your website to any IP Addresses or ISP domain names. Further, you can design your own 403 error page (Forbidden). I design mine to state which IP Addresses are being denied and why. I include my e-mail address. This lets them contact me and explain why they are hitting our website so hard. Remember, it is your website and you can protect.a great deal of it.

3. An additional protection is to not allow GEDCOM downloads.

[Micheal Mathews]

1. Would you mind showing us the DOJ entries you are finding in your log?

Here is a sample; there are pages of this type of activity:

Thu 18 May 2006 06:59:34 AM Photos, Histories and Other Documents, Family of Marshall William Channel (I401) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:59:23 AM Pedigree for Arthur Levi Channel (I7) 4 Generations accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:59:22 AM Individual info for Arthur Levi Channel (I7) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:59:20 AM Timeline (Arthur Levi Channel) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:59:17 AM Descendancy for Arthur Levi Channel (I7) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:59:06 AM Pedigree for Arthur Levi Channel (I7) 4 Generations accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:58:55 AM Descendancy for Arthur Levi Channel (I7) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:58:42 AM Timeline (Arthur Levi Channel) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:58:25 AM Individual info for Arthur Levi Channel (I7) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:57:47 AM Pedigree for Arthur Levi Channel (I7) 4 Generations accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:56:41 AM What's New (past 45 days) accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:56:33 AM Report: Channell accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:56:26 AM Report: Channell accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:56:12 AM Report: Channell accessed by wdcsun16.usdoj.gov.

Thu 18 May 2006 06:54:13 AM Browse All Histories accessed by wdcsun16.usdoj.gov.

Arnold says:

2. You can easily deny an IP Address, or even an ISP, from hitting your website. If your ISP provides you with cPanelX, go to IP Deny Manager and deny access to your website to any IP Addresses or ISP domain names. Further, you can design your own 403 error page (Forbidden). I design mine to state which IP Addresses are being denied and why. I include my e-mail address. This lets them contact me and explain why they are hitting our website so hard. Remember, it is your website and you can protect.a great deal of it.

Unfortunately, Godaddy.com has IMHO a terrible hosting panel. I am messing with the .htaccess file now trying to get a page like you are stating. Care to help me a bit with the proper entries? When I made my first attempt it locked me completely out of the website.

Thanks for you help and interest.

Micheal Mathews

3. An additional protection is to not allow GEDCOM downloads.

[arnold]

Thank you.

There are times when someone from a governmental office is doing personal genealogical research.

When you suspect not, go to Google and enter the address, such as wdcsun16.usdoj.gov in your case, and see what is being discussed. Go to both "Web" and "Groups."

Also, going to WhoIs and entering the IP Address may let you know if it is a robot hitting your website.

[waltreams]

From the looks of it, you may have a distant relative in the DOJ. They are probably interested in family history also.

[kepstein]

3. An additional protection is to not allow GEDCOM downloads.

How do I do this; do I set Max Generations, GEDCOM download to 0?

Thanks,

Ken

[arnold]

How do I do this; do I set Max Generations, GEDCOM download to 0?

Go to Administration/Trees. Look at the bottom option of the "Existing Tree Information" webpage for the Tree you are interested in. Enter a checkmark in the "Don't allow users to download GEDCOM files" option.

My guess is that if you enter a 0, as you suggest, then you or anyone allowed to download a GEDCOM, could not.

[JustTheFax]

Do you allow anyone to access your site, or do you require log in?

I have my site set to require log in and information on living people is not is not displayed for most users.

I also do not allow most users to download gedcoms

I also think a distant cousin has found your web site, see if you can find out who it is first. If it is a distant cousin they may be able to help you fill in missing information.

[Micheal Mathews]

Do you allow anyone to access your site, or do you require log in?

I have my site set to require log in and information on living people is not is not displayed for most users.

I also do not allow most users to download gedcoms

I also think a distant cousin has found your web site, see if you can find out who it is first. If it is a distant cousin they may be able to help you fill in missing information.

Actually, the issue turned out to be that the domain name I was using was previously registered to an organization that was critical of the government. I filed a FOIA request with DOJ to find out what the heck was going on and why, and they (DOJ) responded with 100 free pages of information they had on the domain's previous owners and their activities. That was enough for me to abandon the domain and move to a friendlier family history domain name.

Gotta' love the irony of it all - and no, no relatives that I know of or that the DOJ knows of or admits to having that used DOJ computers to access the former site.

Micheal