Jump to content
TNG Community
pwaldron

More GEDCOM import problems

Recommended Posts

pwaldron

I've installed TNG version 9.2.1 at

http://www.pwaldron.info/tng/

and successful uploaded and imported a small GEDCOM of 120 people into a test tree.

However, I am having problems importing my main GEDCOM of 59,736 people into my main tree.

I have uploaded it by FTP so it is in the GEDCOM folder on the website (Size 36.2m according to TNG; 37,955,637 bytes according to MS Windows) and I can select it at

http://www.pwaldron.info/tng/admin_dataimport.php

I've experimented with ticking various combinations of boxes but I'm sure the import should work if I leave everything unticked and choose to replace all current data in my selected Destination Tree.

When I click the Import Data button, I see a pop-up which says

Opening sandpit.ged ... (followed by a spinning timer)

and under that a Stop link back to

http://www.pwaldron.info/tng/admin_dataimport.php#

The browser tab also says Connecting ... with a spinning timer in front of the word.

The browser remains in this state for 24 hours or more if I do nothing.

I see no progress indicator while the large import is happening (although I saw it for the small import, as http://tng.lythgoes.net/wiki/index.php?title=Import_Data suggests I should).

However, I can watch what's happening if I go to

http://www.pwaldron.info/tng/statistics.php

The Date of Last GEDCOM Import updates immediately to the time that the import started (there doesn't seem to be a field to show the time that the last import ended).

Roughly 100 individuals per second are imported until the import stalls.

My two most recent attempts stalled after 49,095 individuals and after 29,949 individuals respectively.

There is no error message when the import stalls to tell me whether the problem is with the GEDCOM file or the PHP settings or something else.

When I click the Stop link, the message changes to "Import stopped", with no Resume link.

If I close the pop-up and click the Import button again, the database is wiped clean.

At http://pwaldron.info/tng/admin_importconfig.php I have ticked the box which says "Save to allow resume if import fails" but the Resume link (which I saw initially) is no longer appearing if I click the Stop link.

I've tried the import numerous times over the last four days and only once has it completed; TNG has quickly identified a few silly date errors in my GEDCOM, so I am anxious to upload a corrected GEDCOM.

My hosting service is dreamhost.com

php.ini, according to http://pwaldron.info/tng/admin_phpinfo.php, is /etc/php5/cgi/php.ini so I guess I can't change it.

Am I doing something wrong, or can I somehow override the default php.ini settings, or do I need to get dreamhost.com to change some parameters?

Thanks in advance for any help or advice

Paddy Waldron

Share this post


Link to post
Share on other sites
theKiwi

I think you'll need to contact dreamhost and ask them about this - in all likelihood the import is getting timed out on their server based on time a process runs, or other use of resources.

They might be able to adjust the timeout for you.

Roger

Share this post


Link to post
Share on other sites
pwaldron

I think you'll need to contact dreamhost and ask them about this - in all likelihood the import is getting timed out on their server based on time a process runs, or other use of resources.

They might be able to adjust the timeout for you.

Thanks, Roger.

I've sent a message to dreamhost support and will let you know if they can sort out the problem.

\pw

Share this post


Link to post
Share on other sites
genhcn

It looks like the problem I had ....

Except ... my english is bad to explain

If Ken Roy could move, he helped me and Daren

The solution for me

My host response :

Ça ressemble au problème que j'ai eu....

Sauf que... mon anglais est mauvais pour expliquer

Si Ken Roy pouvait passer, il m'a aidé ainsi que Daren

La solution, dans mon cas

Réponse de mon hébergeur :

"Nous avons effectué davantage de vérifications et nous avons été en mesure de constater que le problème était causé par le fait que ce script prend beaucoup de ressources systèmes et le module ModSecurity qui doit analyser toutes les requêtes effectuées par l'importation n'était pas en mesure de les prendre en charge correctement à cause du nombre de requêtes.

Nous avons donc exclu le fichier "/admin_gedimport.php" qui est utilisé pour l'importation des données de la configuration de ModSecurity de votre compte."

Share this post


Link to post
Share on other sites
Ken Roy

It looks like the problem I had ....

Except ... my english is bad to explain

If Ken Roy could move, he helped me and Daren

The solution for me

My host response :

Ça ressemble au problème que j'ai eu....

Sauf que... mon anglais est mauvais pour expliquer

Si Ken Roy pouvait passer, il m'a aidé ainsi que Daren

La solution, dans mon cas

Réponse de mon hébergeur :

"Nous avons effectué davantage de vérifications et nous avons été en mesure de constater que le problème était causé par le fait que ce script prend beaucoup de ressources systèmes et le module ModSecurity qui doit analyser toutes les requêtes effectuées par l'importation n'était pas en mesure de les prendre en charge correctement à cause du nombre de requêtes.

Nous avons donc exclu le fichier "/admin_gedimport.php" qui est utilisé pour l'importation des données de la configuration de ModSecurity de votre compte."

As Roger indicated, it is likely a timeout due to the amount of resources used. In the case of genhcn, Mod Security also came into play. Translation of the host's reply is as follows:

We conducted more audits and we were able to determine that the problem was caused by the fact that the script takes a lot of system resources and the ModSecurity module that analyzes all the requests made by the import was not able to handle it properly because of the number of queries.

We exclude the "/ admin_gedimport.php" file that is used to import data configuration from ModSecurity for your account.

Share this post


Link to post
Share on other sites
genhcn

I understand the difficult problem

Is it the same as me?

Share this post


Link to post
Share on other sites
pwaldron

I understand the difficult problem

Is it the same as me?

I'm not sure that my problem is exactly the same as genhcn's, but it is similar.

On Dreamhost's advice:

(a) At

https://panel.dreamhost.com/index.cgi?tree=...p;dsid=16712945

I changed

PHP 5.2.x FastCGI

to

PHP 5.3.x CGI

and clicked Change settings.

and

(b) I also uploaded

/home/USERNAME/.php/5.3/phprc

reading

; Custom PHP Configuration

max_execution_time = 900

max_input_time = 900

memory_limit = 150M

upload_max_filesize = 64M

post_max_size = 64M

Now http://pwaldron.info/tng/statistics.php tells me that all 59,736 individuals and 21,013 families in my latest GEDCOM have been imported.

However, I am still looking at two spinning timers at

http://pwaldron.info/tng/admin.php

so the Import is still not exiting cleanly.

Share this post


Link to post
Share on other sites
Ken Roy

I'm not sure that my problem is exactly the same as genhcn's, but it is similar.

On Dreamhost's advice:

(a) At

https://panel.dreamhost.com/index.cgi?tree=...p;dsid=16712945

I changed

PHP 5.2.x FastCGI

to

PHP 5.3.x CGI

and clicked Change settings.

and

(b) I also uploaded

/home/USERNAME/.php/5.3/phprc

reading

; Custom PHP Configuration

max_execution_time = 900

max_input_time = 900

memory_limit = 150M

upload_max_filesize = 64M

post_max_size = 64M

Now http://pwaldron.info/tng/statistics.php tells me that all 59,736 individuals and 21,013 families in my latest GEDCOM have been imported.

However, I am still looking at two spinning timers at

http://pwaldron.info/tng/admin.php

so the Import is still not exiting cleanly.

Sounds like Dreamhost is also not configured to allow intermediate feedback of upload results. Note this is not the accurate term. But if TNG gets partial output from a process then it can display the upload progress bar and can be resumed if it times out.

If the host only allows the script results to be posted on completion then all you get is a spinning icon until the process is completed, and time outs cannot be resumed.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×