pwaldron Posted August 5, 2013 Report Share Posted August 5, 2013 I've installed TNG version 9.2.1 athttp://www.pwaldron.info/tng/and successful uploaded and imported a small GEDCOM of 120 people into a test tree.However, I am having problems importing my main GEDCOM of 59,736 people into my main tree.I have uploaded it by FTP so it is in the GEDCOM folder on the website (Size 36.2m according to TNG; 37,955,637 bytes according to MS Windows) and I can select it athttp://www.pwaldron.info/tng/admin_dataimport.phpI've experimented with ticking various combinations of boxes but I'm sure the import should work if I leave everything unticked and choose to replace all current data in my selected Destination Tree.When I click the Import Data button, I see a pop-up which saysOpening sandpit.ged ... (followed by a spinning timer)and under that a Stop link back tohttp://www.pwaldron.info/tng/admin_dataimport.php#The browser tab also says Connecting ... with a spinning timer in front of the word.The browser remains in this state for 24 hours or more if I do nothing.I see no progress indicator while the large import is happening (although I saw it for the small import, as http://tng.lythgoes.net/wiki/index.php?title=Import_Data suggests I should).However, I can watch what's happening if I go tohttp://www.pwaldron.info/tng/statistics.phpThe Date of Last GEDCOM Import updates immediately to the time that the import started (there doesn't seem to be a field to show the time that the last import ended).Roughly 100 individuals per second are imported until the import stalls.My two most recent attempts stalled after 49,095 individuals and after 29,949 individuals respectively.There is no error message when the import stalls to tell me whether the problem is with the GEDCOM file or the PHP settings or something else.When I click the Stop link, the message changes to "Import stopped", with no Resume link.If I close the pop-up and click the Import button again, the database is wiped clean.At http://pwaldron.info/tng/admin_importconfig.php I have ticked the box which says "Save to allow resume if import fails" but the Resume link (which I saw initially) is no longer appearing if I click the Stop link.I've tried the import numerous times over the last four days and only once has it completed; TNG has quickly identified a few silly date errors in my GEDCOM, so I am anxious to upload a corrected GEDCOM.My hosting service is dreamhost.comphp.ini, according to http://pwaldron.info/tng/admin_phpinfo.php, is /etc/php5/cgi/php.ini so I guess I can't change it.Am I doing something wrong, or can I somehow override the default php.ini settings, or do I need to get dreamhost.com to change some parameters?Thanks in advance for any help or advicePaddy Waldron Quote Link to comment Share on other sites More sharing options...
theKiwi Posted August 5, 2013 Report Share Posted August 5, 2013 I think you'll need to contact dreamhost and ask them about this - in all likelihood the import is getting timed out on their server based on time a process runs, or other use of resources.They might be able to adjust the timeout for you.Roger Quote Link to comment Share on other sites More sharing options...
pwaldron Posted August 5, 2013 Author Report Share Posted August 5, 2013 I think you'll need to contact dreamhost and ask them about this - in all likelihood the import is getting timed out on their server based on time a process runs, or other use of resources.They might be able to adjust the timeout for you.Thanks, Roger.I've sent a message to dreamhost support and will let you know if they can sort out the problem.\pw Quote Link to comment Share on other sites More sharing options...
genhcn Posted August 5, 2013 Report Share Posted August 5, 2013 It looks like the problem I had ....Except ... my english is bad to explainIf Ken Roy could move, he helped me and DarenThe solution for meMy host response :Ça ressemble au problème que j'ai eu....Sauf que... mon anglais est mauvais pour expliquerSi Ken Roy pouvait passer, il m'a aidé ainsi que DarenLa solution, dans mon casRéponse de mon hébergeur :"Nous avons effectué davantage de vérifications et nous avons été en mesure de constater que le problème était causé par le fait que ce script prend beaucoup de ressources systèmes et le module ModSecurity qui doit analyser toutes les requêtes effectuées par l'importation n'était pas en mesure de les prendre en charge correctement à cause du nombre de requêtes.Nous avons donc exclu le fichier "/admin_gedimport.php" qui est utilisé pour l'importation des données de la configuration de ModSecurity de votre compte." Quote Link to comment Share on other sites More sharing options...
Ken Roy Posted August 5, 2013 Report Share Posted August 5, 2013 It looks like the problem I had ....Except ... my english is bad to explainIf Ken Roy could move, he helped me and DarenThe solution for meMy host response :Ça ressemble au problème que j'ai eu....Sauf que... mon anglais est mauvais pour expliquerSi Ken Roy pouvait passer, il m'a aidé ainsi que DarenLa solution, dans mon casRéponse de mon hébergeur :"Nous avons effectué davantage de vérifications et nous avons été en mesure de constater que le problème était causé par le fait que ce script prend beaucoup de ressources systèmes et le module ModSecurity qui doit analyser toutes les requêtes effectuées par l'importation n'était pas en mesure de les prendre en charge correctement à cause du nombre de requêtes.Nous avons donc exclu le fichier "/admin_gedimport.php" qui est utilisé pour l'importation des données de la configuration de ModSecurity de votre compte."As Roger indicated, it is likely a timeout due to the amount of resources used. In the case of genhcn, Mod Security also came into play. Translation of the host's reply is as follows:We conducted more audits and we were able to determine that the problem was caused by the fact that the script takes a lot of system resources and the ModSecurity module that analyzes all the requests made by the import was not able to handle it properly because of the number of queries.We exclude the "/ admin_gedimport.php" file that is used to import data configuration from ModSecurity for your account. Quote Link to comment Share on other sites More sharing options...
genhcn Posted August 5, 2013 Report Share Posted August 5, 2013 I understand the difficult problemIs it the same as me? Quote Link to comment Share on other sites More sharing options...
pwaldron Posted August 6, 2013 Author Report Share Posted August 6, 2013 I understand the difficult problemIs it the same as me?I'm not sure that my problem is exactly the same as genhcn's, but it is similar.On Dreamhost's advice:(a) Athttps://panel.dreamhost.com/index.cgi?tree=...p;dsid=16712945I changedPHP 5.2.x FastCGItoPHP 5.3.x CGIand clicked Change settings.and(b) I also uploaded/home/USERNAME/.php/5.3/phprcreading; Custom PHP Configurationmax_execution_time = 900max_input_time = 900memory_limit = 150Mupload_max_filesize = 64Mpost_max_size = 64MNow http://pwaldron.info/tng/statistics.php tells me that all 59,736 individuals and 21,013 families in my latest GEDCOM have been imported.However, I am still looking at two spinning timers athttp://pwaldron.info/tng/admin.phpso the Import is still not exiting cleanly. Quote Link to comment Share on other sites More sharing options...
Ken Roy Posted August 6, 2013 Report Share Posted August 6, 2013 I'm not sure that my problem is exactly the same as genhcn's, but it is similar.On Dreamhost's advice:(a) Athttps://panel.dreamhost.com/index.cgi?tree=...p;dsid=16712945I changedPHP 5.2.x FastCGItoPHP 5.3.x CGIand clicked Change settings.and(b) I also uploaded/home/USERNAME/.php/5.3/phprcreading; Custom PHP Configurationmax_execution_time = 900max_input_time = 900memory_limit = 150Mupload_max_filesize = 64Mpost_max_size = 64MNow http://pwaldron.info/tng/statistics.php tells me that all 59,736 individuals and 21,013 families in my latest GEDCOM have been imported.However, I am still looking at two spinning timers athttp://pwaldron.info/tng/admin.phpso the Import is still not exiting cleanly.Sounds like Dreamhost is also not configured to allow intermediate feedback of upload results. Note this is not the accurate term. But if TNG gets partial output from a process then it can display the upload progress bar and can be resumed if it times out.If the host only allows the script results to be posted on completion then all you get is a spinning icon until the process is completed, and time outs cannot be resumed. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.