Jump to content
TNG Community
Ian Davidson

Importing FTM 2019 GEDCOM with media Categories into TNG Collections

Recommended Posts

Ian Davidson

On my desktop I've been using Family Tree Maker as my preferred data entry software for about the last 10 years. For somewhat longer than that I've been using TNG, more recently integrated with Wordpress, to display it, although I also sync to Ancestry. I've always been disappointed I couldn't get media automatically allocated to the correct TNG Collection after importing a GEDCOM produced by FTM. Sadly, my TNG tree hadn't been updated for quite a while, partly because my media links were a mess, with many "File not found" errors, multiple links to the same file etc and I kept procrastinating about facing the challenge to tidy it all up :-(. Over the Christmas/New Year break I bit the bullet, upgraded TNG from v. 11.x to 12.2 and spent some time modifying the CSS files to harmonise with my Wordpress theme (another thing I'd been putting off). Then I started researching the media collections issue; the excellent "gotchas" item in this very wiki was the hallelujah moment when I started to see light at the end of the tunnel! That article led me to Stephen Conner's FTM to TNG Gedcom Fixer program, and also Robin Richmond's GEDCOM Converter mod. Because I was more specifically interested in the media collections I focused on the former.

So I spent days modifying the media filenames using FTM to match the requirements of Stephen's Fixer program, a process that's easy enough if quite tedious. I had replicated my website onto my desktop (in a WAMP environment) to fiddle with the CSS and some other mods, so I could test importing a new GEDCOM without jeopardising the live site. I deleted all the existing media files from the tng \media folder and copied the newly-renamed files from my FTM media folder into their respective new website folders. When I imported the fixed GEDCOM file (selecting the options Replace all current data and Import media if present) I was confused to see weird results eg only 12 out of 37 headstones correctly allocated in TNG, 50 out of 269 documents, etc. I spent more time researching and studying the GEDCOM files before & after the fixer program but couldn't see any issues. Then I used PHPMyAdmin to study the TNG database and noticed entries in the media table that didn't look right, and furthermore the number of rows didn't correlate with the number of files. It finally twigged that TNG doesn't clear (ie empty) the media tables when doing a GEDCOM import, even if the option to replace all data is selected. I came to realise that renaming the media files had therefore left things even more messy than previously, and that what I should do is clear all the media tables before doing an import so that TNG would rebuild it from scratch on import. With a view to helping other in the same predicament who may not have the expertise or inclination to play around in the database with PHPMyAdmin I quickly found an excellent backgrounder and tutorial by Bob Cumberbatch on doing it the easy way from within TNG itself :-) (which also confirmed my thoughts). When I followed Bob's procedure to clear all the tables, but NOT restoring the media table (thereby forcing TNG to rebuild it) and re-imported the GEDCOM, :-) Happy Days :-)  - all looks great!

So if you've waded through all the background, here's my step-by-step takeaway to get your FTM media Categories reflected in TNG Collections. I've tried to pull together tips and procedures from several different places together with my experience into one resource. This has been produced with FTM 2019 and TNG 12.2, but I believe it should work with at least the immediately previous versions of both (but not FTM 2012). It worked for me, but use it at your own risk ;-).

NOTE: This should only be done if you use only Family Tree Maker (or Ancestry and sync to FTM) for your data entry and use TNG only to display it!

NOTE 2: Sorry Mac users but this process will only work on Windows machines, because Stephen Conner's GEDCOM Fixer program only runs on Windows (though maybe OK using a Windows emulator? Anyone able to confirm please comment below!)

  1. I highly recommend replicating your online TNG site to your desktop and backing everything up (including all your media files, before renaming them) so you don't jeopardise your live site in case this doesn't work for you. Do all your modifications on your desktop site and check everything before going through the same steps on your live site. That's beyond the scope of this article, but if you Google or search YouTube for "WAMP" or "XAMPP" you'll find plenty of guides to get a local webserver set up. Also beyond the scope of this article you'll need to be familiar with a File Transfer Protocol (FTP) program. Google or YouTube "FileZilla" for a start.
  2. Read the "gotchas" article on this wiki and the detailed information available on Stephen Conner's page (with its linked help docs) and Bob Cumberbatch's page, all as linked above, so that you understand the issues and what the proposed fixes do.
  3. Using FTM, rename your media files to match the requirements of Stephen Conner's fixer program. This might take quite some time, depending on your individual situation.
  4. Do the following in TNG. In Administration | Setup | Configuration | General Settings, Paths & Folders, setup the new media folders eg:
    image.png
  5. In Administration | Setup  | Configuration | Import Settings, make sure the Local [*] paths match those used by FTM on your computer, eg on my system this is (where "XXX" is my Windows username and "YYY" is my tree name in FTM) - yours might be very different:
    image.png
    If you're not sure what this path is, either open your GEDCOM file (see below) in Notepad or similar, scroll to the bottom where the media entries are located and look for a line like the following:
    1 FILE D:\Users\XXX\Documents\Family Tree Maker\YYY Media\p_4 generations.jpg
    or in Family Tree Maker (assuming you're using the default database path) go to the Tools | Options menu, General tab, copy the database path, eg for me this is:image.png
    paste it into the TNG box and add "\[your tree name] Media\" on to the end of it, replacing [your tree name] with the actual name of your FTM tree.
    Note that in FTM, all media files are stored in the one folder, so you'll use the same path for all of the TNG settings. This path will be removed from that recorded in the GEDCOM file by TNG for each media file so that it can store just the actual filename.
  6. In Administration | Trees, for your selected or only tree, click the "Clear" icon:
    image.png
    and answer OK to the prompt.
  7. When you get to this step on your live site you'd use your selected FTP program (eg FileZilla) for these steps, but initially of course this should be on your local machine so you'd use Windows Explorer/My Computer on Windows or Finder on a Mac to delete all the files in yout TNG media folder, then copy all your renamed media files from the FTM media folder into their respective new folders on your TNG installation, as described in Stephen Conner's documentation.
  8. In FTM, select the File | Export menu. The important options are highlighted here, other options at your discretion:
    image.png
    then at the next prompt select Other as the Destination:
    image.png
    Note the filename chosen when you click OK; you'll need it next.
  9. Follow the procedures in Stephen Conner's documentation to setup and run the fixer program, including installing any necessary Windows support files (one-off).
  10. From TNG's Administration | Import/Export page, select your "fixed" file, making sure it's not the original one exported from FTM:
    image.png
    Select the important options as highlighted below, other options at your discretion:
    image.png
    and finally click Import Data.
  11. Wait for a short while, then verify what you expect to see. Hopefully like me, you'll be a happy person! Enjoy!

A huge thank you to Stephen Conner, Robin Richmond and Bob Cumberbatch for throwing light on this area, and certainly not least Darrin Lythgoe for making TNG so flexible and such a capable environment for our purposes!

Cheers

Ian Davidson

 

 

Share this post


Link to post
Share on other sites
Ken Roy

Ian,

looks like a nice write up that should be an article on the TNG Wiki.

Share this post


Link to post
Share on other sites
Ian Davidson

Thanks Ken, I might leave it here for a while first, to take into account any comments /amendments!

Share this post


Link to post
Share on other sites
Meetoura

Thanks for this very informative post, Ian.  It will prove very useful to me in the future.

Share this post


Link to post
Share on other sites
Robin Richmond

That's amazing, Ian, and thanks for the kind words.  If there is anything that the Gedcom Converter could do to help, let me know.

I'll note that it has been my experience that Ancestry.com does not like changes that I make to image files or filenames.  I'm sure that Ancestry.com does not pick up the new filename, but if the old filename is not re-downloaded when Ancestry.com and FTM are synced, then most of that issue goes away.  It has become more of a problem when I change an image file and want that change to be reflected in Ancestry.com.  My only choice there has been to delete the attachment from the Ancestry.com gallery and upload the modified image file.

It has been a serious problem, oh, 3 to 5 times in the six years I have been using Ancestry.com and FTM, when the Ancestry.com-FTM sync process fails and I have to build my FTM database from scratch by re-downloading the Ancestry.com tree.  When that happens:

  1. All Gedcom record number change, which is just horrible, 
  2. The image files generated by sources are numbered differently, and
  3. All the original filename and content are downloaded.

But that hasn't happened since FTM2017 was introduced, so maybe all that hassle is in the past.

Ian - agree with Ken that your posting (subject to change, as you mentioned) would make a great Wiki article, but I also wonder (and I hope that others will weigh in on this) if your posting here on the Community will get the audience it deserves.  My inclination is to recommend that you post a message on the discussion list that refers to your tng.community posting, but I'm not confident enough of my understand of the two communication vehicles to go that far.

Frankly, I really don't understand tng.community vis-a-vis the tngusers2 discussion list.  I think that postings on tng.community are seen my a small fraction of the number of people that see discussion list postings, but I can't be sure.  For myself,  I see all discussion list postings but I almost never go to tng.community unless someone tells me about a posting there (as you did today, for which I thank you).

Of course, tng.community handles inline images and attachments much better than the discussion list (actually, I'm not even sure what the discussion list limits are on images and attachments are, but I know that they exist).  And tng.community makes it easier to read a thread with specific quotes, where email clients automatically quote the entire thread.

A decent solution to those problems is to start on tng.community, and then to send a discussion list message that refers to the community conversation, at which point interested parties can discuss the issue without cluttering the discussion list.

Can anyone provide some insight into the visibility of tng.community postings or the wisdom of advertising (as it were) certain tng.community conversations on the discussion list?

- Robin

Share this post


Link to post
Share on other sites
Ian Davidson

Hi Robin,

Wow, thanks, a fair bit to digest here! The Ancestry sync issue re image renaming etc is a definite worry, I'll check that out... Also need to get up to speed on the wiki vs forum etc :-)

Regarding your Gedcom Converter mod, I quite liked the implementation as a mod, integrated as it is with TNG, and what it does is awesome. As I understand it there's a bit of overlap between it and Stephen's Gedcom Fixer, but because my main aim was to reflect FTM Categories in TNG Collections I worked with the latter. In an ideal world FTM would export standard Gedcoms with the Category correctly recorded. Failing that I was actually pondering writing a program (I'm a programmer by profession, though with no more than a passing familiarity with PHP) that would duplicate Stephen's result but get the FTM Category for each media item directly from the FTM database rather than having to rename the files. I had a vague feeling that FTM used an SQLite database but a quick check indicated that's either incorrect, or true but it's encrypted (or both!). I've actually sent off an email to MacKiev seeking info on writing a plugin for FTM and API docs, which might in fact be a workable solution if they respond affirmatively, but cynic that I am suspect that won't happen. My understanding is that a number of other desktop genealogy programs use SQLite (with good reason) but do not encrypt the database, meaning that an active and resourceful community can develop all sorts of goodies that enhance the standard product - it's a shame FTM isn't one of those. Anyway, if your mod could be tweaked to provide similar functionality in terms of the Category - Collection issue that'd be great. As a mod available through the TNG website and written in PHP it's arguably a more attractive proposition than a custom, compiled program because if the originator loses interest, gets hit by a bus, their website goes AWOL etc a mod can be picked up by others in the community for ongoing development and support whereas a compiled program might just fade away...

Thanks again & regards

Ian

Share this post


Link to post
Share on other sites
Robin Richmond
On 1/5/2020 at 8:11 PM, Ian Davidson said:

I've actually sent off an email to MacKiev seeking info on writing a plugin for FTM and API docs, which might in fact be a workable solution if they respond affirmatively, but cynic that I am suspect that won't happen

I'm very skeptical, too. If they could do it, then surely they would have. The lack of a query tool, much less batch update tool is an outrageous omission, and I've spoken to their help desk about it. I use FTM because it used to be the only program that would sync with Ancestry.com and produce Gedcom files, but I really don't like it, especially because of absence of a query tool.  But it also leaves data behind - especially in citations - when it exports Gedcom files.  (Oh, FWIW, the Gedcom export directly from Ancestry.com is Horrible.)

There are not many options, however.  I did look at Roots Magic, and liked it even less.  At this point, the only remaining options seem to be to try Legacy or dump Ancestry.com.

Share this post


Link to post
Share on other sites
GuyMarsh

I have been using My Heritage now for a number of years.   The GEDCOM seems to work good with TNG.

http://www.marshclarkefamilytree.com  

Share this post


Link to post
Share on other sites
Robin Richmond
47 minutes ago, GuyMarsh said:

I have been using My Heritage now for a number of years.   The GEDCOM seems to work good with TNG.

http://www.marshclarkefamilytree.com

Thanks Guy. I hear good things about their desktop software, but My Heritage is a collaborative, resource-rich website that competes with Ancestory.com.  Thus the MyHeritage desktop software (which I think is free) doesn't communicate with Ancestry.com. I wonder if you have (or anyone else reading this has) a good feel for the real differences between their resources.

As I understand it, my Heritage has more European data sources; Ancestry has more American sources.  As it turns out, over 260 of my direct ancestors lived in what is now the USA, and probably 95% of my 11,000 extended family members are/were in the USA. (I have enough holes in my USA ancestry and long-enough ancestral lines in the USA that I haven't bothered much with non-immigrant European ancestors).  With so many hooks between my Ancestry.com tree and Ancestry.com sources, and the expectation that Ancestry.com offers more resources that are useful to me, I believe that it makes sense for me to stay with Ancestry.com. 

But I am still interested in discovering more meaningful comparison between the two sitesr. 

FWIW, Ancestry has  indexed and imaged birth/death/marriage records from a number of states (not all, but, fortunately, several of the states where I have history).  They have  many town and church records from colonial Massachusetts (including Plymouth Colony), wills, probate, and land transactions from many states and counties, and dozens of indexed books about states and counties or specific families.  But of course, the words "several" and"many" don't really quantify what they have.

I'm sure that My Heritage, like Ancestry, has indexed all of the US Federal Censuses, and (again) 'many' state census, with high resolution census page images, and access to FindAGrave and BillionGraves.

I wish that I could find a comparison that is more quantitative than "more American vs more European": :-)

Share this post


Link to post
Share on other sites
Ian Davidson

Thanks Robin and Guy for your input. I had been using Legacy for many years before switching to FTM, primarily because of the links to Ancestry and the data they've indexed. My heritage is 100% European (mainly British Isles) and Australian, and Ancestry has pretty good coverage of both. I looked at MyHeritage a few years ago when Ancestry announced they were discontinuing further development of FTM, before MacKiev took it on. When that happened I decided to stick with what I already had, but I'd be really interested in a full-blown comparison. I feel Ancestry is pricing itself out of the market and what with ongoing issues with FTM, and their closed system, I could be tempted to switch if indeed they have as good or better coverage of the countries I'm interested in...

Cheers

Ian

Share this post


Link to post
Share on other sites
Ian Davidson

PS Forgot to mention I got a return email from MacKiev seeking more information about what I had planned regarding an FTM addin, so it does seem they're not totally closed to such things. I'm sure they'd jump at the chance if I told them I wanted to create a GEDCOM export addin because their implementation sucks ;-)

Share this post


Link to post
Share on other sites
Ian Davidson
On 1/6/2020 at 11:04 AM, Robin Richmond said:

I'll note that it has been my experience that Ancestry.com does not like changes that I make to image files or filenames.  I'm sure that Ancestry.com does not pick up the new filename, but if the old filename is not re-downloaded when Ancestry.com and FTM are synced, then most of that issue goes away.  It has become more of a problem when I change an image file and want that change to be reflected in Ancestry.com.  My only choice there has been to delete the attachment from the Ancestry.com gallery and upload the modified image file.

Hi Robin,

As per my original post, I changed the filenames of all my media files except for documents (which I confirm Stephen Conner's remarks in that they automatically get allocated to the Documents collection in TNG). Since then I've been test-importing into TNG on my local WAMP server before uploading to my live site, fine-tuning my processes to get the results that I want. This particularly included some more bespoke processing to rename all the profile thumbnails I'd created (usually by cropping a larger picture) for FTM/Ancestry to the treeid_personid.jpg format as described in https://www.tng.lythgoes.net/wiki/index.php/Media_-_Default_Images. After I was done, I did a backup in FTM then with some trepidation reflecting on your comments above the first sync between FTM and Ancestry. Pleased to report everything went very smoothly; new and renamed media files were visible in the correct places in Ancestry and no old files were re-downloaded :-), so it looks like the isues you experienced have been fixed.

After all this I worked up some MySQL queries to set the "Living" flag for a bunch of cousins & others whom I knew were still alive but for whom I had no birth dates, because I wanted to protect their privacy in TNG in a similar way to that provided in Ancestry. But that's a different subject...

Whew. What a process. I'll document the whole lot from go to whoa when I have time (and before I forget steps) and put it all on the Wiki.

Thanks again for your input.

Cheers

Ian

Share this post


Link to post
Share on other sites
Robin Richmond

Great work and good news, Ian.

BTW, after a Gedcom Import, I run the "Refresh Living" Secondary Process, which sets everyone without a birthdate and deathdate to "Living", even if the person has parents, siblings, or marriages from the 17th century.  So I try to set a death date of "unknown" for everyone whom I know is not living. 

The more standard way to mark an unknown death date is to enter 'Y'.  TNG used to display a death date of 'Y' as "Yes, date unknown" (or something like that), which was too verbose for me.  I checked just now and saw just "Yes", but I don't know whether TNG works that way now, or if a mod is changing the display value.

- Robin

 

Share this post


Link to post
Share on other sites
Ken Roy
1 hour ago, Robin Richmond said:

I run the "Refresh Living" Secondary Process, which sets everyone without a birthdate and deathdate to "Living

Robin,

The Refresh Living Secondary Process used the rules you defined in the Admin > Setup > Import Settings.  People without birthdate and deathdate on my system are set to deceased.  I use the Living Flag Backup / Restore mod to preserve manually set Living flags when I do not have birth dates for currently living people.  I maintain about 2,000 living individuals with manually set flags.  I did not want to deal with living people in the 1600  and 1700s

Share this post


Link to post
Share on other sites
Ian Davidson

Thanks Robin & Ken,

I too don't want to have to go back to people from hundreds of years ago to set their death dates to "Unknown", so I have TNG's "no birth date"  option in Import Settings set to "Person is deceased". That's why I wanted to set the Living flags with some educated guess queries, erring on the side of caution. I've looked at the Living Flag Backup/Restore mod and will probably use it, but I haven't decided yet; as the years roll on and if I'm not in touch with distant cousins etc manually-set flags will naturally become wrong. I'll probably end up using a combination of both :-).

Cheers

Ian

Share this post


Link to post
Share on other sites
Robin Richmond

Thanks guys.  Boy, TNG has so many configurations settings that I can't keep up.  A few years ago, I worked on a mod that would present an index of TNG system variables, by displaying the labels and variable names from the Admin>>Setup programs. But it got too complicated, and I dropped it

You both have me thinking about changing the way I deal with people who are known to be on the wrong sided of the living flag.  But (by chance) my scheme has the advantage that people who might be living are likely to be marked that way, so I'm erring on the side of not exposing data for people who might be living.  Conversely, I think that both of your methods would do the opposite - expose the data of some people who are living. (Just thinkin')

- Robin

Share this post


Link to post
Share on other sites
Ian Davidson
2 minutes ago, Robin Richmond said:

Conversely, I think that both of your methods would do the opposite - expose the data of some people who are living. (Just thinkin')

Yep, that's the danger Robin. I've tried to be conservative, but your settings are most likely better. If only FTM had a batch update facility, or an open database. Or even a faster & more user-friendly way of telling it someone's deceased than bringing up their record, adding a fact & typing "unk" into the date field, then waiting for FTM to do its bit before moving on to the next one of a few thousand... [sigh].

Also yep, it's a double-edged thing with TNG isn't it - the price of flexibility sometimes is complexity...

Cheers

Ian

Share this post


Link to post
Share on other sites
Ken Roy
3 hours ago, Ian Davidson said:

I've looked at the Living Flag Backup/Restore mod and will probably use it, but I haven't decided yet; as the years roll on and if I'm not in touch with distant cousins etc manually-set flags will naturally become wrong.

On a restore, the Living Flag Backup / Restore mod display records that now have a death date and allow you to uncheck the Living flag.   I also use the Living Flag Display mod which shows users who have the living flag set on the Descendant Text chart.  The big reason I have so many manually set Living Flags is that I compiled a genealogy book for a large family reunion as part of the Acadian Festival in Madawaska, Maine in 1994 and most people who contributed their pedigree and family group sheets for the reunion did not provide information for living people

Share this post


Link to post
Share on other sites
Ian Davidson

Ahh, good to know all that thanks Ken

Share this post


Link to post
Share on other sites
Robin Richmond
14 hours ago, Ian Davidson said:

the price of flexibility sometimes is complexity

Boy, that is sure true.  My mods tend to be kind of complex in the first place (is that an understatement, Ken?), and they get crazy-complex when I add run-time options to try to make them flexible at one level, and add mod options to make them flexible at another. And that complexity applies both to the code and to what what users face when then install and use  my mods.  I just don't seem to learn from experience. 

*** Digression Warning ***

I've been out of the academic computer science world for a long time now, but back in the day, software complexity metrics were all the rage. Actually, I should really know better than to add so much flexibility/complexity to my mods because my my dissertation was on that very subject - it was titled "Maintainability Metrics for MUMPS Programs".  It would be cool if there was a way to measure the complexity of PHP programs, but there's no mod for that, and though computing a program's complexity is not at all complex, it's not trivial, and I'm not going to write one. (Or will I??  Good grief, I hope not.).

*** ALERT:  Deeper Digression ***

MUMPS is an acronym for "Massachusetts General Hospital Utility Multi-Programming System", an ANSI-standard language that which was written by a team of clinicians at MGH in the mid 1960's, and is of the most fascinating languages I've ever used.  (It rivals APL, which was an extremely concise matrix-processing language whose operators were all single Greek letters.  Yep, we generally used keyboard with Greek letters to write APL programs).  Anyway, MUMPS was one of the first languages - maybe the first non-academic language - with a multi-user run-time environment, and has a built-in B-tree database. That is, the database is built directly into the language. Every variable is an un-typed  hierarchical array, almost exactly like PHP arrays, and if you prefix a variable name with a caret, the array is stored in the database as the equivalent of a relational database table.  Thus, you can read and write the equivalent of database records just by specifying the the table name as a variable name, and specifying the primary record keys as subscripts.  MUMPS, not officially called "M", is still widely used in the health care industry.  In fact, he market leader in electronic medical records applications is written in M.

*** End of Digressions ***

- Robin

p.s. If you're curious, see 
https://en.wikipedia.org/wiki/MUMPS  and 
https://en.wikipedia.org/wiki/APL_(programming_language)

Share this post


Link to post
Share on other sites
Ian Davidson

Hi Robin,

I enjoyed the digression :-) And I always thought MUMPS was a disease... If so it seems you caught it haha... I started my programming career with Fortran, done a fair bit of PL/SQL but nowadays spend most of my time maintaining legacy applications in VB6 - still a very capable language that can more than hold its own against more modern ones. Also dabble in PHP but I don't have the time or inclination to get too serious about it.

Cheers

Ian

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×