Support for GEDCOMs from FTM 2017

[georgicus]

WRT FTM, the desktop "gotchas" at http://tng.lythgoes.net/wiki/index.php?title=Desktop_gotchas is getting pretty long in the tooth.  It quits at FTM 2012, and I've already moved on to 2014, then 2014.1, now 2017.

Do the "gotchas" notes apply equally well to FTM 2017?  Is the GEDCOM converter mod useful for GEDCOMs created by FTM 2017?

I have been using FTM 2017 and have been pretty happy importing its GEDCOMs into TNG, but it does seem like I am missing a lot of media links.  But many work fine with no intervention.  On the other hand, I'm thrilled with TNG so some clean-up is fine by me.

 

[bhemph]

FTM 2014 does have a specific mention in there with regards to the media links not being exported in the GEDCOMs even though 2012 does export them.  I am not using FTM 2017, still on FTM 7, but a lot of the GEDCOM creation issues have been carried through from version to version.  The GEDCOM converter mod is helpful in fixing some of the non standard implementation of the GEDCOM standard to have a smoother import process and also if you had some data entry that you wanted to get standardized.  So in those regards it may still be somewhat helpful for FTM 2017 GEDCOMs.  I had to customize some things in the converter mod so that some of my data entry habits could show up as I want in TNG.  I have also tried coming up with a solution for a couple of the other gotchas, one of which (the ID number update as another digit is added to the ID) looks like may not exist in newer FTM versions.  The gotcha page I take as being a look out for these kinds of problems when importing your GEDCOM and see if you have these problems when you do import, so you can see if they affect you and you can determine the impact it creates for you to fix.  You may just be one of the first few FTM 2017 users to import, so the information about the unique differences just aren't well known yet and documented in the wiki especially if they match an older situation.

[georgicus]

Surely I can't be the only FTM 2017 victim -- er, user!

Are you actually changing the converter mod, or just playing with its settings?

 

[Scott B]

I've been feeling your pain with FTM 2017 and doing a lot of testing and digging through the source code of the Gedcom Converter to figure out what it does and may have a solution for you.  Not the best, but it works if you're comfortable with using a text editor and writing a macro or script (skip to the bottom if that's all you care about).

So, the crux of the problem is that FTM 2017 is including map coordinates with any PLAC (place) records that the Gedcom Converter isn't expecting.

2 PLAC Dublin, Delaware, Ohio, USA
3 MAP
4 LATI N40.1468
4 LONG W83.1439

Trying to process this file in the converter returns hundreds of these errors:

INDI=1
=
Line 29. Unrecognized level 0 segment type in ReadZeroLevelRecord: 3 MAP
INDI=2
=
Line 62. Unrecognized level 0 segment type in ReadZeroLevelRecord: 3 MAP
INDI=3
=
Line 110. Unrecognized level 0 segment type in ReadZeroLevelRecord: 3 MAP

It seems that when the converter hits one of these it stops processing the record and writes out whatever media has been found up to that point, which is why we're missing random media links all over the place.  I spent a day trying to modify the converter to ignore/skip MAP, LATI, and LONG, but I don't really know PHP very well and didn't have any luck.

<TL;DR>
So, the obvious solution I ended up with is to just edit your gedcom file and remove all MAP, LATI, and LONG lines.  I have a fairly small tree of 400 individuals and have 688 instances of the MAP record type so I used Notepad++ and created a macro that finds "3 MAP" and removes it and the 2 lines after it.  Then I just tell it to run the macro 688 times.  Processing the file with the Gedcom converter after that returns no errors and importing the converted file to TNG I have all of my media linked to the people as expected.

 

[georgicus]

Oh!  I have not run into that particular problem, but I don't store any lat/lon data in FTM.

I *have* been doing some of my own conversions on the GEDCOM before uploading to TNG (see my notes posted to this thread: Update or Transfer from FamilyTree Maker to TNG).  It had not occurred to me to try to make any edits in the PHP, although that would make a lot of sense, especially if it were picked up in the next version.

 

[slowjim]

I do not know a lot about gedcom files, so I need reassurance that I am doing something correctly.

I use FTM 2017 gedcoms to import updates into my tree. Like many, I use Notepad++ to edit @I???@ individual references and @F???@ references to match my tree where appropriate. I add media later manually because I like to use Elements 11 to change the file sizes of any images I add.  My question is about sources.

Since all references to sources, for example:

2 SOUR @S1@ 

are defined, example:

0 @S1@ SOUR
1 TITL Texas Birth Index, 1903-1997
1 AUTH Ancestry.com
1 PUBL Name: Ancestry.com Operations Inc; Location: Provo, UT, USA; Date: 2005;
1 REPO @R1@

near the end of the gedcom file. Isn't it alright to let them import without changes and then just clean up the duplicate references in my tree using the sources >> merge tool in my administration section? I have done this a couple of times without a problem so far. Am I inviting a problem by doing this?
 

[jayat1familytree]

I dont think so.

When I upload a new gedcom I do NOT empty the Sources, repositories or places tables.

TNG checks to see if an existing record is found and uses that if it finds one,

Only if it doesnt find a matching record will it create a new one.

So you shouldnt have duplicates unless your originals have a different spelling.

But, even if you do end up with some dupp, then use the merge to clear those up.

Hope that helps.

Jay

[georgicus]

I merge duplicate sources pretty often (using 11.1.1)  -- I haven't had a problem yet.

[Robin Richmond]

I wrote the Gedcom Convert mod, and I'll check things out and upload a new version soon.  I haven't been able to sync my Ancestry.com data with FTM for almost a year - even with FTM2017, so I haven't been trying to upload data recently.  But I have done an FTM2017 export and was able to convert it and upload it to a test system without an trouble.  Actually, when I do export a Gedcom from FTM2017, I do not get place Geocodes, and my "Check place authority when entering place names" option is checked - so I don't know why Scott B got geocodes and I didn't.  We may have to check with Mackiev.

Scott - I'm not sure whether you want the Latitude and Longitude to be imported or not; please let us know.  When the Gedcom Converter sees them and displays a warning message about them, it should pass them through without change, and the Gedcom Import code seems to handle the LONG and LATI tags (and to change coordinates preceded by S or W into negative values), so I'm guessing that you don't want them. But I'll try to determine whether those tags do work, and will probably add a Gedcom option that lets you suppress the LONG and LATI tags or not. 

I'll repost when I know more.

[Scott B]

Hi Robin, thanks for responding. I meant to send you a separate private message about the converter and time got away from me.

I actually do want the Lat and Lon for the Places since that is what puts the pins in the Event Map on the TNG site.  I suppose I could let TNG try to Geocode all the places, but figured since I'm also using the place authority in FTM2017 that I'd rather let it be the database of record.  So, what I've been doing is importing twice.  The first time I use the method I described above so the Converter fixes all the Media properly and import that.  Then I import a second time with all the Lat and Lon data still in the file which updates the Places.

Looking at it again, it doesn't appear it's actually the LONG and LATI tags that are the problem, it seems the error is caused by the "MAP" tag. So, maybe that is actually all I need to remove. I assumed they all went together but maybe not.  I'm not sure why my GEDCOM would have it while yours does not.  I don't see any settings in FTM2017 that would appear to control that.  When I export I just pick GEDCOM from the drop down list, leave all the checkboxes under "Include" checked, and pick "Other" as the Destination.

If it will help I can export a small section of my tree that has both map data and media and send you the file so you can see how it stops processing the media when it hits "MAP".