Modern Software Experience

2010-01-22

Three Torture Tests

I recently published Three Torture Tests, three extreme GEDCOM files inspired by genealogical records such as most married, most children and longest name.

test filebrief description
Married1200.gedMuch Married Man
Children1200.gedMany Children
Long26CC.gedLong Names 26 - ConCatenated Lines
Long26LL.gedLong Names 26 - Long Lines

The above table summarises the three test. The fourth entry in the table is a bonus test, a Long Lines variant of Long26; the longest lines are so long that it is not a valid GEDCOM file.

character encodings

The tests below ignore character encoding issues. Several applications write nothing but so-called ANSI GEDCOM, which are not valid GEDCOM files. In the tests below this aspect is ignored.
So, this text may conclude that an application imports and then exports a GEDCOM file correctly, when in actual fact the application does not support real GEDCOM at all. Caveat emptor.

Tests

Daub Ages! 1.53

test fileresult
Married1200.gedImports fine, navigates fine, exports fine.
Children1200.gedImports fine, navigates fine, exports fine.
Long26CC.gedAges! does not recognise the CONC lines, but truncates the names instead.
Long26LL.gedAges! imports most names correctly, but without warning about the long lines, and on export of the long names, Ages! fails to use CONC and thus produces invalid GEDCOM files.

The maximum name length supported by Daub Ages! appears to be 243 characters; the names that are longer than that (starting with Y and Z) are truncated. There is no error or warning, and Daub Ages! does not produce an import log.

Daub Ages! fails the long name tests.

Ahnenblatt 2.62

test fileresult
Married1200.gedImports fine, navigates so-so, exports fine (export individuals in alphabetical order, but preserves marriage order).
Children1200.gedAppears to hang on import. Terminated after five minutes.
Long26CC.gedAhnenblatt complains about the CONC lines, complains and truncates the names.
Long26LL.gedAhnenblatt imports most names correctly, but without warning about the long lines, and on export of the long names, Ahnenblatt fails to use CONC and thus produces invalid GEDCOM files.

After importing Married1200.ged Ahnenblatt defaults to showing wife Eighty Four. The Next Spouse button below the couple is woefully inadequate for navigating to say wife Nine Hundred. Ahnenblatt has a Go to... menu item that presents a dialog box with a list to choose from, which is good enough if you know the name of the wife you want to navigate to. A problem with that dialog is that it is automatically sized to be just too small for the longest name.

Ahnenblatt completes the Married1200.ged test successfully, but fails all the other tests.

Aldfaer 4 (4.2.0.7)

test fileresult
Married1200.gedImport messes up database by inserting comments, navigation is very slow, export includes the comments.
Children1200.gedImport messes up database by inserting comments, navigation is okay, export includes the comments.
Long26CC.gedAldfaer imports the CONC lines correctly, and does use CONC upon export to avoid long lines.
The import messes up the database by inserting comments, and the export includes these comments.
Long26LL.gedAldfaer imports the long lines without warning, but does use CONC upon export to avoid long lines.
The import messes up the database by inserting comments, and the export includes these comments.

Although the SURN tag for the husband in Married1200.ged is an INDI.NAME.SURN tag to be precise, Aldfaer reports 22 GEDCOM-tag INDI::SURN wordt niet ondersteund (is not supported).
This reflects a defect in Aldfaer’s GEDCOM import: Aldfaer still does not support the GIVN and SURN tags on export, and still does not recognise the SURN tag on import. Instead, the Aldfaer GEDCOM import does the worst possible thing: it messes up your database by inserting the SURN line as a comment to the individual record.

Aldfaer does a few things right. Although Aldfaer is a dated design still based on an 8-bit code page, Aldfaer does support names longer than 256 characters. Aldfaer recognises the use of CONC for long names on import, and correctly uses it upon export.

However, Aldfaer also fails to issue warnings for long lines upon import, and messes up the database by inserting comments whenever it encounters a SURN tag.

Aldfaer fails all the tests.

Brother’s Keeper 6 (6.3.37)

test fileresult
Married1200.gedError: 6 Overflow during import. App seems to hang and needs to be terminated.
Children1200.gedBrother’s Keeper shows a messagebox: Gedcom line: 1 CHIL @I76@@ Family has over the limit of 75 children.
Long26CC.gedFamily Tree Builder does not process the CONC lines, does not issue any error or warning, but truncates the names instead.
Long26LL.gedBrother’s Keeper fails to complain about the long lines, but does report that it truncates names, after which none are more than 99 characters long.

Brother’s Keeper seems to have a hard-coded limit of 999 marriages that it does not test for, a hard-coded limit of 75 children that it does test for and a hard-code limit of 99 characters per name that it enforces by truncating names that are longer.

Brother’s Keeper fails all the tests.

Behold Beta (0.99.4)

test fileresult
Married1200.gedImports and mostly displays correctly; the table of contents is incomplete and lists only 92 marriages.
Children1200.gedImports and display correctly.
Long26CC.gedBehold does not process the CONC lines correctly and does not issue any error or warning either, but truncates the names.
Long26LL.gedBehold imports and displays the data correctly, but fails to complain about the long lines.

Behold passes the Children1200.ged tests, but fails all other tests.

Cognatio 1.4.3

test fileresult
Married1200.gedCognatio fails to import the marriages.
Children1200.gedImports fine, navigates fine, exports fine.
Long26CC.gedCognatio does not recognise the CONC lines, issues erroneous Misplaced tag error, and truncates the names.
Long26LL.gedWindows reports that Cognatio has stopped working.

Cognatio complains about the perfectly valid GEDCOM header by highlighting several tags as Misplaced tag. That does not build confidence.

Cognatio passes the Children1200.ged tests, but fails all the other tests.

Deudos 5 (5.1.274)

test fileresult
Married1200.gedDeudos imports and exports fine, but the navigation through the 1200 wives using but a forward and backward button is impractical.
Children1200.gedAgain, imports and exports fine, but awkwardly impractical navigation with nothing more than a see Other children button that shows the next few children.
Long26CC.gedDeudos does not process the GEDCOM correctly, and shows truncated names.
Long26LL.gedDeudos imports the names correctly, but without warning about the long lines, and on export of the long names, Deudos fails to use CONC and thus produces invalid GEDCOM files.

The GEDCOM files that Deudos writes do not have a Deudos GEDCOM header, but the same GEDCOM header as the test file Deudos imported. That is wrong.

Deudos does not really pass the Married1200.ged and Children1200.ged tests, and fails the long name tests. Because its GEDCOM headers are not correct, it fails all the tests.

Embla Family Treasures 8 (8.0.115)

test fileresult
Married1200.gedFamily Treasures imports and exports the data, but its navigation is not up to the task; it only shows wife One through Two Hundred and One.
Children1200.gedFamily Treasures imports and exports the data, but fails to show the children on the mother’s person view. The family view does show them all.
Long26CC.gedFamily Treasures does not process the CONC lines, does not issue any error or warning, but truncates the names instead.
Long26LL.gedFamily Treasures imports the names correctly, but without warning about the long lines, and on export of the long names, Family Treasures fails to use CONC and thus produces invalid GEDCOM files.

Embla Family Treasures GEDCOM import is remarkably slow. It took minutes for each file.

Embla Family Treasures fails every test already, but there is more. In September of last year, Embla Family Treasures 8 GEDCOM noted that Family Treasures does not even write a correct GEDCOM header. This test was done with the latest version, but Embla still has not fixed that major defect.

Embla Family Treasures doubly fails all the tests.

Family Tree Builder 4 (4.0.0.893)

test fileresult
Married1200.gedImports fine, navigates fine, exports fine. Charts are less than perfect, with lines missing, but does not crash.
Children1200.gedImports fine, navigates fine, exports fine. Charts are less than perfect, with overlapping children, but application does not crash.
Long26CC.gedFamily Tree Builder does not recognise the CONC lines, but truncates the names instead.
Long26LL.gedFamily Tree Builder imports the names correctly, but without warning about the long lines, and on export of the long names, FTB fails to use CONC and thus produces invalid GEDCOM files.

Family Tree Builder fails the long name tests.

Family Historian 4 (4.0.2)

test fileresult
Married1200.gedImports and exports fine. Navigation is a bit odd, but works fine. The All Relatives diagram is displayed quickly and works fine.
Children1200.gedImports fine, navigates fine, and exports fine. Diagram display quickly and works fine.
Long26CC.gedFamily Historian does not understand the CONC lines, logs erroneous complaints, and truncates the names. Because the exported GEDCOM file does include the CONC lines, it is accidentally correct.
Long26LL.gedFamily Historian imports the names correctly, but without warning about the long lines, and on export of the long names, Family Historian fails to use CONC and thus produces invalid GEDCOM files.

Family Historian fails the long name tests.

Family Tree Maker 16 (16.0.350)

test fileresult
Married1200.gedMaximum is 99 marriages. Additional marriages are ignored.
Children1200.gedMaximum is 99 children.
Long26CC.gedDoes not understand CONC used with NAME.
Long26LL.gedMaximum is 40 characters; names truncated.
Fails to complain about line length.

Family Tree Maker 16 does not crash and does produce clear import reports, but does not succeed importing any of the files.

That Family Tree Maker does not support names longer than 40 character did not hinder the Married1200.ged and Children1200.ged tests, as all the names in those files are shorter than that.

Family Tree Maker 16 fails all the tests.

Family Tree Maker 2010 (19.0.0.180)

test fileresult
Married1200.gedRemains Not Responding after import. Import log is empty. FTM 2010 can export to GEDCOM again, but it is practically impossible to edit the file. Trying to switch to the People view is enough to make FTM Not Responding again.
Children1200.gedImport proceeds without problems. Try to switch from Family tab to Person tab and FTM becomes Not Responding again. FTM uses more than 100 MB of RAM. FTM 2010 can export the data to GEDCOM again, but it is practically impossible to work with the file.
Long26CC.gedFTM imports the file without problems. It is possible to navigate and edit. FTM exports to GEDCOM, but exports long lines instead of using CONC.
Long26LL.gedFTM imports the file without problems. The import log does not contain errors or warnings for the long lines. It is possible to navigate and edit. FTM exports to GEDCOM, but exports long lines instead of using CONC.

Family Tree Maker starts slow, responds slow. Often becomes and remains Not Responding for minutes on end, necessitating termination from Windows Task Manager.

Family Tree Maker already fails all the tests, but on top of that, all the exported GEDCOM files have an invalid GEDCOM header.

Family Tree Maker 2010 doubly fails all the tests.

Genbox Family History 3 (3.7.1)

test fileresult
Married1200.gedImports fine, exports fine, navigates fine.
Children1200.gedImports fine, exports fine, navigates fine.
Long26CC.gedUpon starting the import, Windows reports that Genbox Family History has stopped working.
Long26LL.gedUpon starting the import, Windows reports that Genbox Family History has stopped working.

Genbox Family History fails the long name tests.

GENViewer Lite 1.15

test fileresult
Married1200.gedGENViewer Lite fails immediately with an Access Violation.
Children1200.gedWindows immediately reports that GENViewer Lite has stopped working.
Long26CC.gedGENViewer Lite import the file, but does not process the CONC lines, and instead truncates names without so much as a warning
Long26LL.gedGENViewer Lite fails to import the 26th name, the one that starts with Z.

GENViewer Lite does not produce errors, warnings or an import log.

The Reload last file viewed option has to be turned off to keep GENViewer Lite from continually producing exceptions.

GENViewer Lite fails all the tests.

Genealogica Grafica 1.12

test fileresult
Married1200.gedLoads fast, displays fine.
Children1200.gedLoads fast, displays fine.
Long26CC.gedGenealogica Grafica imports the file, but does not process the CONC lines, and instead truncates names without so much as a warning.
Long26LL.gedGenealogica Grafica fails to warn about the long lines and truncates the last name, the one that start with Z, to 255 characters without warning.

Genealogica Grafica does not handle either CONC lines or long lines correctly, and cannot handle names longer than 255 characters, but it does warn that the 26 individuals in the long name tests are not connected.

Genealogica Grafica fails the long name tests.

Geves 1.3.28

test fileresult
Married1200.gedImports the individuals, but not their marriages.
Children1200.gedSeems to hang when finishing up the GEDCOM import, but does finish. Becomes Not Responding when switching to Person tab. Navigation too slow for practical use.
Long26CC.gedGeves imports the file, but does not process the CONC lines, and instead truncates names without so much as a warning.
Long26LL.gedGeves imports the names correctly, but without warning about the long lines, and on export of the long names, Geves fails to use CONC and thus produces invalid GEDCOM files.

Geves already fails all the tests, but on top of that creates GEDCOM files that do not even have a valid GEDCOM header.

Geves doubly fails all the tests.

GRAMPS 3 (3.1.1-1)

test fileresult
Married1200.gedImports the data, but hangs on attempt to export it. Even after ten minutes, it remained stuck at 50%. GRAMPS had to be terminated from the Windows Task Manager.
Children1200.gedImports the data, but hangs on attempt to export it. Even after ten minutes, it remained stuck at 50%. GRAMPS had to be terminated from the Windows Task Manager.
Long26CC.gedGRAMPS imports and exports the data correctly.
Long26LL.gedGRAMP imports the names correctly, but fails to warn about the long lines. On export GRAMPS uses CONC to keep line length reasonable.

All tests where done using Portable GRAMPS, the only variant of GRAMPS with an easy Windows install.

GRAMPS passes the Long26CC.ged test, but fails all other tests.

Legacy Family Tree 7.0 (7.0.0.111)

test fileresult
Married1200.gedImport seems to hang, takes more than twelve minutes. The result seems fine and exports takes just a few seconds.
Children1200.gedLegacy’s maximum is 60 children. Legacy refuses to import the rest.
Long26CC.gedLegacy does not recognise the CONC lines, truncates the names and adds the CONC lines to notes.
Long26LL.gedLegacy imports the names correctly, but without warning about the long lines, and on export of the long names, Legacy fails to use CONC and thus produces invalid GEDCOM files.

Legacy 7 refuses to properly import the Children1200.ged files, and pauses the import while showing a messagebox saying You have reached the maximum of 60 children per family. All additional children will be ignored..

Legacy 7 passes the Married1200.ged test, but that success is overshadowed by the embarrassingly slow import speed. Legacy fails all the other tests.

Legacy Charting 7.0 (7.0.120.951)

test fileresult
Married1200.gedLegacy Charting fails immediately with an Access Violation.
Children1200.gedLegacy Charting imports the file just fine, but as soon as you try to draw a descendants chart, Windows reports that Legacy Charting has stopped working.
Long26CC.gedLegacy Charting does not recognise the CONC lines, but truncates the names instead.
Long26LL.gedLegacy Charting fails to import the 26th name, the one that starts with Z.

Legacy Charting does not produce errors, warnings or an import log.

Legacy Charting imports Long26CC.ged correctly, but fails all the other tests.

MyBlood 1 (1.1.0)

test fileresult
Married1200.gedImport and exports fine. Navigation to partners uses a large list, but it is not practical because it allows slow scroll only.
Children1200.gedImports fine, exports fine, navigates fine.
Long26CC.gedMyBlood does not recognise the CONC lines, but truncates the names instead.
Long26LL.gedMyBlood imports the names correctly, but without warning about the long lines, and on export of the long names, MyBlood uses CONC but still produces invalid GEDCOM files.

What MyBlood calls the People tab is really a Family tab. The lack of a real People tab and the absence of a Go To command make it hard to navigate between the disconnected individuals in the long names tests, until you remember to use Find dialog; the People search in its General Find is closest MyBlood comes to a real People tab.

MyBlood uses CONC lines on export, but does not recognise the CONC lines on import. Although MyBlood uses CONC lines on export to avoid long lines, the produced GEDCOM files are still invalid, because the lines are more than 255 characters long; 259 characters to be precise (257 plus Carriage Return and Line Feed).
It seems that Vertical Horizon tried to do the right thing, but forget about the line numbers and the line delimiters. The GEDCOM specification is clear that the total length of the line may not exceed 255 characters.
Most surprising is the lack of symmetry of surprising; MyBlood uses CONC on output, but still does not recognise it upon input. Thus, MyBlood cannot be relied upon to read GEDCOM that MyBlood itself wrote.

MyBlood barely passes the Married1200.ged test, and fails all the other tests.

Personal Ancestral File 5 (5.2.18.0)

test fileresult
Married1200.gedImports fine, navigates fine, exports a tad slow.
Children1200.gedImports fine, navigates fine, exports fine.
Long26CC.gedWindows reports that Personal Ancestral File has stopped working.
Long26LL.gedWindows reports that Personal Ancestral File has stopped working.

Personal Ancestral File has no problem dealing with many partners or many children, but fails to handle long names.

PAF fails the long name tests.

RootsMagic 4 (4.0.7.0)

test fileresult
Married1200.gedThe RootsMagic import crashes, and shows an Access Violation dialog box.
Children1200.gedThe RootsMagic import crashes, and shows an Access Violation dialog box.
Long26CC.gedRootsMagic does not recognise the CONC lines, but truncates the names instead.
Long26LL.gedRootsMagic imports the names correctly, but without warning about the long lines, and on export of the long names, RootsMagic fails to use CONC and thus produces invalid GEDCOM files.

RootsMagic 4 fails all the tests.

The Master Genealogist 7 (7.04.0000)

test fileresult
Married1200.gedTMG does not produce errors and does import all persons, but not their marriages.
Children1200.gedImport problems as detailed below, but TMG does import the file. Apart from general TMG GEDCOM quality issues, the export is fine.
Long26CC.gedTMG does not process the CONC lines correctly and truncates the names instead.
Long26LL.gedTMG does not warn about the long lines, but does fail to import the names correctly, truncating them to a maximum length of 246 characters. On export, TMG fails to use CONC and despite the truncation of the names still produces a longest line of 259 characters, which is invalid GEDCOM.

After import of Children1200.ged, TMG shows a messagebox with the text Menu has not been defined with DEFINE POPUP. 17 FRMMEMO.OLETER1.GOTFOCUS, and asks whether you want to Abort, Retry or Ignore. Retrying does not help, and you need to choose Ignore multiple times before TMG proceeds. TMG then shows an Abort-Retry-Ignore messagebox with the text Menu has not been defined with DEFINE POPUP. 169 TMGAPPOBJ.MREDEFINEMENUEDIT and you once again need to choose Ignore many times. After that, TMG repeats the same behaviour with the same two messageboxes again.
Well, I would not be surprised if the actual error messages varied, but TMG is infamous for its frequent FoxPro run-time errors like these.

The GEDCOM files TMG produces do not include GIVN and SURN tags, but do include illegal trailing white space on DATE and NAME lines.
On import of Long26CC.ged, TMG does not complain about the INDI.NAME.CONC tag, but does complain about every INDI.NAME.SURN.CONC tag. That seems inconsistent and is perhaps best understood as limited support for the SURN tag.

On export, TMG defaults to a maximum GEDCOM line length of 80 characters. That is not wrong, but its tip message that the value should be between 80 and 246 is; there is no official minimum and the maximum is 255, not 246. More worrying than the misinformation is that TMG does not respect the value you choose; even if you leave the default of 80 characters untouched, TMG will still write lines longer than 255 characters.

The Master Genealogist 7 barely passes the Children1200.ged tests and fails all others tests.

conclusions

failure

Of all the applications tested, not one passes all the tests. Many fail all the tests. Embla Family Treasures and Family Tree Maker 2010 and doubly failed because the ostensible GEDCOM files they write do not even have a valid GEDCOM header. If character encoding issues were taken into account, Family Tree Maker 2010 would triple fail all the tests.

name length

Several applications fail to import the 26th name, because they have an name length limit of 255 or 256 characters and that last name, the one that starts with a Z, because it is the only name in the file that is longer than 256 characters.

The Master Genealogist truncates names at 246 characters, Daub Ages! supports a maximum name length of 243 characters and Family Tree Maker 16 is infamous for supporting no more than 40characters.

fixed limits

Various application have arbitrary fixed limits. Family Tree Maker 16 allows only 99 marriages and only 99 children. Brother’s Keeper allows only 75 children.

crashes

Several applications became slow, Not Responding or awkward to navigate. Several appeared to hang on import, and GRAMPS hang on export. A few applications crashed.

The applications that hang or crash may well do so because the vendors decided to use fixed limits but forgot to test against those limits.

long lines

Many applications write illegally long lines and do not bother to alert the user to erroneously long input either.

Few applications use the CONC tag when writing a GEDCOM file as they should and even fewer do support the CONC tag when reading a GEDCOM file. Not one application processed the Long26CC.ged file correctly.

updates

2010-02-07: Behold Beta (0.99.7)

In his blog post Version 0.99.7 Beta released, Louis Kessler claims that Behold version 0.99.7 passes all the tests.

test fileresult
Married1200.gedImports and displays correctly. All tables are complete now.
Children1200.gedImports and display correctly.
Long26CC.gedBehold imports and displays the data correctly, without error or warning.
Long26LL.gedBehold imports and displays the data correctly, and issues a message about the long lines. There is no log file.

Except for the fact that this version fails to produce a log file, Behold Beta 0.99.7 is the first genealogy viewer to pass all the tests. No genealogy editor has passed all the tests yet.

2010-02-26: Family ChArtist 1.0

test fileresult
Married1200.gedImports correctly (but no chart types shows wives).
Children1200.gedImports correctly (but no chart type shows children).
Long26CC.gedFamily ChArtist does not process the CONC lines correctly and truncates the names instead.
Long26LL.gedFamily ChArtist imports the names correctly, but without warning about the long lines.

Family ChArtist did not fail the first two tests, but did not pass them either; to pass, it must show the result on screen. Family Chartist failed the last two test.

2010-12-23: Family Tree Maker 2011

Family Tree Maker 2011 GEDCOM Torture Test Results provides tests results for Family Tree Maker 2011. 

2010-12-31: Sherwood Branches

test fileresult
Married1200.gedImports correctly, displays correctly.
Children1200.gedImports correctly, displays correctly.
Long26CC.gedBranches does not process the CONC lines correctly and truncates the names instead.
Long26LL.gedBranches imports the names correctly, but without warning about the long lines.

Branches passed the first two test. Branches failed both the concatenation and the long lines test.

links

reviews