Modern Software Experience

2011-07-26

Out of memory

GenDetective

GenDetective Introduction explained what GenDetective is and how it works; once you've imported your GEDCOM file using GenDetective Analyzer first, you can create reports using GenDetective Reporter.

reports

Getting Started with GenDetective remarked upon some installation issues, and then discussed the many options you have to deal with when importing your data. The idea was for this third article to focus on what this product is really about; the reports that GenReporter generates, but that did not work out as planned.
This is not a review of GenDetective Reporter, this is a report explaining why there is none.

The Getting Started guide explains that GenDetective analyses your genealogy data to generate reports that highlight holes in your data and sources, and that it can create trip-based itineraries telling you what to research at the repositories you're going to visit.

GenDetective comes with more than hundred predefined reports. The Getting Started guide mentions that the Gold edition allows you to define your own reports. That is really a remark about future plans, as there is no Gold Edition yet. There is only one edition of GenDetective 1.0, and it does not allow you to define your own report.

computers in the repository

GenDetective is genealogy research and trip planning software; GenDetective finds holes in your research and then tells you which repositories may have the records you need. If you plan a trip to some repository, a GenDetective report can tell you what records you should look for once there. That's the idea that led to the creation of GenDetective.

RumbleSoft stresses how handy such a report is when you are not allowed to bring your laptop. I see their point - any research-supporting capabilities your genealogy application may have are useless if you cannot bring your computer inside.
I do wonder how many repositories still disallow computers. My experience in Dutch repositories is that you are not allowed to bring phones, but that your laptop or Palm device is quite welcome. They tend to limit the amount of paper you bring inside, which makes much more sense than disallowing laptops; they don't want to spend time checking what papers you carry outside.
Still, the idea of automatically generating the research plan for the day sounds appealing. Moreover, The GenDetective software generates Adobe PDF files and does offer reports for use together with your genealogy software, for when you are allowed to bring your computer in, as well.

using GenDetective

Public comments show that quite a few users have experienced installation problems, and RumbleSoft has confirmed that the installer isn't all it should be and is working to improve it. GenDetective exists to create reports using GenReporter, but to do so, you first need to import your GEDCOM using GenAnalyzer. GenAnalyzer has an import wizard, but its import is still so complex, even somewhat intimidating, that I dedicated a whole article to importing your GEDCOM into GenDetective already.
One of the choices you make when importing your GEDCOM is what family lines to focus on. I believe that choice should be made in GenReporter, just before generating a report. RumbleSoft has you make that decision in GenAnalyzer. Oh well, it is version 1.0. Surely I can still start GenAnalyzer again, quickly choose another family line, and then have GenReporter create reports for that line. No, it turns out you cannot do that either.

import again

Every time you start GenAnalyzer, it decided to re-import all your data, even if nothing changed. There is no way around it; when you start GenDetective Analyzer again, it immediately prompts you for a GEDCOM file.
GenAnalyzer does show a splash screen first, but it does not even show its application window, it just throws up an File Open dialog box. When I picked the sample file again, GenAnalyzer immediately reads and analysed it again. GenAnalyzer did not make me choose all the settings settings again, which is nice, but it decided to read and analyse the file again, without asking me whether I wanted to import to do that.

You do not want to wait hours for GenDetective to read and analyse your data every time you change your research a tiny bit.

That approach may be fine for a tiny sample file, that isn't fine at all when you have a large database. You want to be able to restart GenDetective and continue where you left off, perhaps merely change a few analysis choices, either because you want to focus on another branch, or because you've come to understand the choices better. You do not want to wait hours for GenDetective to read and analyse your data every time you change your research focus a bit. You certainly don't want wait it on when you are trying out different analysis choices.
I tried cancelling the File Open dialog box, but GenDetective does not like that at all. GenAnalyzer insisted that I must select a GEDCOM file. When you do not agree to open a file after this admonition, but decide to cancel the messagebox instead, GenAnalyzer terminates without any further message.
Discovering what the different analysis choices do by experimenting with the settings is practically impossible.

your own data

GenDetective comes with a tiny sample file. It is great that it comes with a sample file, and it is a good way to get a first impression of the product. However, the best way to find out how useful a product like this is to use it with your own data, so that it what tried.
My database currently contains more than a quarter million individuals. The most important part of that is sourced very well, less important parts are not sourced at all. So, it is not an ideal database, but it seems an ideal testing ground for GenDetective; GenDetective might help me decide what source to look for next.

out of memory

However, it is a large database, and many applications, especially 1.0 applications, are unable to handle large databases. So, I tried importing the infamous HundredThousand database first. I started the import and walked away. When I came back about a quarter of an hour later, the screen was empty; GenDetective had crashed without so much as a dialog box to tell me what the problem was. I did not need that dialog box to guess what had happened. The Windows XP machine I used has 4 GB of RAM, basically the maximum amount of RAM for a 32-bit Windows PC, yet GenDetective had run out of memory anyway.

fan value

I then decided to use the FAN files to figure out what GenDetective's limits are. On the Windows XP machine, GenDetective loads FAN16.GED in about seven minutes. GenDetective Analyzer's own import summary says it took 366 seconds (6m06s) to import the file, and 67 seconds to analyse, that is 433 seconds (7m13s) in total. That isn't very fast, but not the worst import speed in the world either. The worrying part is that, after importing this file of just about 12½ MB, GenDetective is already using 663 MB. That is not only more than half a gigabyte, that is also more than fifty times the size of the GEDCOM file.

I then tried to import FAN17.GED. The import seemed to be mostly done after about twelve minutes, but then the progress dialog box became not responding. I was ready to consider the import of FAN17.GED a failure, but kept it running, slowly increasing its memory usage as it did so. I naturally expected about the double the memory usage, and did notice a peak around 1250 MB. When I noticed that the import had completed successfully, GenDetective was still using 1166 MB.

The import of FAN17.GED had taken more than an hour. GenDetective itself claimed that it took 704 seconds to load the database, and about 132 seconds to analyse the database. The loading time of 704 seconds may be right, but the analysis time of 132 seconds is way off. For some reason, GenDetective does not measure the analysis time, but simply assumes an analysis speed of a thousand records per second. That is a rather optimistic estimate, as the actual analysis speed for FAN17.GED is closer to two thousand individuals per minute.
It is clear that GenDetective cannot handle FAN18.GED, and I'd hardly call the import of FAN17.GED a success, as the application was not responding for too long. GenDetective's fan value is 16. That is low.

speed

RumbleSoft believes that the analysis speed does not depend as much on the number of individuals, as it does on the number of events in the database. That makes sense for a product like this, but does not explain GenDetective's poor performance with the FAN17.GED file; the FAN files do not contain any events. They contains names and relationships only. In the GEDCOM file, these relationships are represented by marriage events without dates or places. That is one event for each two individuals but the proband. So, a processing speed of about two thousand individuals per minute equals a processing speed of about a thousand GEDCOM events per minute.

subset

I contemplated using a subset of my data; export a selection to GEDCOM and then import that file. I decided not to do so, for several reasons. First of all, I simply did not feel like doing it. As understandable as some of the questions that GenAnalyzer asks are, fact is that getting your data into GenDetective so that it can generate reports is too cumbersome already. I really did not feel like thinking about what selection to make, or having to deal with the realisation that I should have made another selection, and then do it all over again.
The second objection is more fundamental; when I choose a subset, I am leaving of data that GenDetective could have used to create better reports. Only a full import of my database allows me judge it reporting capabilities fairly. I tried that, and the unescapable judgement is that it is not up to the task.

conclusion

RumbleSoft is planning to allow users to define their own reports. That is a fine plan,they just have to fix several major issues first.
GenDetective is a new product in a new category, and that shows in many ways. Users have problem installing it, the import wizard is intimidating. Import of small files is fast enough, import of medium size file is bearable, import of large files fails. GenDetective's fan value is only 16.

GenDetective is split into GenAnalyzer and GenReporter; GenAnalyzer import and analyses, GenReporter creates reports. This split into two applications is a bit annoying, but a more fundamental issue is that the way the split has been made is less than optimal; having to re-import all your data when you merely want to change some analysis options is not user-friendly. That isn't a big issue for small databases such as the sample database, but it is a deal-breaker for large ones; you are not going to use a tool much if it is that cumbersome to use. GenDetective should have a smarter import wizard, one that asks less questions, and defers more decisions to reporting time, so that it does not make you import the same file over and over when you merely want to change some analysis options.

The biggest issue is GenDetective inability to handle large databases. GenDetective is the kind of tool you want for large databases, but it only works for small ones.
The idea behind GenDetective seems solid. I have certainly become curious what reports it will generate for my database once it works. RumbleSoft needs to rethink some technical decisions, to stop wasting memory like crazy and make it capable of handling large databases, the very databases that need GenDetective the most.
If RumbleSoft takes care of these issues, the next version might be a great tool.

updates

2012-05-08: GenDetective Mac support

RumbleSoft has added support for the GEDCOM dialects of various Mac Genealogy programs such as Family Tree Maker for Mac, MacFamilyTree, Reunion, and iFamily for Leopard.

product details

propertyvalue
productGenDetective
version1.0
companyRumbleSoft
websiteRumbleSoft
priceUS$ 32,95 (digital download)
US$ 37,95 (CD-ROM & download)
requirementWindows XP SP3
note 
VerdictOut of memory
RatingOut of memory

links