Modern Software Experience

2010-12-12

to import directly or not

direct import

A lot of vendors offer to ability to directly import your data from the database format used by other vendors. This seems attractive, because it is just one step (direct import) instead of two (export to some standard file format, then import from standard file format).
This makes a lot of sense for add-ons that only read the data; you'd like to use the add-on without having to export your data first. What few people realise is that it although it is handy for add-ons, that supporting direct import makes limited sense for full applications competing with each other. It seems counter-intuitive, but vendors that provide direct import from their competitors' database formats may be hurting themselves.

It seems counter-intuitive, but vendors that provide direct import from their competitors' database formats may be hurting themselves.

Fantastic, Amazing and Marvelous

Consider three applications, named Fantastic, Amazing and Marvelous. Judging by their descriptions, reviews and trial editions, all three applications are fantastic, amazing and marvelous, but there is a significant difference; they do not share the ability to import data from each other. Fantastic only supports import and export of some industry standard file format in addition to its own database format. Amazing supports its own database format, the standard file format and direct import from Fantastic. Marvelous supports its own database format, the standard file format and direct import from both Fantastic and Marvelous.
So, if these applications are otherwise just as good as each other, which application should you pick and why?

which application?

The correct answer is whichever application you want. After all, all three are fantastic, amazing and marvelous. All three support an industry standard file format that allows you to import data from other applications and export your data to other applications. The simple truth is that the direct import feature is irrelevant. You don't need it, so it does not matter.

Now, just for arguments sake, let's assume for a moment that the direct import feature does matter. Let's assume the direct import feature is of great importance, for example because it is important to transfer the data as fast as possible. Having to do just one direct import instead of an export followed by an import saves precious time. Which application should you pick and why?

choice

This time the best choice is Fantastic; if you use Fantastic, both Amazing and Marvelous will be able to import your database directly. Because it is the only product to support direct imports from both other products, Marvelous is the worst choice; If you opted for Marvelous, you'd always have to export and then import your data again to get it into either Fantastic or Amazing. To be the choice, Marvelous would not only have to offer direct import from those competing products, but direct export as well.

Fantastic is the best choice, not because it lacks direct import, but because both Amazing and Marvelous support direct import from Fantastic. Their support of Fantastic is a non-obvious feature of Fantastic that only becomes evident when you compare the direct import support of the three applications; Fantastic is the best-supported application. The other two applications support direct import from Fantastic, making Fantastic the best choice.

If every vendors support direct import from Fantastic, then its file format is a de facto industry standard.

vendors

By supporting direct import from competing applications, vendors are not only advertising the existence of their competitors, they are also acknowledging the importance of those competitors, and adding to their support. If all vendors supports direct import from Fantastic, then Fantastic is the best-supported application in the industry. If every vendors support direct import from Fantastic, then its file format is a de facto industry standard.

Supporting direct import from competing products is a nice idea. It is very convenient, and that convenience can be advertised. Yet, paradoxically, direct import is hardly a feature of the product it is part of. When the direct import works, it is a feature of the supported product. When direct import doesn't work, it's a misfeature of the product it is part of. And oh, whenever the competition updates their file format, the direct import doesn't work anymore…

resources

Creating the direct import feature requires valuable resources; good developers, time and money. Those same resources could be spend developing features of the product itself. Moreover, direct import isn't a feature you create once and then forget about. It needs to be kept up to date. Maintaining the direct import feature takes resources too, resource that could be spend improving features of the product itself.

You can develop direct import for a competitor, or a feature for your own product. You can improve the direct import for a competitor, or improve your own product. Put like that, the choice seems clear. Yet many vendors choose to provide direct import for their competitors.

reasons

There are several reasons why vendors choose to provide direct import rather then spend their resources improving their own product. Users may find exporting and importing too much of a bother, so vendors may need to provide the convenience of direct import to lure them away from competing products. Some vendors even believe that users find the two-step process to complex, which is a rather insulting view of their own user base.  Some vendors see a huge list of import formats as an important feature of their own product, and believe that having more import formats their product a better one.

A possible reason for having a direct impor feature is that the industry standard file format cannot transfer all data. It is the best reason, but also a very worrisome one; if the standard file format isn't good enough, then it should be updated. To overcome limitations of existing standards, vendors often extend the standard format with product-specific extensions, and other vendors could import all data by supporting these extensions on import.

import component

Some vendors invest many resources into developing code for directly importing for many competing products, at the cost of the quality of their own product. As a result, their own product may not compare well with their competitors anymore, but they now have another, and perhaps more lucrative business opportunity: license their import code to other developers. They just need to componentise their code; clearly separate from the rest with an Application Programming Interface (API), document that interface, and come up with a product name. If they can get all other developers to license it, and rely on the component to take care of importing other formats, they would make money no matter which product the user chooses to use.

Licensing a component isn't a license to pass the buck.

However, if you are one of the other developers, you may not want to license anything from a competitor until that competitor ceases to be a competitor; you'd license it from them if the import component and perhaps some other components were all they offered, but not as long as they also offer a directly competing product.
Moreover, licensing a ready-made component does not solve your worries. Licensing a component isn't a license to pass the buck. It does not discharge you from any of the responsibilities you have towards your users. Users do not care that you licensed a component, but they still want your application to work fine. If you decide to license a component, you skip the development of that component, but you cannot skip the quality assurance. You remain responsible for the quality, user-interface, speed and memory-usage of the import options you offer your user. The components you license have to meet the product specifications and need to be tested just as much as the code you develop yourself. Making sure the component is good enough may be so much work, that you'd rather roll your own direct import code.

complete

Generally, licensing a ready-made component only makes sense when that component is complete enough. If the import component does not support the file formats of major competitors, or does not support these formats completely, then it probably shouldn't pass your quality assurance, simply because it does not offer what you are looking for.
If you were to use an import component that only partially satisfies your requirements, you will have to develop support for the rest of the requirements yourself. Such a mixed approach may end up being more complex and resource-consuming than developing your own direct import code.

strategic

Even if a component works perfectly, you still may not want to license it. Using components may be a good idea, but a generic component for reading a bunch of different image file formats and a component that specifically supports your and your competitors' proprietary file formats aren't the same thing. If you consider direct import to be of strategic importance for your product, then you may want to have full control over it by developing it in-house.

advantages and disadvantages

There are advantages and disadvantages to licensing a ready-made import component and to doing direct import it yourself.

advantages of licensing

The advantage of licensing a ready-made component is that you do not have to develop it yourself. Figuring out file formats and writing import code for those formats is a significant effort. Benefitting from someone else's work in that area, and their continual updates frees up a considerable amount resources that can now be used for something else. You do still need to do quality assurance on the component and integrate it into your own product. You still need to make sure that the claims made for the component are true.

disadvantages of licensing

The obvious disadvantage of licensing is the cost, which is likely to be a per-copy cost. If the cost isn't an issue, the lack of competitive advantage is likely to be one; it is impossible to be much better than your competitors when they can license the same component as you're using.

Another disadvantage is that you have no control over how quickly defects are fixed or new file formats are supported. A less obvious disadvantage is that you are bound by the component's API and the limitations thereof. Moreover, although the import process seems a single step to the end-user, it is actually still taking two steps; conversion to the component's internal model, followed by conversion to your own database. Because using a component actually takes two steps, it is not really a direct import, it merely is a single user action, but perhaps that is good enough. Using an import compoent is neither as fast nor as memory-efficient as custom code could be, and having to funnel all the data through the component's API entails a certain amount of overhead, and may even be a performance bottleneck.

logging

A lot can succeed or fail during an import, whether done directly or through a component. That is why it is common practice to write an import log file during import that can be examined later to verify the import status. Using a ready-made import component is no reason to forgo logging, it is a reason to make doubly sure that every success and failure is logged, so that if anything goes wrong, you know what succeeded, where it went wrong and whether it is the component or your own code that is at fault. Thus, the component should report all significant events. The component could do so directly to its own log file, but would ideally do so through a callback API, so that you can integrate the information it provides into a single log file for the entire import process.

component source

There is risk inherent in licensing a component from a third party. A small company may go out of business. A large company may decide they are no longer interested in the niche product anymore. A components vendor may go out of business or stop updating their components for whatever reason, and you should be prepared to handle that. To address such worries, many component vendors offer a source license. Some even publish their components as Open Source Software.

Having the component source has additional benefits. One benefit of having the source is that it is easier to assess the component quality and identify, perhaps even resolve, any defects you run into. It allows making small enhancements, and then either sharing these with the vendor, or keeping these modifications as small competitive advantages. Perhaps most significant, it allows examining their support for your own file format, and addressing any issues of that support.

If a vendor does not offer the component source, you may want renegotiate the license fees and rethink your options. Having the source is important for generic components that you can buy from many component vendors, it is even more important for a specialist component that for which there are few or no alternatives.

advantage of doing it yourself

The advantage of developing direct import yourself is that it is your own product. You have full control over all aspects of its quality, and are not dependent on a third party to fix any defects that occur. When you develop it yourself, you gain knowledge of the file formats used by your competitors, which may lead to some valuable insights, not in the least about how to import their format efficiently. You are not bound by some third part API and its limitations, but can develop a direct import that fit your product perfectly.
You have full control over the user interface, error handling and logging.

disadvantages of doing it yourself

The disadvantage of doing it yourself is that it is a significant effort that takes valuable resources away from developing and maintaining the core product. You will have to figure out and document database formats, develop the import code, and then do all the usual quality assurance. You will have to do the ongoing maintenance as well; fix any defects and update the code whenever one of the supported products changes file formats.

add-ons

The advantages and disadvantages of the two approaches listed so far are for complete products. Users may switch from one product to another, but are unlikely to do so everyday, so neither the small inconvenience of having to take two steps, nor the time the process takes is likely to be very important. That is no reason to ignore the advantages of direct import. There are some users that use multiple competing products at once, frequently transferring their data between those. Moreover, for users switching to your application, the import experience is likely to be the first impression of your product.

For application add-ons, things are somewhat different. A proper add-ons does not compete with the application, but enhances it. However, add-ons do compete with each other, and an add-on that requires export and import will have a tough time competing with an add-on that imports data directly. There are other ways for add-ons to integrate with products, but add-ons that need to import the data should import that data directly and import it fast. An add-on vendor should, resources permitting, develop their own import code, to make sure it is as straightforward and fast as technically possible.

It is generally in a product vendor's interest to stimulate the creation of add-ons for their product, and to convince creators of add-ons for competing products to support their product as well. Many product vendors are willing to provide documentation of their database format to add-on vendors, to ease the creation of add-ons for their product, typically in exchange for nothing more than a non-disclosure & non-compete agreement.

Supporting vendor-specific extensions to a standard file format takes care of issues most users aren't aware of - and do not want to become aware of either.

direct import alternative

Import efficiency is one good reason to import a database directly. Another good reason may be that the industry standard file format does not support everything that's in the database, or at least not yet. However, vendors often address such limitations by adding product-specific extensions to the standard, and that practice provides an alternative to licensing or developing direct import.
The alternative to direct import of proprietary file formats is adding support for vendor-specific extensions to the code that imports the standard file format. This is not only is considerably less work than adding direct import, it is also a more stable approach. Vendors are not likely to drastically change their extensions to standards. When a vendor changes their file format, direct import code cannot handle it until it is updated to support the new format, but import of the standard file format - plus extensions - should continue to work just fine.

Supporting third-party extensions to a standard file format is a good idea for several reasons. The most important one is probably that it maintains the idea of standard file format, even when it isn't so standard anymore. Many users will only know that it is a standard format, and simply expect it to work. They may even blame your application for failing to import data hidden in vendor-specific extensions, because the application they exported it from imports it just fine.; it seems obvious to them that your application is the problem. Supporting vendor-specific extensions to a standard file format takes care of issues most users aren't aware of - and do not want to become aware of either.

When the standard is actively being maintained, vendor-specific extensions are likely to be replaced by an updated standard, thus allowing support for older extensions to be phased out. Perhaps more importantly, being able to import existing vendor-specific extensions should imply already being able to handle the data it contains. When your application is already able to handle the data, updating it to support the next version of the standard shouldn't be easy; you should even be able to provide that support through a patch to users of the existing version of your product without having to update your own file format.

if you want to encourage third-party support for your own extensions to an industry standard file format, take the guess-work out by providing clear, public documentation for those extensions.

conclusion

This is a general overview of arguments pro and con various possible choices regarding direct import. There is no general conclusion, a practical choice depends on the product, the available resources, the industry, the competition and even such vagaries as corporate culture. Stil, I do have one simple piece of advice;  if you want to encourage third-party support for your own extensions to an industry standard file format, take the guess-work out by providing clear, public documentation for those extensions.