Migrations, I've known a few

By Derick Deleo

As I have mentioned before, I have been in the ECM/DAM space for decades, and I have been directly and indirectly involved in many migrations for and at many organizations. I do not think I would ever say something like “That migration was fun.” or “That migration was so smooth and uneventful.” because no migration is fun or uneventful. But migrations are necessary to leverage advancements in technology and reduce costs. Whether your organization has acquired another organization, or your organization allowed various internal entities to select their own technology, or a new IT initiative is mandating consolidation or embracing advanced systems, at some point in time you will be faced with a migration. You can cringe and moan and perhaps even twitch at the thought of performing one or more migrations to the new system and then either choose to face the challenge or go do something else.

But why is it that migrations are such anxiety causing activities? That’s easy; because no two migrations are ever the same and there are always issues encountered during the migration process. The older the system (and some have been in place for decades and are no longer officially supported without some kind of special or “extended” support agreement that costs considerably more than typical software support agreements) the more the chances of issues such as orphaned documents (documents with no metadata, probably because the document was ‘deleted’ but in reality, only the metadata, including the pointer to the document was deleted). Or the cringe-worthy occurrence of a corrupted file – those are never fun to deal with. You can always count on the fact that you will encounter challenges. I am actually working with a client now on a migration from a very old ECM system and we are facing several challenges, as I expected to.

And then there is the challenge in how metadata, system-level data, annotations, etc. are managed. Older systems typically had flat data models which made extracting metadata relatively easy; as long as you can access the data directly that is. Newer systems often have joins between tables for metadata and system-level data and pointers to documents and annotations. And the formats used to store annotations can be complex and proprietary which means you need to convert from one format to another, which introduces another level of complexity and risk (e.g. - will the annotations map to the same location on the document).

Many migration tools have been developed – just do a search on “ECM migration tools” and you will soon be lost in a plethora of tools, strategies, and best practices, all claiming to be the best. And yes, some of these tools work fine. But many of the tools available use the source and target system APIs to extract and then import from one system to the other. Now when you are looking at migrating small volumes, like less than 100 million, that may be OK. But when you need to migrate hundreds of millions or billions of documents this could be a very slow process. And let’s face it, you want to migrate to eliminate the cost of the original system as quickly as possible, because in the meantime you are paying for two systems.

The fastest possible approach to extracting metadata and the objects themselves (documents, images, videos, etc.) is by avoiding the source system APIs and going directly to the database for the metadata and the file storage system for the objects. This is how at Maretha we have been able to achieve migration rates of up to 150 million objects/day; and even that rate was throttled by the client. Now please do not email me or my teammates at Maretha and say “I have a billion objects in my ECM system and I want you to migrate them to my new system in one week; as I said earlier, NO TWO MIGRATIONS ARE THE SAME.

First of all, extraction is just one part of the migration process. Once you have all the metadata and the objects extracted, now the real fun begins. Objects may require conversions, perhaps from an old, no longer easily supported format to a new format, annotations most likely will need to be converted. And then what do you want to do about the metadata? Older systems tended to index much more than newer systems for many reasons. Business users often insisted that more was better, and therefore would index content on almost any imaginable value that might someday be used to search for content – typically when you analyze how end users search you will find that far fewer indexes could be used. And if your system was used by multiple lines of business each dictating how common indexes are defined and captured, you will find that there really is no commonality. So then you need to determine if you will normalize metadata across your organization – hopefully your organization has already addressed this in a master data management initiative. All of this converting and normalizing is when the really hard work begins.

I have not even mentioned the issue of content that should have been purged long ago but never was. No matter how many times I suggest that clients purge what they legally can before they tackle a migration, it often does not happen. Which is a shame really because you can save substantial time, which of course means money, by reducing the volume of content that you migrate. There are many reasons why this happens. Getting business users to agree to purge content is never easy. No one really wants to be the one who decides what can be purged. And sometimes regulations are so difficult to understand that it becomes easier to just say you will never purge content. Of course, for services companies this is not really a problem because you are often paying them based on the volume of content to migrate and how long it will take.

And then you finally get to import the content along with the metadata into your new system. And again, there are tools that can do this for you, and they will often use the new system APIs to do so. Which, you guessed correctly, is not always the fastest way to import content and metadata; at least if you were paying attention earlier you may have guessed this. Newer systems, such as Nuxeo, offer multiple options for bulk importing to make it possible to hit rates up to 150 million objects per day. But other systems may require that you use their APIs to import, which will be limited in many ways, including how the system handles mutli-threading. Some systems, including Nuxeo, will allow you to spin up worker nodes that are separate from end user nodes. These worker nodes can be dedicated to the import processes while end user nodes allow normal use to continue with minimal to no impact on performance. Other systems will require that your import processes may only be run outside of normal business operating hours. If your head is starting to hurt and you are starting to twitch just thinking about an upcoming migration, well, I would like to tell you don’t worry, be happy (and if that song is now stuck in your head, you are welcome), but I can’t really say that. But then, that is where companies like Maretha can help you to stop cringing and twitching, because they have been there and done that many times over, and can help make the import process go as smoothly as possible, while they cringe and twitch for you, LOL.

So what is the moral of this article? Well, one moral could be if you sense an ECM or DAM migration is approaching, run away, become a hermit in the mountains living off the land and reading lots of good books and sketching and painting (OK, maybe that’s just my dream). But the more realistic moral is to never tackle an ECM or DAM migration alone, unless you too have been there and done that many times over. Companies like Maretha have been performing ECM and DAM migrations for customers for years. We have been faced with issues and challenges and have worked on, improved on and repeatedly tested and fine tuned strategies for handling the challenges you will most certainly be faced with. Besides, a wise person once said “Never walk alone.” or maybe that was just someone who lived in New York City in the 70s.

← All insights