Media Drives and Fragmentation

An Accurate Vision Research & Development Report

16 November 2004

Editor's Preface

This white paper was originally published in the December 15th edition of Accurate Vision's newsletter for videographers and video editors who are members of our Worldwide Legal Video Network. Visit us at http://www.accurate-vision.com.

It has been reprinted here because it contains information that will benefit anyone who uses a computer to store and view depositions or other types of digital media.

For your convenience, certain technical terms appear in blue text, which means that a definition is available for the term. Simply click on the blue text to access the definition.

Media Drives and Fragmentation

It is no secret that most NLE Systems are "touchy" — and I put that word in quotes because most readers of this newsletter will consider it a severe understatement.

NLEs like to live on dedicated machines. They demand the very latest and fastest processors, refuse to run without tons of memory and disk space, and won't work without pricey video cards and a wall full of LCD monitors. And of course they demand to live in dust-free, air-conditioned penthouse suites that have an uninterrupted supply of power.

After you've fed them all of the above, they turn around and discriminate against nearly everything regular people like to do on a computer — like multi-tasking, web browsing and sharing files on your LAN.

Without mentioning names, I even know of one editor who comes in to work early every morning, before anyone else shows up in the studio, so that he can have a polite one-on- one conversation with his NLE system. Reportedly they discuss the projects they will be working on that day!

All exaggerations aside, pampering our NLEs is a fact of life that we will all have to contend with to one degree or another, until NLE technology advances to a higher level of maturity. For that reason, part of my job description here at AVI is to remain on the cutting edge of any and all technology that has the potential to lessen the degree of pampering, maintenance and downtime that AVI production staff contends with. And I've just completed testing a very new technology that has paid big dividends in that regard.

Background

Interestingly enough, I was prompted to engage in this particular line of research when I read a reply that a tech support rep posted in a forum for video editors. The forum is hosted by one of the more prominent NLE manufacturers. The support rep was replying to a post from an editor who was plagued with an NLE system that insisted on crashing, despite all of the customary remedies and pampering. The solution offered by the rep was "defragment all of your system and media drives."

A few days later, the editor posted a reply about the remedy: "That worked like a charm. Haven't crashed since!"

Now, that scenario piqued my interest for several reasons.

First of all, it was completely out of character for an NLE manufacturer's support rep to be suggesting that someone should do anything with an NLE system, other than to try and edit with it.

Think about that for a moment and I'm sure you'll agree that, just about every time you've called an NLE support rep about a problem, you were asked, "Have you installed any other software on the system?" Or, "Have you installed any system utilities, service patches or security fixes on the system?"

There are a dozen other variations of those questions, all of which lead us to conclude that NLE systems will break if you merely look at them the wrong way. In fact, it has been proven by survey that most NLE system engineers and editors cringe at the thought of doing any type of preventative or routine maintenance on their editing machines. Per the same surveys, the general consensus is "only fix it when it breaks," and "install nothing on the computer, other than the raw operating system and the NLE."

That leads to the second aspect of the forum post that interested me — the subject of fragmentation itself. It made me begin to wonder just how my systems fared with regard to fragmentation, and whether or not regular defragging would improve the critical ratio of "time spent editing" vs. "time spent pampering".

To answer that question, I set up my lab and began a series of tests.

Each test machine was an HP xw8000 workstation with dual 3.06 processors, 8 gigs of memory, a 250 gig system drive, and a terabyte worth of media drive storage. The test systems were each outfitted with different professional-grade NLE systems. There were three phases to the tests.

Test Phase One

First we loaded up each test machine with source media and began editing, periodically checking disk fragmentation levels with the default defrag utility that ships with Windows XP Pro.

The object of this first test phase was to get the media drives up to a high level of fragmentation. For purposes of this test, we defined "high" as anything over and above an average of 5 fragments per media file. And that didn't take long to accomplish. The average time per machine was 3.4 hours of editing to obtain a high level of fragmentation on the media drives.

Test Phase Two

With the drives heavily fragmented, the next phase consisted of executing a battery of common editing tasks that we routinely perform in our digital studio.

We intentionally chose time-intensive tasks for this part of the test, including timeline editing with up to eight tracks of video and eight tracks of audio, rendering layered and nested special effects, and exporting edited sequences into various digital media formats such as QuickTime movies, Windows Media files, MPEG-1 files for Video CD, and MPEG-2 files for DVD.

In each case, we recorded the elapsed execution time for the task in relationship to the total file size of all media involved. For example, if we were rendering a color correction, we would record that it took 15 minutes to render 20 gigs of video media.

This phase told us how long common editing tasks took with highly fragmented media drives.

Test Phase Three

Phase three of the test was to execute each of the common editing tasks a second time, after we defragged the media drives, and to then compare the new elapsed execution times to those achieved in phase two.

First we tried using the default defragger that ships with XP Pro and found that it was not the right tool for digital media files because it was not designed to handle files and drives of that capacity. It simply took too long to do the amount of defragging that we needed to accomplish. So we obtained and installed the three most popular defrag utilities that are commercially available, all of which promote that they are "industrial strength" to one degree or another.

With three different utilities at our disposal, we decided to test them individually, so that we could compare the amount of time each utility had to run in order to defrag the media drives. Also, separate tests on each utility would tell us if any of them fell short on accomplishing a full defrag of the media drives.

Thus, the third and final phase of the test was to:

1. Defrag each media drive on each test system with the same defrag utility;
2. Make a record of how long the utility ran on each drive, and how effective it was at producing a fully defragmented media drive;
3. With the media files and drives defragmented, re-execute the same editing tasks that were executed when the drives were highly fragmented in phase 2;
4. Record the elapsed execution time for each task in relationship to the total file size of all media involved;
5. Edit and render until the media drives were highly fragmented again; and
6. Repeat the above sequence using the 2nd defrag utility, and then repeat it again with the 3rd defrag utility.

Lab Results

The following table shows before-and-after results from five of the most time-consuming studio tasks that were used in the three-phase test.

*Elapsed times appear as hh:mm:ss (for example, 01:02:03 equals 1 hour, 2 minutes and 3 seconds).

As you can see above, the numbers we obtained from our lab tests were impressive.

When we edited with defragmented media drives, elapsed execution times for common tasks were reduced by an average of 12.1%.

If these numbers could hold up in actual practice, they meant we had the potential of reclaiming just over 7 minutes per hour, per day, per editor.

That may not seem significant at first glance, but when you take a minute to add it up, it proves to be very significant.

For example, the test results seemed to indicate that a single editor working one eight-hour shift was potentially losing 57.5 minutes of production each day because of fragmented media drives. That is nearly an hour of production each day. This adds up to 4.8 hrs of lost production at the end of a five-day work week — which is more than a half-day of work lost to fragmentation.

Stated in the positive, the above example shows that an editor should gain 4.8 hours of additional editing time during every 40-hour work week when working with defragged media drives.

So we now knew what the lab tests were telling us.

Would these lab numbers hold up in actual practice, in the studio, under the stress of rigorous daily operations?

That was the next question we had to answer.

Practical Results

I'm sure you're familiar with the popular disclaimer that usually accompanies testimonials and test-result reports — Individual results may vary...

In my opinion, it is a very fair and accurate disclaimer, particularly when test results were generated in a lab environment, because it is not always cost-effective for a test lab to be set up in a way that takes every possible variable into account.

With that in mind, I fully expected that the results obtained by defragging in the active studio environment would differ from those we had achieved in our lab. And they did — but with a few very interesting twists, as you will see.

When we implemented media-drive defrag on all systems in the fully operational studio environment, it initially appeared that the test results were holding up under intense real- world editing conditions. On average, editor productivity immediately jumped up by 11.4% when compared to the quantity of finished media that the same editors produced before defrag was implemented. But then we encountered the first of two important variables that we had failed to take into account in the lab, which was that top-end NLE software refuses to multi-task. That variable meant we could not edit and defrag at the same time without encountering disruptive and time-consuming protest errors from the NLE software. So we had to cease editing on a system when defrag was needed.

That first overlooked variable brought our real-world results down to an average of 3.6% improved productivity per editor. While not quite as substantial as the 12.1% average we had seen in the lab, it was a definite improvement — so we stuck with it. We took the NLE software off-line when we needed to defrag, and then went back to editing after the defrag utility had finished its work.

This went on for a week before we discovered that we had omitted a second important variable during the lab tests. Fortunately, this second missed variable played in our favor.

After one week of daily studio operation consisting of two eight-hour shifts per day, I examined the studio's system maintenance logs. The logs showed that, even with time off to run defrags, system downtime across the entire studio was 7% lower than it had ever been in any week since the studio first went online.

Closer examination of the logs showed that this increase in studio uptime was the result of fewer fatal NLE errors, fewer system-level errors, less need for system reboot, and fewer incidents where media files became corrupted.

So the bottom line was that we had fewer disruptions during editing, and less need to pamper the systems.

Coupled with the 3.6% average improvement in productivity, this increased uptime brought us back up to the level of benefit we had realized in the lab tests.

The Defragger Shootout

In the first draft of this report that I submitted, I named the utility that won the defragger shootout, and I listed specific reasons why it won. I also named both of the utilities that bit the bullet, and explained why they failed to meet the standards needed for digital-media production and post production.

As a techie, I love getting into heated discussions about the merits and shortcomings of computer software, and I was fully prepared to be challenged by the losing software manufacturers in that regard.

Our Department of Corporate Affairs did not share my enthusiasm for engaging in battle, however, and politely pointed out that AVI is not a non-profit consumer affairs organization, that we really don't need such distractions, and so on and so forth.

After considerable wrangling and hand-waving on the matter, I agreed to edit out the trademarked names of the losers — provided that I could name the winning utility and tell you why it won.

The hands-down winner of the defragger shootout was Diskeeper® 8.0, Server Enterprise Edition, manufactured by Executive Software International®, www.executive.com.

The first and most important reason why Diskeeper out-performed the competitors is written right on the front cover of the shrink-wrapped product:

The "new breakthrough" is a Diskeeper exclusive, aptly named their Terabyte Volume Engine (TVE).

In our tests, the TVE proved to be the only defrag engine capable of defragging digital media drives in an acceptable period of time. The other two utilities we tested literally took forever to accomplish what Diskeeper could do in a matter of minutes.

The second reason why Diskeeper was our defragger of choice is a feature known as Selectable Run Priority.

To a certain degree, the Selectable Run Priority feature enabled us to defeat our NLE software's refusal to multi-task with the defrag utility. It did not defeat that insistence in all cases, but it won the shootout because the other two utilities were incapable of defeating it at all. If you as much as tried to use them while the NLE was online, they would crash the NLE every time, and in most cases crash the OS as well.

The last (but not least) reason why Diskeeper won the shootout was its ability to provide us with important information about the performance and reliability of our media drives.

The feature is called Performance and Reliability Analysis. By using it, we were able to circumvent imminent data loss on several occasions, and avoid having to recapture valuable project footage because of file corruption and drive failure.

You can get additional information about Diskeeper and buy the product online at www.executive.com/buydk.

Summary

Disk fragmentation is a process whereby files are constantly being broken into multiple pieces, so that they will fit into available spaces on a drive.

From the tests we conducted as described in this report, we are convinced that drive fragmentation is one of the major culprits that impede the performance, stability and productivity of NLE systems.

Fragmentation appears to be similar to dirt on a VTR head: the more it builds up, the more problems you encounter. As fragmentation accumulates, your system slows down and becomes more prone to crashes and freeze-ups.

We have solved that problem at AVI by defragmenting our media and system drives on a regular basis. Chances are good that you can do the same.

To accomplish that, you will need a utility that can cope with large drive volumes.

And now you know which one to buy!