In today’s well connected world of electronics and instant communications I received a text from a friend asking if I had seen the recent PC World magazine (February, 2012).  He said it had some tidbit of information concerning one of my favorite subjects; system performance, defragmentation, and SSDs.  I located a copy here at the office and found the article. As I read the first line I realized the debate on the virtues of defragmentation especially on SSDs will be one that goes on indefinitely as no one really talks about the issue with supporting hard facts and numbers.  Most articles are rehashing ideas and opinions long since debunked.  They continue to surface because very few truly understand the intricacies of the Windows NTFS file system and that of the storage media, whether it is rotating magnetic hard disks or electronic solid state disks.

So let’s set the record straight… Fragmentation is exponentially more of a problem with today’s data explosion. Defragmenting once a week will still cause the user to experience slowdowns from the degradation effects and doesn’t address the issue when files are initially being written.  And yes, never do a traditional defrag on SSDs.

NTFS file and free space fragmentation happens far more frequently than you might guess.  It has the potential to happen as soon as you install the operating system.  It can happen when you install applications or system updates, access the internet, download and save photos, create e-mail, office documents, etc…  It is a normal occurrence and behavior of the computer system, but does have a negative effect on over all application and system performance.  As fragmentation happens the computer system and underlying storage is performing more work than necessary.  Each I/O request takes a measurable amount of time.  Even in SSD environments there is no such thing as an “instant” I/O request.  Any time an application requests to read or write data and that request is split into additional I/O requests it causes more work to be done.   This extra work causes a delay right at that very moment in time.  Whoever thought that defragmenting once a month or weekly was good enough, simply didn’t understand fragmentation.

Disk drives have gotten faster over the years, but so have CPUs.  In fact, the gap between the difference in speed between hard disks and CPU has actually widened.  This means that applications can get plenty of CPU cycles, but they are still starving to get the data from the storage.  What’s more, the amount of data that is being stored has increased dramatically.  Just think of all those digital photos taken and shared over the holidays.  Each photo use to be approximately 1MB in size, now they are exceeding 15MB per photo and some go way beyond that.  Video editing and rendering and storage of digital movies have also become quite popular and as a result applications are manipulating hundreds of Gigabytes of data.  With typical disk cluster sizes of 4k, a 15MB size file could potentially be fragmented into nearly 4,000 extents.  This means an extra 4,000 disk I/O requests are required to read or write the file.  No matter what type of storage, it will simply take longer to complete the operation.

Suppose I chose to do some editing of my family videos on Tuesday evening.  Even the built-in defragmentation tool in Windows 7 doesn’t do me much good because it isn’t schedule to run until Wednesday morning at 1:00am.  This also means that quite a bit of fragmentation has built up since the previous week when it last ran.  Maybe I’ll manually run it, but that can take quite a while and I’ve wasted time that I would have rather spent on my project.  Unfortunately, the Windows built-in defragmentation utility doesn’t prevent fragmentation so even after running it manually; I still will wind up with fragmentation and slow access speed of my newly created files. 

I’ve often thought about why Wednesday at 1:00am was chosen as the time to schedule defragmentation.  Why isn’t it scheduled all the time?   It is because there could be system resource conflicts that either interfere with getting the task done or the defragmentation process has difficulty throttling back under a variety of conditions.  Regardless, this wait a week to clean up fragmentation doesn’t really help me when I need it most.

As pointed out in the article, the built-in defragmenter does not have the technology advancement to properly deal with fragmentation and SSDs. The physical placement of data on an SSD doesn’t really matter like it does on regular magnetic HDDs.  With an SSD there is no rotational latency or seek time to contend with.  Many experts assume that fragmentation is no longer a problem, but the application data access speed isn’t just defined in those terms.  Each and every I/O request performed takes a measurable amount of time.  SSD’s are fast, but they are not instantaneous.  Windows NTFS file system does not behave any differently because the underlying storage is an SSD vs. HDD and therefore fragmentation still occurs.  Reducing the unnecessary I/O’s by preventing and eradicating the fragmentation reduces the number of I/O requests and as a result speeds up application data response time and improve the overall lifespan of the SSD.  In essence, this makes for more sequential I/O operations which is generally faster and outperforms random writes.

In addition, SSD’s require that old data be erased before new data is written over it, rather than just writing over the old information as with HDDs.  This doubles the wear and tear and can cause major issues with the speed performance and lifespan of the SSD.  Most SSD manufactures have very sophisticated wear-leveling technologies to help with this. The principle issue is write speed degradation due to free space fragmentation.  Small free spaces scattered across the SSD causes the NTFS file system to write a file in fragmented pieces to those small available free spaces.  This has the effect of causing more random I/O traffic that is slower than sequential operations.

I think I have clearly made my point….  The built-in defragmenter in Windows 7 is not a solution for neither the consumer/home user, nor the enterprise business user.  Data access speeds are far more critical in the business world where time is money.  In the enterprise environment there are generally many more files that are used by higher number of users that are accessing data across shared type of storage such as SAN.  Even virtual platforms benefit from the same points covered.  This opens the door and is the reason why robust solutions such as Diskeeper exist.  More data about Diskeeper and the superior technology it offers can be found at http://www.diskeeper.com.

One of the most significant features in Windows 2008R2 (for Hyper-V) is Cluster Shared Volumes (CSV) for virtual disks (vhd). This allows NTFS to behave similar to a clustered file system, addressing many limitations found in Hyper-V storage with the original release (Windows 2008).  

IntelliWrite is “DKRtWrt” (its code names in development stages was WriteRight and then later RightWrite -hence "RtWrt"). To see or load/unload filter drivers, use the Filter Manager Control Program (fltmc):

“I am the person who proposed Diskeeper a few years ago in our company because we had some people who were complaining about slow machines. Most of the times the problem was related to the hard disks that were non-stop reading/writing. I tried a few times the internal defragmenter; it helped in reducing the slowness of the machine but it was always for a short time. So I looked for a better product and found Diskeeper.  

I made contact with Diskeeper UK and we had the pleasure to deal with an employee who arranged for an evaluation version of Diskeeper and Diskeeper Administrator to test in our company. 

We have a high number of computers, 16000, which are now running smoothly. The IntelliWrite does a good job preventing fragmentation. The number of calls for slow machines have dropped but we never had real measurements about the performance of Diskeeper. I am very curious about Diskeeper 2011 and what it can bring more so than this version. Diskeeper works and it is a good product. The price is also very good." 

Marc Vanderhaegen, SNCB (Société nationale des Chemins de fer belges)

Desktop Management

Brussels, Belgium

Joe Marion is founder and Principal of Healthcare Integration Strategies, specializing in the integration of imaging technologies with the overall healthcare IT landscape. His blog (at Healthcare Informatics) covers challenges and opportunities specifically relevant to optimizing Healthcare IT initiatives.

Medical images are a significant percentage of the the world's storage requirements, and have been predicted to encompass an even greater percentage of future storage demand. In Joe's recent blog post he posed the question "Is Defragmentation a Boon to Healthcare IT Performance?"

In his post he includes personal observations and insight into performance implications fragmentation can incur on IT as healthcare departments themselves consolidate and standardize application use:

"With departmental solutions, there very likely was less emphasis on system tools such as defragmentation applications.  Now that PACS technology is becoming more intertwined with the rest of IT, there should be greater emphasis on inclusion of these tools.  In addition, server virtualization can mean that previously independent applications are now part of a virtual server farm."

He also makes the astute observation that centralizing computing and storage magnifies bottlenecks, making a solution such as defragmentation increasingly more vital:

"The addition of disk-intensive applications such as speech recognition and imaging could potentially impact the overall performance of these applications.  As data storage requirements within healthcare grow, the problem will potentially get worse.  Think of the consequence of managing multiple 3000-slice CT studies and performing multiple 3D analyses.  As more advanced visualization applications go the client-server route, the performance of a central server doing the 3D processing could be significantly impacted."

You can read Joe's blog here.

Diskeeper 2010 was reviewed on Top Ten Reviews and won the Gold award.

Diskeeper offers a large list of powerful features for both general computer users and advanced experts. Diskeeper is a top performer, surpassing all the other disk defragmenter software reviewed as a result of its impressive feature set and very fast performance.

The video is also here.

We have been running Diskeeper 2010 EnterpriseServer for two months on an HP EVA SAN 4000 and 4400, with 4 1TB volumes each.  

Diskeeper removed over 45 million fragments in the last two months on a specific volume that had only 15% free space, and IntelliWrite prevented 24,000 fragments. I believe that will be even better as soon as we can extend this volume to two TB. 

We see a big improvement on the backup time which came down from 48 hours to 32 now, and it’s still going down. 

I believe Diskeeper worth the price and I never had any trouble with software from Diskeeper Corporation, so that alone narrowed the field of choices. 

Jean-François Poirier
Technicien Telecommunication
Spectra Premium Industries Inc.

Sure I'd love to give a recommendation for Diskeeper.  We are running 2010 now with Diskeeper Administrator and so far I love the improvements. Many of our Lotus Domino servers were so busy that realtime defrag had trouble keeping up, but with the new IntelliWrite technology preventing most of the fragmentation in the first place, we finally have it under control.

The systems were running well prior to introducing Diskeeper, but we deployed the software during a project where we virtualized over 80% of our production servers.  Because of the shared storage aspect of our virtual platform I knew that fragmentation would become a serious issue if we didn't deal with it from day one.  The  Lotus Domino servers often had trouble keeping up with the fragmentation rate but most other servers did well with Diskeeper Server 2008.   

The new Diskeeper 2010 has our fragmentation in check across the boards and I couldn't be happier with it.

Amesbury, MA

http://www.muntersglobal.com

“We have been a Diskeeper customer for several years now and have not had any complaints. Installation is quick and easy with the admin console tool. The application does not have code bloat with each new version. I believe that the traditional hard drive in our environment has lasted longer since the drive heads have to do less work. I have 5 year+ old PCs with the original hard drives in them.  I have been very impressed with the new Intelliwrite prevention technology. I have seen 90% frag prevention on workstations and have a 380gb MS SQL database with 99% prevention. Our Domain Controllers and Fileservers range from 78% to 88%. This average has been done since we have installed the Diskeeper 2010 version in November of 2009.  Attached [below] is a screenshot of Diskeeper 2010 Server running MS Server 2003 R2 X64 with MS SQL. I hope to replace some aging servers with SSD drives and am looking forward to testing Diskeeper with HyperFast” .

Mike L., Information Technology Manager, Ohio Valley Community Credit Union

If you live in on the west coast (USA), you may have seen an ad campaign about happy cows coming from California. Given we're a CA-based software company we concur.

OK wait a minute. Why are we talking about cows? What does this have to do with defrag? Maybe you're thinking "Michael's talking bovine, he's had a few too many".

Well maybe yes, but... I do have a point.  

Copy on Write refers to a low level technology that looks at block data changes and then takes a particular action. Every time there is a write, a copy on write technology would seek to make a copy of just the changes - not the entire new file. That action may be part of inline data dedupe, may be a snapshot, etc. The basic point is that copy on write is used to isolate changes to data at a granular level rather than at a file level.

The consideration is that most copy on write technologies are unable to distinguish between changes to data or movement of pieces of data, due to a defragmentation job. In other words, if you run a defrag, copy on write may think there are actual data changes to files, and then take action to make block-change copies, run a dedupe, etc...

Holy copy on write you say!

So, it is possible that a defrag, any kind of defrag, causes a copy on write based technology to go into hyperactive mode. That may mean a CDP (continuous data protection)/snapshot program may be triggered in to keeping/taking far more copies that it needs to. It is basically fooled into thinking that a defrag represents actual changes to data, and that they need to process those changes. Now this is a false processing but, because most copy on write solutions function underneath the file system layer, they simply cannot differentiate that actions that take place at the file system are not changes to data, but rather data-movement-for-optimization reasons.

So, if we're talking about a copy on write solution that takes snapshots, we could see an increase in the amount of copies that a snapshot solution would have to take. That would make for a fairly fat copy on write (a lot of extra data storage demands created by not understanding defrag).

Copy on write based solutions that work at the file system level (NTFS), have the possibility to recognize changes due to defrag and differentiate them from changes due to new actual data to a file. One such copy on write solution is Microsft's VSS (Volume Shadow Copy Service). However, as noted, most copy on write technologies operate at storage layers that are abstracted from the local disk file system - they operate at levels underneath the file system. Again, that means defrag jobs in the file system will not be recognized by these solutions, and a defrag job of any kind, will cause unnecessary copies.

In comes IntelliWrite. Now, it makes sense to think that IntelliWrite was invented as a faster and, perhaps cooler, solution to fragmentation. But, the truth is that solving fragmentation at the source (when it is created) is vital to ensure compatibility with many of the copy on write solutions that may be implemented at virtualization host or SAN layer (i.e. underneath and hence, unaware of the file system).

So, in a nutshell, IntelliWrite was very much designed to ensure that defrag offered full compatibility with modern storage technologies. That, we feel, makes for happy copy on writes.

A cure for cancer it is not, but in the category of performance (defrag) software, IntelliWrite is quite the bold, unorthodox step forward. Like many things new and revolutionary it is to be questioned, critiqued and qualified.

When we created IntelliWrite we had to first question the old way of doing things. Same goes for you, the user. You also need to form your own opinion on whether a new approach to eliminating fragmentation is warranted and worthwhile.

Now of course new does not necesarrily mean better, but new can, when done right change the way of things... for the better.

Our customers are our best and favorites critics. We hope we have provided you value and have earned your approval.

IntelliWrite technology is, as the name would imply, smart. Not only is it a better solution to removing fragmentation (it prevents fragmentation), but it actually self-learns; i.e. it gets smarter over time. Or perhaps more simply as one customer, who per his company policy must remain anonymous :-(, said, "crazy smart".

IntelliWrite learns how the various different applications on your system write data, much like advanced heuristic anti-virus solutions that learn "patterns" and can then proactively block new, as yet unreported, malware. IntelliWrite adapts to the applications on your system and how they write, and thereby fragment your files. That adaptive learning makes IntelliWrite increasingly more effective on your computer.  

Another part of the technology is an awareness of its surroundings. IntelliWrite automatically detects conditions where it might affect system performance and backs off, thus providing the best balance for performance – improves performance when fragmentation can be prevented and does not impact free space fragmentation when it might matter (i.e. when the available space is very low). One such easy-to-demonstrate case is on volumes with less than 2GB free space; IntelliWrite disables itself.

We began the process of securing the intellectual property rights to this revolutionary innovation prior to the release. You don't have to be a crazy Irishman to know that if you're looking for "The only way to prevent fragmentation before it happens(tm)", you've come to the right place.