A poor SP1 comparison (updated)

As I read up on the latest TechMeme news this morning I saw a headline that didn’t seem quite right, “Vista SP1 a Performance Dud” it said.  Intrigued, I clicked on the link and discovered an entire blog devoted to rather poor performance testing and entirely unhelpful rants.

The author of the SP1 comparison claims to have run the SP1 “RC1 Preview” build on a virtual machine to compare its performance in various tasks against that of an unpatched RTM system.  I assume by “unpatched” he means no updates at all – as there are several “pre-SP1” hotfixes out there that will undoubtedly make it more difficult to quantify the degree of difference between Vista and Vista SP1.

So what were the various tasks he compared?  Apparently, he used two scripted productivity benchmarks I’ve never heard of.  One drives Office 2007 through various tasks, and the other is supposed to measure multitasking performance by exercising “ADO, MAPI, and WMP.”

The author then notices a 1-2% performance improvement, and goes on to say that Vista SP1 is a “performance dud.”

The problems

1. Virtualization.  Running performance tests in a virtual machine is good for one thing:  Measuring performance of the virtual machine.  It is far from an accurate representation of what users will encounter on real hardware. 
2. Coverage.  The most substantial performance gains in SP1 affect the areas most hindered under Vista RTM: games, file operations (copy / move / delete), standby/resume, and citizenship of background services.  It appears that none of these were tested in this comparison.
3. Perceived Performance.  While measured performance of file operations in Vista SP1 is significantly improved, even further gains exist in “perceived performance.”  This is mainly due to improvements to Vista’s file transfer dialogs, which now begin reporting data about transfer rate and time much sooner and more accurately.
4. Advertising.  The comparison includes advertising for the benchmark tool used.  Apparently it is developed by the same company publishing these results, and they want to rope you into signing up for some kind of membership.  This increases my skepticism level slightly.
5. Ranting.  The entire blog seems devoted to ranting about Microsoft, complaining about Vista and Office, and making sweeping conclusions from insufficient data.

It is my hope that we’ll see more accurate comparisons start to spring up that clearly demonstrate the sometimes substantial performance gains included in SP1.  Of course, SP1 isn’t going to turn a slow computer into a fast one.  But if you’re facing compatibility issues that are affecting your system’s performance – it very well may help.  In some areas, like gaming or copying/moving files via Explorer, everyone will notice the improvements.

Update: Apparently, virtualization wasn’t used in this comparison, only the post the compared Vista to XP.  Still, results from that virtualized test are used as the basis for some of the conclusions of this article.