As I read up on the latest TechMeme news this morning I saw a headline that didn’t seem quite right, “Vista SP1 a Performance Dud” it said. Intrigued, I clicked on the link and discovered an entire blog devoted to rather poor performance testing and entirely unhelpful rants.
The author of the SP1 comparison claims to have run the SP1 “RC1 Preview” build on a virtual machine to compare its performance in various tasks against that of an unpatched RTM system. I assume by “unpatched” he means no updates at all - as there are several “pre-SP1″ hotfixes out there that will undoubtedly make it more difficult to quantify the degree of difference between Vista and Vista SP1.
So what were the various tasks he compared? Apparently, he used two scripted productivity benchmarks I’ve never heard of. One drives Office 2007 through various tasks, and the other is supposed to measure multitasking performance by exercising “ADO, MAPI, and WMP.”
The author then notices a 1-2% performance improvement, and goes on to say that Vista SP1 is a “performance dud.”
The problems
Virtualization. Running performance tests in a virtual machine is good for one thing: Measuring performance of the virtual machine. It is far from an accurate representation of what users will encounter on real hardware. - Coverage. The most substantial performance gains in SP1 affect the areas most hindered under Vista RTM: games, file operations (copy / move / delete), standby/resume, and citizenship of background services. It appears that none of these were tested in this comparison.
- Perceived Performance. While measured performance of file operations in Vista SP1 is significantly improved, even further gains exist in “perceived performance.” This is mainly due to improvements to Vista’s file transfer dialogs, which now begin reporting data about transfer rate and time much sooner and more accurately.
- Advertising. The comparison includes advertising for the benchmark tool used. Apparently it is developed by the same company publishing these results, and they want to rope you into signing up for some kind of membership. This increases my skepticism level slightly.
- Ranting. The entire blog seems devoted to ranting about Microsoft, complaining about Vista and Office, and making sweeping conclusions from insufficient data.
It is my hope that we’ll see more accurate comparisons start to spring up that clearly demonstrate the sometimes substantial performance gains included in SP1. Of course, SP1 isn’t going to turn a slow computer into a fast one. But if you’re facing compatibility issues that are affecting your system’s performance - it very well may help. In some areas, like gaming or copying/moving files via Explorer, everyone will notice the improvements.
Update: Apparently, virtualization wasn’t used in this comparison, only the post the compared Vista to XP. Still, results from that virtualized test are used as the basis for some of the conclusions of this article.
November 19th, 2007 at 2:39 pm
[...] A poor SP1 comparison As I read up on the latest TechMeme news this morning I saw a headline that didn’t seem quite right, “Vista SP1 a Performance Dud” it said. Intrigued, I clicked on the link and discovered an entire blog devoted to rather poor performance testing and entirely unhelpful rants. more Windows Vista [...]
November 20th, 2007 at 3:13 am
i think you should read things before making those false claims:
[quote]Research Staff said…
mog0,
I think you’re confusing the two articles. In the “What Intel Giveth” posting we indeed used VMware for some of the tests. However, the SP1 vs. RTM testing was all done on native hardware, specifically, a Dell XPS M1710 notebook with a 2GHz Core 2 Duo CPU and 1GB DDR-2 667MHz RAM.
RS[/quote]
November 21st, 2007 at 7:53 am
Sharktooth - as that quote shows, I wasn’t the only one to misunderstand the testing environment of original article. I have since updated my post to reflect this. However, that doesn’t change the other faulty aspects of the comparison.
November 22nd, 2007 at 11:47 am
I can’t stand microsoft products anymore.. In my opinion Microsoft Office is the only thing that they did well..
I got impressed by the new mac os x Leopard…And thats why I moved to apple’s OS..
You should do what this mac ad says: give up on vista!
weshow.com/us/p/24256/apples_ad_dont_give_up_on_vista
Apple FTW!
November 23rd, 2007 at 2:05 am
they used a framework that’s like a virtualization system
December 1st, 2007 at 9:00 pm
Games are listed under your “most substantial performance gains”… this is a good thing. That and most of the other things in there are the main reasons I’m not in Vista right now. My processor just can’t take both games and Vista at once.
I’m overdue for an upgrade in that department… it’ll be interesting to see whether the speed gap goes away or not with a new processor + Vista SP1 + XP SP3, or at least if I can get > 10 fps in Half-Life 2 under Vista.
December 19th, 2007 at 6:43 pm
So basically you’re posting a rant, ranting about somebody else’s posted rant. Clever you.
February 14th, 2008 at 6:18 am
I use Vista and think its great. Check ZDNet for SP1 comments.
I am very pleased with the speed and performance3 of Vista. Can’t wait for the upgrade.
Ron H.