pseudo-flaw.net

Jeff Jones has recently released a new paper comparing vulnerability counts for Windows Vista in the first year with the equivalent time frame for Windows XP. The results are that the Vista had fewer vulnerabilities in the first year than Windows XP did. Somehow that is not surprising given that XP was released prior to implementation of the Security Development Lifecyle (SDL).

In fact, if the vulnerability counts in Vista weren't significantly less, the SDL would have been declared an abysmal failure and the Microsoft security employees would have slinked off meekly into the night. But this didn't happen and that is a good thing. It lends credence to the idea that well structured security software engineering and development processes work in reducing the total number of vulnerabilities.

Of course, after Jones' previous stinker of a paper (which I discussed here), there was bound to be controversy. You can watch the piling on in the usual places (ZDNet or Slashdot). There are the usual arguments about counts, methodologies and rhyming apple with orange. Trolling and flamebait at its finest. Fanbois and zealots arise.

But all of this serves to cloud the real issue. The comparisons between Windows, the Linux versions and Mac OS X aren't getting to the core of the problem. They are simply a distraction. Most people don't stand there pondering which operating system to buy, because it may be more secure. The choice of operating system has already been made for them. For most people, it is going to be some OEM version of Windows.

So maybe a more appropriate question to pose is, if one has to purchase a computer to run Windows, should it be Windows XP SP2 or Vista? That is where the Jones paper fails to reach its full potential. Comparing an outdated, unsupported Windows XP release with Vista and at the same time comparing Vista with Linux and Mac OS X just confuses the issue of assigning some sort of "best security" mantle.

There is significant value to be found in Jones' paper if it is read dispassionately. He has promised a more interesting work that includes the Days-of-Risk (DoR) metric for the products. Personally, I am looking forward to it, because it should help clarify how much exposure an individual user had to a given vulnerability. Unfixed vulnerabilities (not undiscovered vulnerabilities) are the basis for most risk faced by users.

I hope the DoR metrics are enlightening, because a careful reading of the side-by-side comparisons showed that Ubuntu LTS (reduced) had the fewest number of unfixed vulnerabilities in the first year. I find that to be an intriguing discrepancy among the other conclusions of Windows Vista security superiority.