How CVSS is DOSsing your patching policy (and wasting your money)

(U.S) Rule-based policies to mitigate software risk suggest to use the CVSS score to measure the individual vulnerability risk and act accordingly: a high CVSS score according to the \NVD\ (National (U.S.) Vulnerability Database) is therefore translated into a Yes. A key issue is whether such rule is economically sensible, in particular if reported vulnerabilities have been actually exploited in the wild, and whether the risk score do actually match the risk of exploitation. We compare the \NVD\ dataset with two additional datasets, the \EDB\ for the white market of vulnerabilities (e.g. those in Metasploit), and the \EKITS\ for the exploits traded in the black market. We benchmark them against Symantec's dataset (\SYM) of actual exploits in the wild. We analyze the whole spectrum of CVSS submetrics and use them to perform a case-controlled analysis of CVSS scores (similar to those used to link lung cancer and smoking) to test CVSS' reliability as a risk factor. We conclude that (a) fixing a vulnerability just because a high CVSS score was assigned to it only yields a negligible risk reduction; (b) the additional existence of proof of concept exploits (e.g. in \EDB) may yield some additional but not large risk reduction; (c) fixing in response to presence in black markets yields the equivalent risk reduction of wearing safety belt in cars. On the negative side, our study shows that, as an industry, we miss a metric with high specificity (i.e. that rules out vulnerabilities for which we should not worry).