Security Metrics (book)/review

From Meritology

I'd really like to give this "3 ½" stars, but I rounded up to 4 stars. There is, currently, no book that is the "last word" on security metrics. The field is just not mature enough for that. However, this is certainly a very good and useful book for most people.

This book is for you if you are a practicing information security professional and you want to know the latest ideas about how to define, deploy, and use security metrics to improve security management. Written in an informal, personal style, Andrew's book reads like "letters from the front lines" (by analogy) than a treatise on military strategy. The informal style makes the reading, at times, both fun and funny.

He's up front about his preferences and biases, so you know where he's coming from. But he's not bombastic. If you disagree with him on some points (as I do), so be it. His writing invites open debate on the important issues. He's also generous in quoting and crediting various members of the security metrics communities that he participates in.

Andrew falls into the "bag-o-metrics" school of thought, as contrasted from the "risk modeling" school. (This is currently a raging debate within the community.) Basically, Andrew is pessimistic about the possibility of defining any models that integrate security metrics into an overall assessment of business risk. He's especially caustic in his comments about "asset valuation" and other related approaches. Given their current state of development, I don't blame him.

Given this philosophy, Andrew proposes a long list of operational security metrics, each of which measure something very specific (and quantitative), but don't necessarily aggregate. With enough of these "point metrics", some correlations may emerge, he reasons. To help give structure to the "bag", he offers some material relating metrics to various control frameworks (eg. COBIT) and also the Balanced Scorecard. The latter was a noble attempt to fit security metrics into enterprise performance management, but I don't think he really succeeded. But he chews on the right issues and questions, so it help people who are willing to do your own research into corporate scorecards.

Two of most notable parts of the book are the "Introduction: Escaping the Hamster Wheel of Pain", and "Chapter 7 Automating Metrics Calculations". The first is good because he talks in plain, blunt language about the current dismal state of security management in most organizations (who don't effectively use metrics to drive decisions). Ch. 7 is good because it gives a good framework and snapshot in time for a fast-emerging field.

I do have some criticisms of the book, even accepting Andrew's philosophies and premises.

First, there should have been an Appendix that compiled all the suggested metrics into one place.

Second, there is inadequate coverage of how security metrics can mesh with other security and risk management needs - privacy, digital rights, IP protection, forensics, fraud prevention, physical security, and business continuity, to name a few. InfoSec should not be an island. Therefore the metrics system needs to "play well with others", so to speak.

Third, there is inadequate attention to measuring knowledge of attackers and attack strategies. Basically, over the time scale of months or years, InfoSec is an evolutionary strategic "game" between attackers and defenders. This makes information security an arms race, so you need to know if you are falling behind, fighting the last war, etc. Every organization needs to constantly learn about potential attackers and new attack strategies. At least a few metrics in these areas would add big value.

Finally, there is a 49 page chapter devoted "Visualization", which I don't think is the best use of that space. While I think visualization and reporting are *critical*, I think there are plenty of other books and guides that provide guidance. I didn't see anything in this chapter that was specific to InfoSec. That said, the material is useful and valuable if you aren't skilled at visual design.

In conclusion -- a good book that has plenty of useful material, as reported from the front lines.