Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the Web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 7486008, or online at http://www.wiley.com/go/permissions.
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 7622974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www .wiley.com.
Library of Congress Cataloging-in-Publication Data is Available:
Douglas Hubbard’s dedication: To my children, Evan, Madeleine, and Steven, as the continuing sources of inspiration in my life; and to my wife, Janet, for doing all the things that make it possible for me to have time to write a book, and for being the ultimate proofreader.
Richard Seiersen’s dedication: To all the ladies in my life: Helena, Kaela, Anika, and Brenna. Thank you for your love and support through the book and life. You make it fun
Doug and Richard would also like to dedicate this book to the military and law enforcement professionals who specialize in cybersecurity.
Foreword for the Second Edition
Jack Jones Chief Risk Scientist at RiskLens Chairman of the FAIR Institute
Iclearly recall my first conversation with Douglas about fifteen years ago. In the midst of trying to build a consulting practice around my Factor Analysis of Information Risk (FAIR) model, I had just read the first edition of his brilliant How to Measure Anything book and wanted to pick his brain. But what stood out most during our conversation wasn’t Doug’s incredible depth of knowledge—it was his passion for sharing insights with others. Similarly, when I first met Richard at an SIRA conference some years ago, he exhibited the same depth of knowledge and oozed the same passion. And although deep expertise is obviously important for their work, it’s their passion for helping others that provides the energy and intestinal fortitude to challenge conventional wisdom and lead our profession to higher ground.
In this book, Doug and Richard continue to apply their passion to the topic of reducing uncertainty and making (much) better decisions in a profoundly complex problem space. As a cybersecurity professional for over thirty-five years and a CISO for over 10 years, I can attest to how important this is.
Anyone who’s been in the cybersecurity trenches for any length of time will be familiar with some of the common measurement-related challenges we face. “Religious debates” about whether something is “high risk” or “medium risk,” an inability to effectively measure and communicate to stakeholders the dangers associated with changes in the risk landscape or the value of improved controls, even simply being confident and logically consistent in determining which problems deserve the most attention has been an elusive objective for many in the profession. This is also why I’ve focused so strongly on understanding and measuring cybersecurity risk for over 20 years, which led me to develop the FAIR and FAIR Controls Analytics (FAIR-CAM) models.
My own research also allows me to attest to the strengths of Doug and Rich’s methods. It is, unfortunately, incredibly easy to apply quantitative methods badly, which results not only in poorly informed decisions but also a false sense of security because you used numbers. This book does an outstanding job of laying the groundwork for defensible measurements.
That said, if you’re concerned about whether this book might be too “mathy,” rest assured that you do not need a background in calculus or statistics to find tremendous value here. Doug and Rich discuss the pitfalls of common approaches such as risk matrices and unaided intuition, dismantle prevailing misperceptions surrounding risk measurement, and describe the fundamental principles of good measurement in terms that anyone can understand and apply. They also provide very pragmatic methods and tools that anyone can use.
If you’re in the early stages of your cybersecurity career, this book will undoubtedly help you mature your understanding of the problem space far more quickly. Conversely, if you’ve been in the profession awhile, you will be able to better leverage your hard-won experience by looking at the cybersecurity landscape through a new and clearer lens. In either case your performance and value as a professional will increase.
The first edition of this book has been a key contributor to cybersecurity’s growing but still nascent transition from a qualitative, grossly oversimplified approach to risk measurement to one that is quantitatively based and logically defensible. If you already own the first edition, you’ll find that this second edition contains significant new and updated material. A few examples of the new material include:
■ Updated research on the impact of data breaches on stock prices;
■ New examples of how to make incremental reductions in uncertainty from just a few observations;
■ New research regarding how issues such as MFA, encryption, and user counts change risk;
■ New distributions for modeling risk and how to implement them in Excel or R;
■ How cybersecurity can/should integrate with enterprise risk management and decision analysis;
Consequently, upgrading to this edition should be a no-brainer.
The bottom line is that managing cybersecurity well requires being able to identify and focus on what matters most. This is true whether the scope of your responsibilities is strategic and enterprise- wide or more tactical. Regardless, these decisions are always based on comparisons, which are inevitably based on some form of measurement. And if your measurements
are poor, your ability to make good decisions is hamstrung. As a result, if I had my way, this book would be required reading for every cybersecurity professional. Short of that, if I was still a CISO, this book would be required reading for anyone in my organization. It’s simply that important.
Acknowledgments
We thank these people for their help as we wrote both editions of this book:
■ Jack Jones
■ Jack Freund
■ Wade Baker
■ Jim Lipkis
■ Thomas Lee
■ Christopher “Kip” Bohn
■ Scott Stransky
■ Tomas Girnius
■ Jay Jacobs
■ Sam Savage
■ Tony Cox
■ Michael Murray
■ Patrick Heim
■ Cheng-Ping Li
■ Michael Sardaryzadeh
■ Stuart McClure
■ Rick Rankin
■ Anton Mobley
■ Vinnie Liu
■ SIRA.org Team
■ Dan Geer
■ Robert Brown
Also, a special thanks to the rest of the team at Hubbard Decision Research for their help with analysis and proofing including
■ Elizabeth Swan
■ Peter Mallik
■ Philip Martin
■ Bobby Weant
■ Andrew Adams
■ Richie Modad
Preface
Whenwe wrote the first edition of this book, we were cautiously optimistic about how well it would do. Some constructive criticisms of popular methods were long past due. We knew a lot of published research showed that although the ubiquitous risk matrix provided a kind of placebo effect and increased confidence in decision making, it actually harmed the quality of decision making.
The book quickly exceeded our expectations in terms of demand and the influence it has had on the content of training and standards in cybersecurity. Consequently, the publisher, Wiley, wanted to capitalize on this demand with a second edition.
A lot has happened in cybersecurity in the six years since the first edition was published. There have been new major cyberattacks and new threats have appeared. Ransomware was not perceived to be nearly as much a threat in 2016 as it is now. But there will always be a new threat, and there will always be new technologies developed in an attempt to reduce these risks. Any book about cyber risk that only addresses current problems and technical solutions will need to be rewritten much more frequently than once every few years.
So, if the only changes were technical details of threats and solutions, then a new edition would not be required. We felt it was time to write a new edition because we realize that some organizations need quicker, simpler solutions to quantify cybersecurity risks. At the same time, others are ready for a little more depth in the methods. In this edition we have attempted to provide more for both audiences.
How to Measure Anything in Cybersecurity Risk
Introduction
Why We Chose This Topic
The first edition of this book was planned to be the first of a series of spinoffs from Douglas Hubbard’s successful first book, How to Measure Anything: Finding the Value of “Intangibles” in Business. For future books in this franchise, we were considering titles such as How to Measure Anything in Project Management or industry-specific books such as How to Measure Anything in Health Care. All we had to do was pick a good idea from a long list of possibilities.
Cybersecurity risk seemed like an ideal first book for this new series. It is extremely topical and filled with measurement challenges that may often seem impossible. We also believe it is an extremely important topic for personal reasons (as we are credit card users and have medical records, client data, intellectual property, and so on) as well as for the economy as a whole. The success of the first edition is why we chose to write another edition instead of exploring another How to Measure Anything topic outside of cybersecurity risk.
Another factor in choosing a topic was finding the right co-author. Because Doug Hubbard—a generalist in measurement methods—would not be a specialist in any of the particular potential spinoff topics, he planned to find a co-author who could write authoritatively on the topic. Hubbard was fortunate to find an enthusiastic volunteer in Richard Seiersen—someone with years of experience in the highest levels of cybersecurity management with some of the largest organizations.
So with a topical but difficult measurement subject, a broad and growing audience, and a good co-author, cybersecurity seemed like an ideal fit.
What Is This Book About?
Even though this book focuses on cybersecurity risk, it still has a lot in common with the original How to Measure Anything book, including:
■ Making better decisions when you are significantly uncertain about the present and future; and
■ Reducing that uncertainty even when data seems unavailable or the targets of measurement seem ambiguous and intangible.
This book in particular offers an alternative to a set of deeply rooted risk assessment methods now widely used in cybersecurity but that have no basis in the mathematics of risk or scientific method. We argue that these methods impede decisions about a subject of growing criticality. We also argue that methods based on real evidence of improving decisions are not only practical but already have been applied to a wide variety of equally difficult problems, including cybersecurity itself. We will show that we can start at a simple level and then evolve to whatever level is required while avoiding problems inherent to risk matrices and risk scores. So there is no reason not to adopt better methods immediately.
In this book, you should expect a gentle introduction to measurably better decision making—specifically, improvement in high-stakes decisions that have a lot of uncertainty and where, if you are wrong, your decisions could lead to catastrophe. We think security embodies all of these concerns.
We don’t expect our readers to be risk management experts or cybersecurity experts. The methods we apply to security can be applied to many other areas. Of course, we do hope it will make those who work in the field of cybersecurity better defenders and strategists. We also hope it will make the larger set of leaders more conscious of security risks in the process of becoming better decision makers.
If you really want to be sure this book is for you, here are the specific personas we are targeting:
■ You are a decision maker looking to improve—that is, measurably improve your high-stakes decision making.
■ You are a security professional looking to become more strategic in your fight against the bad guys.
■ You are neither of the above. Instead, you have an interest in understanding more about cybersecurity and/or risk management using readily accessible quantitative techniques.
■ If you are a hard-core quant, consider skipping the purely quant parts. If you are a hard-core hacker, consider skipping the purely security parts. That said, we will often have a novel perspective, or “epiphanies of the obvious,” on topics you already know well. Read as you see fit.
We Need More Than Technology
We need to lose less often in the fight against the bad guys. Or, at least, lose more gracefully and recover more quickly. Many feel that this requires better technology. We clamor for more innovation from our vendors in the security space even though breach frequency has not been reduced. To effectively battle security threats, we think there is something equally important as innovative technology, if not more important. We believe that “something” must include a better way to think quantitatively about risk.
We need decision makers who consistently make better choices through better analysis. We also need decision makers who know how to deftly handle uncertainty in the face of looming catastrophe. Parts of this solution are sometimes referred to with current trendy terms such as “predictive analytics,” but more broadly this includes all of decision science or decision analysis and even properly applied statistics.
In order to help decision makers with this task, we hope this book will explain why we should be skeptical of many current methods, how quantitative methods (even some very simple ones) improve the situation, and how to scale and evolve the solution.
