What Happens In Vegas…

Project proposals often rely on risk estimates in their valuations. Just how reliable are these estimates?

Las Vegas bets the house on knowing the odds. The probability of rolling seven at the craps table is 1 in 6; the probability of being dealt a royal flush is 1 in 649,740. In our jobs we deal with uncertainty and probabilities every day, which must make us experts at estimating probabilities, right?

GUESS AGAIN
With objective data we can calculate probabilities with great accuracy. But when we rely on our own experience and expertise, research clearly shows we are bad (actually very bad) at estimating probabilities.

There’s a simple quiz that demonstrates this. It’s ten questions such as “How old was Martin Luther King Jr. When he died?” Rather than guessing at the exact answer, participants provide a 90% confidence interval — a range for which they are 90% certain the correct answer falls within (leaving only a 10% chance the correct answer lies outside the range). So, for Martin Luther King Jr.’s age at death one might give a range of 45 to 55. If Martin Luther King Jr. was 49 at his death, they are correct. If he was 56, they are incorrect.

This is not a test of knowledge. It is actually a test for over-confidence… and the results are jaw-dropping. If we were good at understanding the limitations of our knowledge and translating those limits into probabilities we would give ourselves wide ranges and should score an average of 9 out of 10.

How do people usually do? Let’s look at managers, people regularly making decisions under uncertainty. Less than 2% of managers get 9 or 10 answers correct, a horrible track record. Why? Because we are overconfident in thinking we have a good handle on the topic. Added to that, our brain’s are terrible at contemplating what being 90% sure truly means. As a result we make our ranges far too narrow, increasing our error rate considerably.

An experienced Project Manager heard me talking about all this and exclaimed, “Oh! Do me!” At the end of the questions he admitted that, aware of my warnings, he had used much wider ranges than he otherwise would have. We then scored his answers and he still had only 6 correct!

One might think our scores improve when we move from general trivia questions to their area of expertise (e.g. IT, industry, medicine, etc.). Wrong again. That same overconfidence simply makes us narrow our ranges even further (because now we’re into the stuff we “know”) and we end up scoring about the same.

So, what are the implications for decision-makers? Be very skeptical of subjective probability estimates, even by experts. This is particularly true for projects highly vulnerable to uncertain outcomes (e.g. cost over-runs, operating under-performance, and other negative events).

The Good News
While our natural abilities to estimate probabilities are woefully inadequate, it is a skill we can develop with training, practice, and rigorous feedback. Unfortunately, few organizations invest the time and money to train their folk. After all, they’re already the experts… Right?

© 2011 Verax Point Consulting, LLC

P.S. If you’re curious about that test (even after all these warnings), you can take it here.

About Dave Wittenberg

I am all about equipping business teams to enjoy greater payoffs on their biggest bets. I bring a unique combination of evaluation expertise and behavioral insights developed over the course of more than 300 major strategic initiatives. Applying a highly effective, disciplined framework, teams unleash their full capabilities to generate better solutions, wring out hidden waste, and deliver superior returns. Specialties: Investment Evaluation, Decision Excellence, Strategic Analysis, Mergers and Acquisitions, Investor Relations, Leadership Development, Education & Training
This entry was posted in Biases & Traps, Risk Assessment, Valuations. Bookmark the permalink.

2 Responses to What Happens In Vegas…

  1. Pingback: Monte Carlo Traps | The CapEx Compass™

  2. Pingback: When Brains Are Not Enough | The CapEx Compass™

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s