Real life probability quiz

Mark Bowden adapted a piece of his upcoming book about the assassination of Osama Bin Laden for Vanity Fair. It's a crackling read.

What's most fascinating is the insight into how Obama finally made the decision to move on the compound in Abbottabad. As Obama has noted before, he only gets called on to make the tough decisions. The easy ones are made for him.

“One of the things you learn as president is you’re always dealing with probabilities,” he told me. “No issue comes to my desk that is perfectly solvable. No issue comes to my desk where there’s 100 percent confidence that this is the right thing to do. Because if people were absolutely certain then it would have been decided by someone else. And that’s true in dealing with the economic crisis. That’s true in order to take a shot at a pirate. That’s true about most of the decisions I make during the course of the day. So I’m accustomed to people offering me probabilities. In this situation, what you started getting was probabilities that disguised uncertainty as opposed to actually providing you with more useful information.” The president had no trouble facing reality. If he acted on this, he was going to be taking a gamble.

As Nate Silver notes in The Signal and the Noise, one of the reasons more data doesn't always lead to better predictions is that people interpret the data subjectively. The intelligence on Abbottabad was weighted by each member of Obama's administration along with all the evidence they'd accumulated in each of their lives, and so everyone put different likelihoods on whether the man seen pacing in the garden of the compound (nicknamed The Pacer) was actually Bin Laden. The result, Bowden notes, was a distribution of confidence, with most at 80 percent confidence but some as low as 30 percent.

At one meeting, Obama asked Morell, who was seated in a chair against the wall behind him, under the presidential seal, for his own view. Morell put the probability that The Pacer was bin Laden at 60 percent.

Morell had been personally involved in the flawed analysis of Saddam’s weapons capability and yet had felt more certain about that than he felt about this. “People don’t have differences because they have different intel,” he said. “We are all looking at the same things. I think it depends more on your past experience.” He explained that counterterrorism analysts at work on al-Qaeda over the past five years had enjoyed a remarkable string of successes. They had been crushing the terror group inside Pakistan and systematically killing its top leadership. So they were very confident. Those who had been at work longer, like himself, had known failure. They knew the fragility of even the soundest-seeming intelligence analysis. The W.M.D. story had been a brutal lesson.

“Mr. President,” he said, “if we had a human source who had told us directly that bin Laden was living in that compound, I still wouldn’t be above 60 percent.” Morell said he had spent a lot of time on both questions, W.M.D. and Abbottabad. He had seen no fewer than 13 analytical drafts on the former and at least as many on the latter. “And I’m telling you, the case for W.M.D. wasn’t just stronger—it was much stronger.”

Emphasis in the excerpt above is mine. Examples like this make a good case for electing a President with a good handle on statistics.

Bowden's article does not shed any light on who Jessica Chastain plays in the upcoming movie Zero Dark Thirty. The second trailer for the movie and the priority of the advance billing seem to make her CIA analyst a central figure. These days, she really does seem like the busiest girl in show business.