America's top spies go up against a crowd

By Ken Dilanian

Aug. 21, 2012 12 AM PT

WASHINGTON — Nine years ago, Congress blocked a Pentagon agency from setting up a website that would have allowed anyone with a credit card to bet on the likelihood of foreign assassinations, coups and terrorist attacks.

The idea was to take advantage of the “wisdom of crowds,” a social science maxim that contends the average of a group of forecasters, under certain circumstances, tends to be more accurate than even the most knowledgeable single forecaster.

But lawmakers worried the proposed predictions market could allow terrorists to profit from their own misdeeds. Congress forced the Defense Advanced Research Projects Agency -- the military’s cutting-edge research arm, known as DARPA -- to scuttle the program.

Now terrorism futures are back.

DARPA’s sister agency -- the Intelligence Advanced Research Projects Activity, which funds experimental projects for the U.S. intelligence community -- is running a four-year, $50-million program that pays people willing to predict major world events, including wars and terrorist strikes. Unlike the earlier scheme, participants can’t profit from their predictions.

Now in its second year, the so-called crowd-sourcing project involves competing corporate and university teams, including UC Irvine. Each team includes more than a dozen social scientists and as many as 2,000 participants, who can answer hundreds of questions each if they want.

The study, known as Aggregative Contingent Estimation, is designed to see whether the 17 agencies in the U.S. intelligence community can aggregate the judgment of its thousands of analysts -- rather than rely on the expertise of just a few -- to issue more accurate warnings to policy makers before and during major global events.

Participants in the project give their best guesses on whether the Free Syrian Army will gain control of Aleppo, for example, or whether Kim Jong Un will resign as the leader of North Korea before April 1, 2013, according to project websites.

Initial results are promising, according to Philip Tetlock, a University of Pennsylvania professor who leads one of the teams.

His 2005 book, “Expert Political Judgment: How Good Is It? How Can We Know?” showed that political pundits often fared worse in predictions than the proverbial dart-throwing monkey.

“The idea that the average of a group of forecasters is more accurate than any one forecaster goes back 100 years,” Tetlock said. In 1906, British scientist Francis Galton determined that the average of 787 guesses about the weight of an ox at a county fair was startlingly accurate.

The teams are trying to beat the average by using algorithms that, over time, give greater weight to the most successful crystal gazers among their participants and weed out the worst.

Tetlock’s group, the Good Judgment Project, is in the lead so far.

“In year one, we beat the unweighted average by about 57%, which was big,” he said. A control group, run by Mitre Corp., averages scores without giving weight to participants who tally the best results.

Predicting world-changing events is never easy. U.S. intelligence agencies failed to foresee the fall of the Soviet Union, to predict the terrorist attacks of Sept. 11, 2001, and, more recently, to anticipate political upheaval across the Arab world.

The CIA and other agencies tend to rely on small groups of experts who hash out consensus judgments. If there is strong disagreement among the analysts, or between different agencies, an intelligence report will note that.

Crowd-sourcing would mean, in theory, polling large groups across the 200,000-person intelligence community, or outside experts with security clearances, to aggregate their views about the strength of the Taliban, say, or the likelihood that Iran is secretly building a nuclear weapon.

For now, there is deep skepticism in intelligence circles.

“I don’t believe in the wisdom of crowds,” said Mark Lowenthal, a former senior CIA and State Department analyst (and 1988 “Jeopardy!” champion) who now teaches classified courses about intelligence. “Crowds produce riots. Experts produce wisdom.”

Others argue that intelligence analysis can benefit from data-driven social science.

The CIA’s venture capital arm, In-Q-Tel, was an early investor in a private company called Recorded Future, headquartered in Cambridge, Mass., and Goteborg, Sweden, which uses data on the Internet, from securities filings to Facebook posts to government statistics, to make connections and spot trends.

A National Academy of Sciences report last year commissioned by former National Intelligence Director Dennis C. Blair found that intelligence practices “have been only weakly informed by the behavioral and social sciences.” The report recommended that “science and evidence-based protocols” be introduced and evaluated alongside the spy agencies’ traditional reliance on expert judgment.

The crowd-sourcing project is one result. Each team recruited its own forecasters, whose names are not made public. A few are or have been affiliated with intelligence agencies, organizers said. Some teams require participants to have college degrees, while others take all comers.

“We would prefer people who actually have a background and an interest in current events,” said Charles Twardy, who is co-directing the team at George Mason University in Virginia. “But it’s open to anyone who wants to participate.”

Pure futures markets using real money, such as an Irish company called Intrade, allow participants to buy shares in a prediction. For example: “Bashar al-Assad to no longer be President of Syria before midnight ET 31 Dec 2012.”

If Assad is out by year’s end, the contract will pay $10 a share. If he stays in power, it will pay zero. Intrade’s website recently had five shares available for $5.99, meaning the market set the probability of Assad’s ouster by Dec. 31 at about 60%.

Intrade’s consensus called every state correctly in the 2004 U.S. presidential election and all but two right in 2008. But it also estimated a 77% chance the U.S. Supreme Court would throw out President Obama’s signature healthcare legislation (it didn’t) and recently predicted a cautious vice presidential pick by Mitt Romney (he surprised many when he chose Rep. Paul D. Ryan).

Tetlock praised the willingness to fund a study that could expose weaknesses in how spy services do their work.

“It’s not often that government agencies do something that has the potential to make them look not so good,” he said.

[email protected]