University of California at Berkeley psychologist Philip Tetlockmade an experimental project.
He picked 284 people who made their living "commenting or offering advice on political and economic trends" and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be reelected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the pundits were asked to rate the probability of several possible outcomes. Tetlock then interrogated the pundits about their thought processes so he could better understand how they'd made up their minds. By the end of the study, Tetlock had quantified 82,361 different predictions.
After Tetlock tallied the data, the predictive failures of the pundits became obvious. Although they were paid for their keen insights into world affairs, they tended to perform worse than random chance. Most of Tetlock's questions had three possible answers; on average, the pundits had selected the right answer less than 33 percent of the time. In other words, a dart-thro wing chimp would have beaten the vast majority of professionals. Tetlock also found that the most famous pundits in his study tended to be the least accurate, consistently churning out overblown and overconfident forecasts. Eminence was a handicap.
I love all this stuff, it's a bit of an unnerving eye opener. I'll try and dig out another study I read about that supposedly found that people were making decisions almost entirely without facts, and then just using the facts to consciously justify their unconscious decision.
Also interesting and slightly related is the Dutch stock gorilla Jacko, who has beaten the Dutch stock exchange for the past ten years in a row. He chose his stock portfolio by picking and later exchanging bananas which represented certain stock. Again, random chance beating so called experts.