Central Asia

The hazards of expertise–and listening to experts
 Articles like this, that call into question what I am doing in my life and in this venue, make me want to turn off the news and go to the closest July barbecue: I can be a witty guest, so please send invite immediately.

In a 2005 review of  Expert Political Judgement: How good is it? How can we know?, Louis Menand discusses the author’s bleak findings for those of us who follow expert opinions and try to stay interested in world developments.  According to the book’s author, Philip Tetlock, news experts and forecasters–people that comment upon the outcomes of political news/events–are worse at predicting the future than the average Joe or Jane.  And this is what Menand at least has to say:

When they’re wrong, they’re rarely held accountable. . . they insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons.  They have the same repertoire of self-justification that everyone has. . .

Anyway, the work is based upon a twenty-year study of commentators, and he used statistics to help him codify the solutions.  One thing he noted was that “the more facts that a person knew, the less likely he was to predict the outcome”, because in part, the expert included more variables into his prediction without assigning probability to these factors.  In other words, choices such as State A will go to war/ State A will not go to war have a statistical probability of 50% each, but an expert will add factors such as the temperament of the leader, the economic exchanges, the trend of the most recent speeches in State A’s legislature, et cetera, using more factors to obscure probability rather than view it–a “diminishing marginal predictive return” for each fact or factoid the expert pulls forward to make his or her case.

This just sounds so contrary to the usefulness of scholarship that I had to think about it for awhile, and I believe I partly understand why: we are accustomed to writing, from say middle school on, essays where we defend a position by using specific facts to argue our case.  In other words, we are accustomed to rhetorical arguments rather than mathematical ones–personally I wouldn’t be able to stand news that included a lot of equations in order to decide the human dimension–but apparently this is wrong-spirited of me, as Tetlock has apparently proved. 

One reason experts ignore the probable: they have pet theories, which they hold to loyally, in the face of overwhelming uncertainty, and even in the face of overwhelming evidence.  
Second reason: 50-50 isn’t good enough to make a plan, especially when the outcomes are lucrative or dangerous.   
Third reason: it has to be interesting, so that we will watch them on television or read their articles and books and blog entries. 
Fourth reason: experts as well as audiences tend to self-select data.  For analysts, that means a more rigorous fact-checking of that which falsifies their theoretical knowledge; and for the public, it means that they “self-select” what they read and study.  The example Menand uses is U.S. liberals reading The Nation and conservatives The National Review.  It actually goes much deeper (or I should say, wider) than that: as a former bookstore manager, I have noted the increasing number of self-prescribing titles on the bookshelves these days.  Titles such as  these tend to aid in that self selection, from marriage to elections to World War III.  Such books cut off self-study and can be scurrilous and detestable, but they do make the reader, who actually already knows the contents, feel comfortably angry at the other side’s opinion.
Fifth reason: we believe scenarios that are filled in with details, e.g., the rhetorical argument rather than the mathematical argument.

So how do we protect ourselves from experts? And how, as potential experts, do we evaluate the world?  First, Tetlock says that experts who follow some kind of universal theory, a “one big pattern” are more likely to be wrong, where experts who believe in a kind of flexible “ad hoc” approach are more likely to be right.  This cuts out any unitary belief in “Democratic Peace Theory”, “Balance of Power”  “Great Man Theory” “Trade Peace Theory”–at least, if exercised to the exclusion of any other theories.  The problem for those who use an ad hoc approach, however, is that they never fail largely, but they don’t win largely, either.  It’s more of a gamble, but one looks more statesmanlike and profound when one is right with the “One Big Theory”.

Back in the 1990’s Susan Strange discussed the insularity of the U.S. in both academic congress–for instance, U.S.  academic journals on international relations most frequently accept U.S. authors–and in their continued use of “one big” theories when writing policy and predictions.  In contrast, she described Europeans as more able to accommodate more than one theory at a time and use them in concert when studying and planning.  We can argue the relative dogmatism of the U.S. versus Europe all day, but that’s not the point: the point is that no one-paragraph theory can encompass all the variables.  It’s also interesting that in that same talk (to Europeans) she takes them to task for self-selection and pet projects.

But once again, this only serves as a reminder against rhetorical reasoning and may very well bring us back to probability theory, and, since greater numbers of opinions tend to follow probability rather than rhetoric, “the wisdom of crowds” may well have a point.  Last of all, two quotes by eminent logicians, who would disdain the wisdom of crowds, but not that of simplicity or academic rigor rather than personal theories:

“I know nothing except the fact of my ignorance.”
Socrates, from Diogenes Laertius, Lives of Eminent Philosophers
Greek philosopher in Athens (469 BC - 399 B.C.E.)

Entia non sunt multiplicanda praeter necessitatem.
Never increase, beyond what is necessary, the number of entities required to explain anything.
William of Ockham, English philosopher (1288- ca. 1348 C.E.)

And that’s how we are going to have to measure our experts, and their arguments, as well: by watching for clues of their world-view as prejudicial to their predictions, no matter what world-view it is; and by discarding the irrelevant in their arguments.

And now that we’ve discovered yet again how hollow the world is, I’d like to come over for barbecue: that is, if you also have potato salad.  I can bring lemonade.  You’ll find me out on the street, taking my pet theories out for a little walk around the park.

Further Reading:
Sample Chapter of Tetlock’s Book

References: See Worth Reading Page for Strange (1998 ) and Tetlock (2005); also, thanks to various quotation pages on the Internet and Wikipedia for Occam dates.

This entry was posted on Wednesday, July 18th, 2007 at 9:37 am and is filed under Sourcenotes, Issues, States & Regions. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.