PO 4700: Contemporary International Relations

Cognitive biases

People look for patterns where there are none

## Press [enter] to continue

Are you smarter than a rat?

T maze

On Forecasting

Motivation

  • Conspicuous failures of existing methods
  • Success of forecasting models in other behavioral domains
  • Increased processing power

Predicting vs forecasting

  • Sound theory, but do not know whether the antecedent conditions have been satisfied.
  • Even with info + theory, randomness can play a role
  • Prediction is possible without explanation

A problem is that these excuses are often used to justify poor forecasts

  • Explanation is possible without prediction:
    • Pacifists do not abandon Gandhi's worldview just because he said in 1940 that Hitler is not as bad as "frequently depicted" and that `he seems to be gaining his victories without much bloodshed'
    • Martin Feldstein predicted that the legacy of the Clinton 1993 budget would lead to stagnation for a decade.
  • Prediction is possible without explanation when people have forecasting successes

Judging judgement

What is a good judge?

Two criteria:

  • Getting it right
  • Thinking the right way

Getting it right

How do we measure it?

  • Accuracy
  • True positives at the cost of false alarms?
  • Risks of overpredicting vs underpredicting Should false alarms and hits be weighed equally? E.g., what is riskier:
    • in the 1980s,
      • underestimate the Soviet Union, tempting them to test the US's resolve?
      • Overestimate them and pay high military costs I.e., the risk here is to treat as `wrong' forecasters those who have made value-driven decisions to exaggerate certain possibilities.
  • How early?

Thinking the right way

  • Do not violate basic probability theory. i.e., probabilities should sum to 1
  • Adjust your probability estimates in the face of evidence

Political Forecasting: Is it blind luck?

Ontological Skeptics

Interdeterminacy is due to the properties of the external world. A world that would be just as unpredictable if we were smarter.

Psychological Skeptics

We mispredict because of the way our (limited) minds work

Results

  • Most existing research makes no effort at testing their theory on future data
    • "isms"
    • statistical models
  • Tetlock: let's see how well experts perform. 284 participants,
    • most with doctorates, almost all with postgraduate training in polsci, econ, international law, diplomacy, journalism
    • avg of 12 years of work experience
    • academia, think tanks, governments, IOs
    • Very thoughtful and articulate
    • Broad cross-section of political, econ and national security outcomes

Results

Source: Tetlock, p. 51

The experts fight back

  • Perhaps we didn't select the right experts
  • Perhaps our dilettantes are really experts
  • Maybe experts are very cautious.

Foxes v hedgehogs

see Tetlock's Expert Political Judgement

What can and cannot be predicted?

Where algorithms do well

  • Nate Silver
  • Routine elections

Where algorithms do less well

  • Nate Silver fails too, even for elections
  • For international events, we often lack data
  • Even simple indicators are tricky
  • Events are rare
  • Heterogenous environments

Going further

Challenge yourself!

Further readings

  • Philip Tetlock. Expert Political Judgment (remarkable book)
  • Nassem Nicholas Taleb. The Black Swan (very self-absorbed but entertaining)
  • Daniel Kahneman. Thinking Fast and Slow (decades of research condensed)
  • Nate Silver. The Signal and the Noise (a good overview)

What Data to use?

  • Structural indicators are too slow
  • Social media too fast
  • Event data

Existing projects