Natural scientists are big on replication. When one lab reports an important finding, other labs try to replicate it. If they can’t, as with Fleischmann and Pons on cold fusion, you have a problem.
Social scientists are getting bigger on replication. Leading social science journals now require authors to upload empirical datasets. But in practice, replication is rare, as Andrew Gelman notes. Replication is still not widely expected – and besides, there are far fewer social scientists than natural scientists.
What about political philosophy and history of political thought? I’m not sure how much replication has been discussed in these areas, except in empirical areas e.g. experimental philosophy. (Let me know if you have references about replication in other areas of philosophy! I’m not thinking about such things as checking someone’s logic, of course.)
This strikes me as an important issue. Indeed, much of my work – and much of this blog – is about showing the intellectual links between philosophy, history and social science.
This post will thus address replication in philosophy. Part 2 will cover history of political thought.
A common tool in political philosophy is thought experiments. Is it worse to kill someone than to let someone die? This is a hugely important moral question. It is also a hugely complex moral question. How do we approach such difficult problems?
One method used is thought experiments. Is it worse to intentionally drown a child if one wants to get its inheritance than to fail to help the same child if it slips in a bath and starts to drown? This is Frances Kamm’s example – see p. 18 of Morality Mortality, volume 2. And there are many similar examples: Nozick’s experience machine, the ticking time-bomb scenario, Jim and the Indians, trolley problems, and so on.
My suggestion is that philosophical thought experiments are actually easier to replicate than almost any natural or social science research, provided the thought experiment is outlined clearly enough. You simply think through the experiment as described by the author, and see what your intuition/answer is.
Sometimes you reach the same conclusion. Sometimes you don’t, as with Frances Kamm’s retort to Peter Unger – see p. 13 of Morality, Mortality, volume 2.
Sometimes you question whether your intuitions are reliable. This may be because you reject the nature of many of these thought experiments, as with Robert Goodin – see pp. 8-9 of Political Theory and Public Policy. Or it may be because you think your own intuitions have been primed by previous thought experiments, as Mike Otsuka discusses – see pp. 109-10 of his 2008 paper in the journal Utilitas.
Sometimes you re-run the thought experiment with a different model, e.g. different order or different frames. Otsuka does this in the paper mentioned just above.
And sometimes – perhaps most importantly – you re-run the thought experiment with different variables. For example, if one adds uncertainty to the ticking time-bomb scenario, even many people who initially advocated torture become less willing to do so. I use the ticking time-bomb scenario with first-year politics students as a way of getting them to think about thought experiments in terms of variables.
Replication with different variables is also important in the social sciences. My favourite example is Daniel Treisman’s 2007 paper in the Annual Review of Political Science. He attempt to replicate many well-known cross-national analyses of corruption, and finds that small and reasonable changes to the independent variables often alters the results (see pp. 222 onwards). This challenges the reliability of the data and the models. It’s strikingly similar to the way that Otsuka questions our intuitions and alters the model (see above).
Replication matters, and it’s pleasingly common in philosophy, at least in relation to thought experiments.