More on the estimation of war deaths

Following up on our recent discussion (see also here) about estimates of war deaths, Megan Price pointed me to this report, where she, Anita Gohdes, Megan Price, and Patrick Ball write:

Several media organizations including Reuters, Foreign Policy and New Scientist covered the January 21 release of the 2009 Human Security Report (HSR) entitled, "The Shrinking Cost of War." The main thesis of the HRS authors, Andrew Mack et al, is that "nationwide mortality rates actually fall during most wars" and that "today's wars rarely kill enough people to reverse the decline in peacetime mortality that has been underway in the developing world for more than 30 years." . . . We are deeply skeptical of the methods and data that the authors use to conclude that conflict-related deaths are decreasing. We are equally concerned about the implications of the authors' conclusions and recommendations with respect to the current academic discussion on how to count deaths in conflict situations. . . .

The central evidence that the authors provide for "The Shrinking Cost of War" is delivered as a series of graphs. There are two problems with the authors' reasoning.

First, the mortality estimates of children under five described in Figure 2.1 of the report should include an appropriate measure of uncertainty. The purported trend could be overwhelmed by the error of the estimates describing child mortality, but these errors are not presented in the report. Therefore, it is impossible to test the strength of the trend versus the magnitude of the error.

Second, and even more importantly, the graphs showing a worldwide decline in war-related lethality, in Figures 2.5 and 2.6 of the report, include data from the PRIO Center for the Study of Civil War, International Peace Research Institute, Oslo and the UCD/ HSRP Uppsala Conflict Data Program /Human Security Report Project Dataset, the World Bank, "World Development Indicators" and the Inter-Agency Child Mortality Estimation Group (IACMEG), "Child Mortality Estimates Info."

These datasets include expert opinions, convenience datasets, and reproducible estimates from multiple-systems estimation and probability-based surveys. The quality and uncertainty of such an incompatible collection of datasets cannot be evaluated. The plausibility of the trends presented by the authors cannot be assessed.

In particular, expert opinions about the magnitude of violence are little more than guesses. These numbers are the statistical equivalent of hearsay. It is impossible to scientifically debate speculations, and it is impossible to reproduce speculations by a principled process in alignment with the scientific method.

Some of the other sources for the authors' conclusions are convenience samples. A convenience sample is simply data that can be conveniently observed via witness accounts, press sources, or other means. These databases may be useful collections of cases, and organizing information in this form enables many non-statistical descriptions of violence.

However, convenience samples are highly unlikely to represent the underlying statistical patterns or magnitude of conflict related deaths. They cannot be used to extrapolate to a population beyond what was observed, and they are not necessarily representative of any population. In our experience, no two convenience samples about the same country tell the same statistical story.

The core problem with both expert opinions and convenience samples is that we have no way of scientifically measuring just how unrepresentative they actually are of the population they are attempting to measure.

Getting to the conclusion, they write:

The HSR's broadest claim is that "[t]he average conflict in the new millennium kills 90 percent fewer people each year than did the average conflict in the 1950s (p. 2)," and that there has been a "20-year decline in conflict numbers" (p. 7). Due to the weaknesses in the data, there is no way of reproducibly or transparently verifying or testing this claim.

In the HSR and elsewhere, the HSR authors have shown that recent conflicts have been seriously mismeasured. Certainly this debate would benefit from a scrutiny of expert opinions and convenience samples as intense as that which the HSR authors' have brought to the study of the IRC surveys.

Perhaps earlier conflicts were as mismeasured as recent conflicts. The HSR authors' conclusion that the number of deaths in today's wars is declining may be right, or it might be wrong. But we just do not know much about the quality of estimates from further back in history. Therefore, the only responsible conclusion is that we simply don't know what the trend in war-related deaths looks like. More rigorous research is needed.

In summary, they write:

We welcome the authors' contribution to the ongoing debate about measuring war-related mortality. Technical critique of existing work is at the core of the scientific process. In our opinion, the HSR authors have done the IRC and the community of human rights analysts a service by highlighting errors in the Congo survey.

The response to errors with statistical estimates must not be that we abandon science by relying on expert opinions and convenience samples. Quite the opposite. The fact that the IRC's work has been shown by the HSR authors to be flawed should remind us to limit our conclusions narrowly to what can be defended by the most appropriate and advanced scientific methods for the question at hand.

My only contribution to this discussion was to question some of the low numbers that are floating around, for example for Guatemala, which makes me think that some of the difficulties come from how exactly "war deaths" are defined.