Science has Burnham and Roberts' reply (subscription required) to the criticisms that Science published on Lancet 2:
Bohannon fails to appreciate that cluster sampling is a random
sampling method. Sampling for our study was designed to give all
households an equal chance of being included. In this multistage
cluster sampling, random selections were made at several levels ending
with the "start" house being randomly chosen. From there, the house
with the nearest front door was sampled until 39 consecutive houses
were selected. This usually involved a chain of houses extending into
two or three adjacent streets. Using two teams of two persons each, 40
houses could be surveyed in one day. Of our 47 clusters, 13 or 28%
were rural, approximating the UN estimates for the rural population of
Iraq.Bohannon states that Gilbert Burnham did not know exactly how the
Iraqi team conducted its survey. The text sent to Bohannon, which he
fails to cite, said, "As far as selection of the start houses, in
areas where there were residential streets that did not cross the main
avenues in the area selected, these were included in the random street
selection process, in an effort to reduce the selection bias that more
busy streets would have." In no place does our Lancet paper say that
the survey team avoided small back alleys. The methods section of the
paper was modified with the suggestions of peer reviewers and the
editorial staff. At no time did Burnham describe it to Bohannon as
"oversimplified."Those who work in conflict situations know that checkpoints often
scrutinize written materials carried by those stopped, and their
purpose may be questioned. Unique identifiers, such as neighborhoods,
streets, and houses, would pose a risk not only to those in survey
locations, but also to the survey teams. Protection of human subjects
is always paramount in field research. Not including unique
identifiers was specified in the approval the study received from the
Johns Hopkins Bloomberg School of Public Health Committee on Human
Research. At no time did the teams "destroy" details, as Bohannon
contends. Not recording unique identifiers does not compromise the
validity of our results.Concerning mortality estimates, Michael Spagat may be content,
as Bohannon claims, with mortality data collected barely 1 year into
an escalating 3.5-year war. Others might not find these so helpful.
Bohannon's reply:
I do appreciate that cluster sampling relies on random samples. It is
indeed the very bone of contention. "Sampling for our study was
designed to give all households an equal chance of being included,"
Burnham and Roberts write. But according to their methods as published
in The Lancet, that is not the case.My article reports the concerns of Sean Gourley and Neil Johnson, who
point out that the starting house was always on a street "randomly
selected from a list of residential streets crossing the main street."
This excludes all the smaller streets -- including back alleys -- that
do not cross a main street. Maps of Iraqi cities, freely available at
www.earth.google.com, show that many residential areas would be
excluded by this survey protocol. People living in those
underrepresented households, Gourley and Johnson argue, are less
likely to be exposed to the violence -- car bombs, drive-by shootings,
airstrikes -- that accounts for most of the reported deaths.When I asked Burnham by e-mail about this possible source of bias, he
replied that "in areas where there were residential streets that did
not cross the main avenues in the area selected, these were included
in the random street selection process, in an effort to reduce the
selection bias that more busy streets would have." When I asked him
why the published methods leave out this wiggle room, he replied that
"in trying to shorten the paper from its original very large size,
this bit got chopped, unfortunately." I used the term "oversimplified"
to describe this discrepancy.I stated that "the details about neighborhoods surveyed were
destroyed." The details in question are the "scraps" of paper on which
streets and addresses were written to "randomly" choose households,
and as Burnham and Roberts explained to me, that record has indeed
been destroyed. I appreciate the difficulty of conducting a study in a
combat zone and also the researchers' desire to protect the survey
team and respondents. At the same time, scientists concerned about the
true number of Iraqi casualties want to know which method was used to
select households and whether sample bias can explain the high number
of violent deaths reported by Burnham et al. But without a clear and
explicit methodology or raw data to independently examine, it is
impossible to know.
Hat tip: Pinko Punko.
It seems to me that instead of looking at sample bias in the study, one can look at death bias for reported deaths- not where the people that were killed got killed but where they lived. I assume that people that live in back alleys still go to mosques to pray and markets to shop. I also note that it is quite rare to have an author reply, and this author is showing he has a very particular point of view, which strikes one as a little different that the usual Science write-ups. Additionally, the nature of how the Lancet study describes "random" should be able to be simulated without having the actual details.
I stated that "the details about neighborhoods surveyed were destroyed." The details in question are the "scraps" of paper on which streets and addresses were written to "randomly" choose households, and as Burnham and Roberts explained to me, that record has indeed been destroyed.
Kinda like a crashed hard drive.
Kinda like a crashed hard drive.
No, ben, more kinda like working in a war zone. Not equivalent to 'the dog ate my homework' at all.
Of course, many in our country can't relate, so the difficulties sometimes aren't apparent & don't make it into civic consciousness. But, of course, a symptom of that is Tim's bandwidth-eating denialist commenters who fill up this space with their ululations [ben doesn't ululate, BTW].
Best,
D
There's a difference between saying sensitive information that could endanger people's lives was deliberately destroyed and claiming that you failed to make up a back-up copy of any of the material relating to a project and being unable to produce a single scarp of paper relating to the project; a single colleague who you told about the project; any receipts or other documentation for the costs of the project or any of the people who supposedly worked for you on it.
Sorta, but it still matters. So all Lott had to do was keep his research computer in a dangerous location, say Detroit, and then he could just tell us that someone from da hood stole it. That would have made his results correct, right?
Bandwidth? How much "bandwidth" did this sentence just take up? Maybe we could organize drives to recycle all this bandwidth.
Owwwwwuuuuuuuuuu!
Wait, dang, I forgot that Lott had nothing else to back up his claims. Fudge.
ben ben ben ben ben ben ben. ben. buh en. ben. ben ben.