Polling creation/evolution

Attention conservation notice: 3000 words about how smart people who ought to know better are reading way too much into a poll.

Last May, NCSE reported on a poll on evolution conducted by Virginia Commonwealth University. The results were, to my eye, in line with most of the other polling out there, so I never wrote about it here, other than a passing mention in a post about my WaPo review of Elaine Howard Ecklund's Science vs. Religion.

Anyway, a month and a half later, Jerry Coyne and Jason Rosenhouse have discovered the poll, and each has found different questions in the poll that they find worthy of comment. Before we delve into this, I'll suggest that folks read professional pollster George Bishop's essay on the vagaries of polls on evolution and creationism, and the pitfalls to beware.

Jason picks up on the question:

Which of these statements comes closest to your views on the origin of biological life: biological life developed over time from simple substances, but God guided this process, biological life developed over time from simple substances but God did not guide this process, God directly created biological life in its present form at one point in time?

43% chose "directly created," 24% chose "God guided" evolution, 18% chose "God did not guide" evolution, and 6% were undecided or refused to answer.

These results are nicely in line with historic answers to Gallup's surveys. For the last 30 years, Gallup has asked the similar question:

Which of the following statements comes closest to your views on the origin and development of human beings? (1) Human beings have developed over millions of years from less advanced forms of life, but God guided this process. (2) Human beings have developed over millions of years from less advanced forms of life, but God had no part in this process. (3) God created human beings pretty much in their present form at one time within the last 10,000 years or so.

Note that this question asks about humans, while VCU asked about "biological life." For the last 30 years, the Gallup numbers have varied within a 5 point range of this: 45% chose "created in present form," 37.5% chose evolution "guided by God", 11.5% chose "God had no part" evolution, and 5.5% were undecided or refused to answer.

As Jason notes, the big differences are VCU's significantly higher support for "did not guide" than Gallups for "had no part." It's a 9 point difference between VCU's result and Gallup's latest poll, and a 6.5 point jump from the long-term Gallup average. The "God guided" option took the hit in VCU's poll relative to Gallup, as the "directly created" is basically unchanged.

Jason looks at that difference and tries to develop it into a falsification of accommodationism (a term which I've noted could do with a clear definition some day). This is a deeply unsatisfying argument, as it glosses over some pretty important issues.

Not least among the problems, I don't know that anyone from the "accommodationist" camp has ever proposed the hypothesis Jason is testing. Without some clearer citation, it's hard to know if this is a legitimate extension of â I'm guessing â Chris Mooney's writings. I can see where someone might get it, but I don't know that Chris has ever claimed explicitly that New Atheism will actively decrease public acceptance of evolution or increase acceptance of creationism. I certainly know I haven't made that claim. Most significantly, I wouldn't claim that because I think the main effect of that fracas is to simply turn people away from experts whose opinion they might otherwise listen to. This could swell the ranks of the undecided, but I doubt it will move many people out of that broad middle. My concern, and what I understand Chris (and his co-authors Matt Nisbet and Sheril Kirshenbaum, at various times) to have principally argued, is that the New Atheism is likely to turn people away from folks they'd otherwise regard as trustworthy experts, and that this could hold back ongoing outreach efforts. These data don't really allow testing of that hypothesis. (I've hedged my language here because it may be that there is such an explicit prediction, and this paragraph would be moot.)

The more significant error I see in Jason's post is that he's comparing apples and oranges. Gallup asks about human evolution, VCU asks about "biological life." Framing the question in terms of humans tends to reduce support for evolution while framing it in terms of all life tends to boost support for evolution. I don't know of other polls using the "biological life" phrase, but I'd imagine that mentioning biology would also cue respondents to answer in terms of their understanding of the science, rather than their personal beliefs.

As I noted in Jason's comments, Harris tends to ask a question about human common ancestry in parallel with a question about common ancestry of all life, which lets us get a feel for the magnitude of that effect. The most recent such survey found:

In reply to one question almost half (45%) of adults say they believe humans were created directly by God and only 29% say they evolved from other species. In reply to another question 53% of these same people say they believe that âplants, animals and human beings have evolved over time,â and only 21% say they do not believe this, with fully 25% who are not sure or decline to answer.

Based on this, the magnitude of the effect is roughly 8 points. That's statistically indistinguishable from the difference between Gallup's question and VCU's question. Even without considering the inherent problems of comparing one pollster to another â the house effects and order effects that can consistently, week after week, bump results up or down by several points when two pollsters ask identical questions â we can account for the difference between Gallup and VCU simply by considering the starkly different wording of their two questions.

Furthermore, Jason compared VCU's question in 2010 to Gallup's from 1998. But Gallup has been asking the question for 28 years. So why compare Gallup 1998 to VCU 2010? When we look at Gallup's results, we find a fairly steady increase in acceptance of "God had no part," from 9% in 1982 to 14% in 2008. But the Gallup poll's margin of error is 3 points, making it statistically impossible to say that the 1982 result is different from the 2008. I could probably do a time series analysis to figure out if the trend from all 9 data points is statistically significant, but it doesn't matter for our purposes here. (NB: Thanks to a tip from Jack Weiss at Jason's blog, we can say that there is a significant upward trend throughout the whole dataset.)

It doesn't matter because the rise is fairly steady over time, with no obvious inflection when the "New Atheists" started publishing. Since 2004, when Harris published The End of Faith and not long before Dawkins published The God Delusion, support for "God had no part" has moved from 13% to 14%. Since 1999 (a low outlier from the general trend, selected by Jason for no obvious reason), it has risen 4 points, but since the Four Horsemen began their march, it's risen 1 point, statistically no different than noise. If there's an inflection point â and I don't think statistical analysis would find one â it would be between August 1999 and February 2001, with (if anything) a flattening since 2004. If there's a statistically significant trend to the data at all, though, I don't think there'd be statistically significant inflection points. Certainly not from 9 data points (6 in Jason's discussion).

Finally, Jason should have compared VCU's 2010 poll with their poll from September, 2005 (see update below). I plotted those results above, and you can clearly see that none of these values shifted significantly. If "New Atheists" had an effect on "God did not guide," you'd expect to see some of that signal since 2005, but you don't. Not in Gallup, not in VCU. Note also the much larger fraction of VCU's sample that is "undecided." This is probably an artifact of the different pollsters and not of question differences, as some pollsters push a respondent to choose if they first offer "don't know" or "undecided," while other pollsters simply enter that as a nonresponse. If this tells us anything about accommodationism, it says that when you push people without strong feelings to weigh in on evolution, they'll tend to pick the "God guided" option if it's available. It'd be interesting to have data where a similarly ambivalent population was forced to make a binary choice, and see how many break for "God created" versus "evolution only." I would guess they'd pick "God only." This is supported by findings from, for instance a 2005 Harris poll, in which respondents chose between evolution (not described as guided or unguided), ID creationist language about complexity requiring "a powerful force or intelligent being" not identified as God, and direct creation. 22% chose evolution (9 points up from 2004 and 2006 Gallup results for unguided evolution), 10% chose ID (compared to 36-38% in Gallup's guided evolution question that could cover ID or TE), and 66% chose creation by God (compared to 42-46% in Gallup). In other words, less than a third of the "guided evolution" respondents may have jumped to a general evolution option, a third favor some form of ID creationism, and the rest were comfortable with unabashed creationism.

In short, there's simply nothing in Gallup's data, or in an informed comparison with VCU's data, that supports Jason's claimed "doubling in ten years," nor is there justification for his attempt to attribute causality, one way or another, to the rise of the "New Atheists." Jason is a mathematician, the author of a good popular book on statistics probability, and is working on a book that will (among other things) examine public polling on creationism and evolution. I'm confident he'd never try to read causality or claim both a trend and an inflection point based on 9 data points in his professional writing, nor would he claim equivalence between differently worded questions from different pollsters, and I don't know why he's doing it here.

Turning to Jerry Coyne, we find a different question being discussed. He's interested in this question:

In general, would you say the theory of evolution conflicts with your own religious beliefs, or is mostly compatible with your own religious beliefs?

As I mentioned in May, 42% found conflict, 43% say evolution is "mostly compatible," and 16% did not give an answer. In May, I compared these to results from a Pew survey in 2008, which found 55% of Americans thought religion and science are "often in conflict" in general, but 61% say science does not conflict with their own faith.

Anyway, Coyne takes the VCU result and opines:

The large chunk who see conflict is bad news for accommodationists. But the accommodationist responseâat least that of the National Academy of Sciences, the National Center for Science Education, and the American Association for the Advancement of Scienceâis this: You donât understand your own faith, because if you did, you would see that thereâs really no conflict. They have a big theological task in front of them.

First, I don't quite see the bad news. Second, that most Americans are theologically illiterate would not be news, though these data don't get us down to religious affiliations to evaluate where respondents stand relative to their denominations. Third, public opinion polls do not tell us what is true, only what people think is true. Fourth, the statistically even split means that Coyne and the "New Atheists" have to make the same "you don't understand your own faith" argument to the 43% who see no incompatibility. Why is this "bad news for accommodationists" but not equally bad news for Coyne?

To the first point, then, Coyne could be making either of two arguments. Either a) it's bad news because when 42% of people see a conflict, it means that there really must really be a conflict (while when 43% say there's no conflict, it means there could still be a conflict) or b) it's bad news because there are so many people who have to be convinced that evolution is not incompatible with their religion. Option a makes little enough sense that I'll assume he means option b. And yes, it is a problem, but it isn't news (see the Pew survey from a year earlier). Furthermore, it's a problem of exactly the same proportion as the one facing anyone who, like Coyne, wants to convince Americans that evolution is incompatible with their religion, not to mention the further challenge such a person would face of having to get people, once they accept that conflict, to reject religion and accept evolution. In what sense is the news worse for "accommodationists" than for the "New Atheists" and creationists trying to woo that 43% while squabbling over the 42% who already see a conflict?

As for the second objection, many members of religious groups that have officially endorsed evolution still believe that their religion is incompatible with evolution. It seems reasonable to tell such people that they are at odds with their own church's doctrine. That's not insulting, and it isn't a theological task. It's an educational task. Telling someone that the Methodists have endorsed evolution is no more advocacy for Methodist theology than telling them that Methodists believe Jesus is their Lord and Savior. One can disagree with either belief while still accurately informing people that those are the official views of United Methodism. Given that most people cannot identify Genesis as the first book in the Bible and 2/3 know that Jesus delivered the Sermon on the Mount (and I can), I feel comfortable telling people about the basic doctrines of those denominations whose leadership has issued clear statements in support of evolution. Where my own knowledge gets thin, I can rely on advice from a trained theologian on NCSE's staff, our Director of Religious Community Outreach.

To the third point, I'll note that these public polls are less informative about the truth of this matter than a review of the number of religious scientists, a statistic Coyne has derided. There's no question that it signals a problem when 31% of nonscientists think scientists have doubts about evolution, but it does not signal that those people are right. If you want to understand a technical topic, you consult relevant experts. In the case of evolution, you talk to biologists, and get near unanimity about the central role of evolution in biology. In the case of theology, you talk to theologians, and in the case of the intersection of science and religion, you talk to people who study science and religion, who agree that there need not be conflict (though conflict is, of course, possible).

And the fourth point speaks for itself. Why is this bad news for "accommodationists" but not for "New Atheists"? The numbers are nearly identical, so it would seem that the bad news is symmetrical as well.

Then Coyne makes an argument that never ceases to amaze me when I find it in the wild:

How do we solve the problem? Many scientistsâatheists and accommodationists alikeâare trying to educate people about what evolution is and how much evidence supports it. Sadly, that doesnât seem to be working very well, at least judging by how little acceptance of evolution has budged over the last few decades.

It is not sufficient to look at polls, see no change, and declare that what we're doing isn't working. It would only be valid if you had a controlled experiment, or some sort of randomization or statistical controls that let you isolate the effects of these education efforts from all the other effects out there.

What other effects? Answers in Genesis has a budget that dwarfs NCSE's, and AiG is only about 2/3 of the creationist marketplace. Toss in funding to natural history museums, NSF educational funding and funding for "broader impacts" of research grants, and more may be spent on evolution outreach than on creationist outreach, but creationists also have a lot of volunteers, and a lot of churches and other groups promoting creationism outside of official channels. Note also that creationist funding seems to be rising more rapidly than NCSE's funding.

We also know that 1 in 6 high school biology teachers are creationist, and 1 in 8 of those teachers spend at least an hour advocating creationism. Teachers are rarely required to have a college class in evolution, and those who do not have such a class are less likely to spend time on evolution. These are all serious issues, and there's no way that someone as versed in experimental design and data analysis as I know Coyne to be would ignore those. The science education efforts Coyne describes are not happening in a vacuum, and any evaluation of their success given public polling data has to account for everything else happening in society.

Given what we'll call a rough parity of concerted effort, is it surprising that there's little to no trend in the polls? No. All it means is that science education efforts are canceling out the effects of creationist efforts. Note also that polls today will not reflect data from students currently in school, and that there's far less outreach to adults than to school children. This would introduce a significant lag into any signal one might find in the polls. Assuming kids take high school biology in 9th grade, when they're 14, they are unlikely to be polled until they're 18, so that's a minimum 4 year lag, and longer until enough students reach the adult population to show up in polls to significant degrees. Naively waving poll numbers around without any sort of mention or consideration of these factors is not the solution.

Now I'm sure that someone with the right data and some serious statistical models could tease out these effects. I don't know that the right data exist at all, but maybe the GSS have data with which we could isolate a signal and identify the effects of creationist outreach vs. scientific outreach, and see what works and what doesn't. But Jerry cites no such research, he simply asserts that a flat poll line means that what is currently being done is not working. I know he wouldn't approach data from an uncontrolled field study so blithely, and I can't imagine why he'd treat these data that way.

Update: Thanks to a commenter, I found out I misentered the VCU '10 data for the "guided evolution" option when graphing results (basically, all the Gallup data for that option were in the 30s, and I accidentally put the first digit as 30 for the VCU poll, making it 34 when it should be 24). This made it seem like there was an increase in that value, and based on that typo, I made some inaccurate comments about what you find in comparing VCU '05 to VCU '10. I've corrected the paragraph on this above, I think without changing the substance of my argument. Hopefully this error does not cause confusion based on comments I left at Jason's blog.