About a week ago, my brother sent me a couple of interesting papers about funding in science, one in Canada, the other in the UK. I barely had time to skim the abstracts at the time, but thought I would put it up for discussion online and come back to it later. So I posted the link, abstract and brief commentary a few days ago to the article: Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant:
Abstract: Using Natural Science and Engineering Research Council Canada (NSERC) statistics, we show that the $40,000 (Canadian) cost of preparation for a grant application and rejection by peer review in 2007 exceeded that of giving every qualified investigator a direct baseline discovery grant of $30,000 (average grant). This means the Canadian Federal Government could institute direct grants for 100% of qualified applicants for the same money. We anticipate that the net result would be more and better research since more research would be conducted at the critical idea or discovery stage. Control of quality is assured through university hiring, promotion and tenure proceedings, journal reviews of submitted work, and the patent process, whose collective scrutiny far exceeds that of grant peer review. The greater efficiency in use of grant funds and increased innovation with baseline funding would provide a means of achieving the goals of the recent Canadian Value for Money and Accountability Review. We suggest that developing countries could leapfrog ahead by adopting from the start science grant systems that encourage innovation.
A long and interesting discussion ensued in the comments, with the author of the paper himself showing up and offering to send reprints to those who are interested. More discussion also happened on FriendFeed here and here.
Several other bloggers also posted about it, and discussions happened on their posts as well. T. Ryan Gregory posted about it both on his Nature Network blog Pyrenaemata and on his indy blog Genomicron.
Larry Moran was largely in agreement with the article, but some commenters were not, including Rosie Redfield whose comment motivated T. Ryan Gregory to post again, just to explain his disagreement with Rosie.
Jonathan Eisen pointed out to a related post of his and Cameron Neylon to a related post of his. Finally, Zen Faulkes used it as a starting point for three posts here,
here and here.
I have finally managed to find time to read the paper myself so I think I can say something semi-intelligent about it. It became obvious that many who commented have not actually read the paper, just the Abstract, and thus were not in the position to respond to it intelligently (the paper actually answers, clearly and in detail, all the questions and complaints voiced by the commenters). The abstract is just, ...well, an abstract. The paper is full of thought-provoking ideas and really needs to be read in its entirety.
Finally, my brother showed up in the comments and I would like to use his comment as a starting point today. That is - once you read the actual paper (ask for a reprint if you cannot access it), the linked blog posts and comment threads. I'll be right here, waiting for you to come back....
I am assuming that the Canadian funding system is not very dissimilar to that in many other countries, including the USA - there is a central governmental body that gets its budget from the government and uses committees of unpaid peer-reviewers to decide how the money will be allocated to the researchers. The paper explains in detail at least a dozen reasons why and how this system is flawed: how it stifles truly innovative science, repels students from entering science, disproportinately pushes women out of science, encumbers students and postdocs with tasks they are not supposed to be doing (e.g., clerical, or technical), introduces an element of uncertainty about one's livelihood, gives universities excuses to completely get out of research funding, shafts teaching and outreach as criteria for promotion, etc. But the clincher, for politicos at least, is that this system costs more than if a set sum of about $30,000 (Canadian) was given to every academic employed by a Canadian university who asks in any given year.
Yes, giving every Canadian scientist who already has a job and a lab this small amount of money no-questions-asked, geared toward innovative exploratory research, costs the government less than going through the peer-review system that gives money to some and no money to others (not to mention the reinforcement of the Old Boys Club this way).
This does not mean, in their proposal, that all of the Canadian money earmarked for science would be given this way - this is still just a small part of it. If you have a big lab or do expensive research and need to apply for much bigger grants, that would be done by the traditional peer review. But in order to get to the point where you have a good proposal, you need to have some neat stuff done (the "preliminary data"). With the proposed system, that preliminary data can be really exciting or revolutionary, something that, as an initial proposal, would never fly by peers.
Would people send out proposals for crap? Some would, I'm sure, but that doesn't matter. Most would not. Scientists are curious about nature and would like to test their hunches. Some will flop, some will be amazing - it is the latter that this new system is worth doing for, as they may never be done otherwise. Anyway, how many $5,000,000 grants produced amazing stuff? All? I.Don't. Think.So.
Where does quality control come from? First, it already came from universities who hired these researchers out of hundreds of applicants for each position. Aren't they going to trust those best-of-the-best they hired? Second, the research itself will be judged after it is done - at conferences, in journal articles, and in post-publication metrics (citations, downloads, online chatter, etc., including perhaps a Nobel Prize here and there). If it is not up to standards, $30,000 Canadian dollars is not a big price to pay, and even the negative or inconclusive results can be useful to others if the thinking is original. If it is up to standards or more, that person will now have something exciting to base a bigger grant proposal on.
This also goes back to something I like to rant about (oh yes, go read that again!) - the bandwagon of Big Science. Biology, for example, does not equal running gels (hmmm, that's chemistry, isn't it?). But many people are given that appearance. "No gels - no grants, no papers, no career" (yup, I was told that a few years ago). Unless you already have a big molecular lab, this small grant will not build you one. Instead, you can do some really cool stuff at other levels - from tissues up to ecosystems and everything in-between, including computer modeling. You can use it to travel to some jungle that has never seen a Westerner and see what species live there - not hypothesis-testing, exploratory and exciting, definitely useful, but not something that is easily funded with a current system. If your proposal includes research on live vertebrates, you first have to get an IACUC proposal, something that will take 6-9 months of extremely frustrating fighting and proposal-modification - getting an IACUC proposal is the toughest peer-review known to science: if they say Yes to your proposal, no other committee of peers can add any more wisdom to it. And if you decide to work on invertebrates - it is much cheaper.
Another paper looks at this from another perspective - four stages of science. The grants, especially the big ones, disproportionately target science in Stage 3. The small baseline grants would target primarily Stage 1, the exciting, innovative stage - and this is a Good Thing. They could also more easily fund research in Stage 2 and Stage 4, also a Good Thing - from the article:
In this article I propose the classification of the evolutionary stages that a scientific discipline evolves through and the type of scientists that are the most productive at each stage. I believe that each scientific discipline evolves sequentially through four stages. Scientists at stage one introduce new objects and phenomena as subject matter for a new scientific discipline. To do this they have to introduce a new language adequately describing the subject matter. At stage two, scientists develop a toolbox of methods and techniques for the new discipline. Owing to this advancement in methodology, the spectrum of objects and phenomena that fall into the realm of the new science are further understood at this stage. Most of the specific knowledge is generated at the third stage, at which the highest number of original research publications is generated. The majority of third-stage investigation is based on the initial application of new research methods to objects and/or phenomena. The purpose of the fourth stage is to maintain and pass on scientific knowledge generated during the first three stages. Groundbreaking new discoveries are not made at this stage. However, new ways to present scientific information are generated, and crucial revisions are often made of the role of the discipline within the constantly evolving scientific environment. The very nature of each stage determines the optimal psychological type and modus operandi of the scientist operating within it. Thus, it is not only the talent and devotion of scientists that determines whether they are capable of contributing substantially but, rather, whether they have the 'right type' of talent for the chosen scientific discipline at that time. Understanding the four different evolutionary stages of a scientific discipline might be instrumental for many scientists in optimizing their career path, in addition to being useful in assembling scientific teams, precluding conflicts and maximizing productivity. The proposed model of scientific evolution might also be instrumental for society in organizing and managing the scientific process. No public policy aimed at stimulating the scientific process can be equally beneficial for all four stages. Attempts to apply the same criteria to scientists working on scientific disciplines at different stages of their scientific evolution would be stimulating for one and detrimental for another. In addition, researchers operating at a certain stage of scientific evolution might not possess the mindset adequate to evaluate and stimulate a discipline that is at a different evolutionary stage. This could be the reason for suboptimal implementation of otherwise well-conceived scientific policies.
Now, the proposal in this paper is quite definitive about allowing only researchers employed by universities to apply for such grants. But my mind instantly started thinking about those outside. How about amateur scientists? How about people not affiliated with the academia? How about distributed citizen science projects? Those are usually Stage 1 or Stage 2 projects, attractive to a particular kind of researcher (myself included - don't try to lure me into a big Stage 3 lab). If I wanted to get some crayfish or spiders (or even birds, if a local IACUC would let me) and do experiments at home, this kind of a small grant would be just ideal. Could I have a local University, or some peers, write a letter in support of my proposal? Would that fly?
The paper also mentions, in a couple of places, similarities and differences between peer-review of grants and peer-review of manuscripts, including the importance of Openness to science. In one place, it mentions new journals "where ideas may be published initially unreviewed, but anyone may append public discussions to each article". I am hoping this refers to arXiv and Nature Precedings, or even the concept of Open Notebook Science, but it smells too much like one of the pernicious myths spread by the enemies of Open Access about PLoS ONE which is, as readers of this blog are aware, stringently peer-reviewed.
One thing that the article mentions is that the current granting system allows researchers to buy time for research away from their teaching time. They note this as bad for teaching, true, but there is another angle to it. As danah writes in regard to the new proposed NSF funding of qualitative research, this kind of work does not require much in terms of equipment, but much in terms of time. It is essential for people, especially in social sciences, who do qualitative research, to be able to buy the time they need to do their research correctly.
Oh, and I mentioned at the beginning that my brother sent me two papers, yet we talked here only about one of them. The other one, if you are interested in starting a whole new discussion, is this one: Life after death? The Soviet system in British higher education
Recent studies of British higher education (HE) have focused on the application of the principles of the 'new managerialism' in the public sector, ostensibly aimed at improving the effectiveness of research and teaching, and also on the increasing commercialisation of HE. This article examines HE management in the light of the historical experience of the Soviet system of economic planning. Analogies with the dysfunctional effects of the Soviet system are elaborated with regard to financial planning and the systems of quality control in academic research and teaching. It is argued that Soviet-style management systems have paradoxically accompanied the growing market orientation of HE, undermining traditional professional values and alternative models of engagement between HE institutions and the wider society.
A FriendFeed discussion has started. Read the entire paper before chiming in, of course - we are scientists here!
Gordon, R., & Poulin, B. (2009). Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant Accountability in Research, 16 (1), 13-40 DOI: 10.1080/08989620802689821
- Log in to post comments
More like this
This makes no sense, as it kills all the truly innovative science by funding it paltrilly, so that a fuckton of boring-ass shit can putter along.
Read the paper.
Innovative science does not get funded. Expensive science does.
If you want a reprint of:
Gordon, R., & Poulin, B. (2009). Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant Accountability in Research, 16 (1), 13-40 DOI: 10.1080/08989620802689821
or our previous papers:
Poulin, B.J. & R. Gordon (2001). How to organize science funding: the new Canadian Institutes for Health Research (CIHR), an opportunity to vastly increase innovation. Canadian Public Policy 27(1), 95-112.
Gordon, R. (1993). Grant agencies versus the search for truth. Accountability in Research: Policies and Quality Assurance 2(4), 297-301.
contact me at: gordonr@cc.umanitoba.ca
Bryan Poulin (bpoul@lakeheadu.ca) and I would be glad to answer any question, and entertain suggestions on how to turn this proposal into reality.
-Dick Gordon
The paper makes it fairly clear that the idea of establishing this baseline support would probably only work under a restricted set of conditions: small requests and high funding rates.
NSERC is one of the few funding agencies in the world that might fulfill these criteria right now. While someone in another blog was surprised by the average $30K value, he probably wasn't aware that the success rate of proposals to NSERC is about 70%.
Another factor that might be worth clarifying in this thread: Does all the money in an NSERC grant still go directly to the researcher? I remember that it used to be that universities did not get overhead from NSERC, but I'm way out of touch with the current state of affairs back home.
Sounds very intriguing. Don't have the time to read into it in detail, though I may later. I do have one surface question, though:
Let's say that this is the best possible way of funding science for right now. What will this do to the community of scientists down the road? By this I mean the following:
1. Would the cost of such a project start to balloon because more people ask for grants?
2. What sorts of scientists would this kind of project attract to Canada?
3. How would it affect Canadian scientists' ability to perform well in other nations? (grant-writing being a very important skill for scientists in most places)
Anyway, all I have are questions at this point. Sounds intriguing, may give it a look when I have some time later.
Reply to Zen Faulkes:
First of all, we are proposing that 50% or so of an agencyâs funds be used for baseline grants. The rest would be traditional competition and industrial (say matching) grants. Any agency could switch from even 15% âsuccessâ rate to 100% independently of their current rate, and trigger higher rates of innovation than they now do, leaving plenty in the pot for bigger projects than baseline grants permit.
In Canada 100% of a grant goes to the researcher, though some university departments âtaxâ grants at about 6% to create a general pool of money (i.e., help people through grant failures, a de facto form of baseline funding). We also generally do not have soft money faculty appointments, i.e., our salaries do not come from grants, and they are 100% time (i.e., include summers). Canadians can also apply to USA agencies for grants, but not vice versa to the best of my knowledge.
Reply to Jason Dick:
1. Please read the paper.
2. Canada offers fewer grant $ per capita than the USA, so those who lead huge projects generally wouldnât stay in or come to Canada. (The traffic is two way, Canada and the USA perhaps being safety valves for one anotherâs malcontents.) I would suggest that baseline funding allows long range planning of research, independent of peer fads. Thatâs what tenure is for, and baseline funding would just back up that impoverished institution.
3. In fields that Iâm familiar with, Canadians, at 10% of the population of the USA, produce > 10% (compared to USA) of the world literature, with average grants almost an order of magnitude less. What we are poor at is commercialization, so that we generally export ideas and import products.
-Dick Gordon, gordonr@cc.umanitoba.ca
I remember that it used to be that universities did not get overhead from NSERC, but I'm way out of touch with the current state of affairs back home.
Universities are allowed to charge overhead on at least some NSERC grants. The overhead rates are much lower than for comparable US universities, but not zero.
Canadians can also apply to USA agencies for grants, but not vice versa to the best of my knowledge.
The programs that I am familiar with restrict salary money to US researchers--not a problem for Canadian faculty apparently, but the restriction also applies to grad students and postdocs except when they are guests at a US institute. Non-US institutes can be collaborators, and their travel to the US can be covered, but the PI must be from a US institute.
In general, the US system is very highly competitive. The programs I am familiar with are typically oversubscribed by factors of between 3:1 and 5:1. The result is that researchers spend lots of time writing multiple proposals in hopes that one gets funded. Having soft money researchers at levels beyond the postdoc doesn't help. But having a way to get small grants funded without excessive worry would probably be a net plus for the system: less time spent writing proposals that will never pay off, and instead spending the time in preliminary projects which can then be developed into fundable proposals. A soft money scientist at a university with modest (by US standards) overhead rates could get 2-3 months of salary and benefits out of a $50k/year grant--compare with the present system, in which (at least for the programs I am familiar with) small program budgets are more like $80-100k/year, and the agencies spend a considerable amount of time and money in bringing these proposals before a competent review panel. I'm not sure if the numbers would work, but it's close enough to not dismiss the idea out of hand.
Thank you for continuing the discussion. I have responded further in my blog post "Economics of peer review at NSERC â a pernicious discussion".
I like the idea a lot. I've written a handful of phase 1 SBIR grants here in the US, and the amount of time spent by me, and the hours spent by the reviewers probably burned up the total value of the grants. (well almost)
I'm not so sure about the grant process in any case. My last attempt at a grant listed a respected nematologist as a collaborator, there was an entire page dedicated to his accomplishments. one of the reviewers didn't seem to care. The primary negative comment from reviewer #2 was "they need a nematologist"
"This makes no sense, as it kills all the truly innovative science by funding it paltrilly, so that a fuckton of boring-ass shit can putter along"
I think that we need to consider the context of the pool of research money in Canada when thinking about this proposal. Nearly all of the NSERC researchers that I know get grants of this size (it was common, even in my old field of physics). When the pool of money is small enough, it can make sense to just give small grants to all qualified researchers rather than slightly larger amounts to a fraction of them.
The Informaworld page indicates the paper was made available under T&F's Open Access option. But it's not possible to download it. What gives?