Proposed changes in the NIH grant process

Effect Measure has a good post about the NIH granting process. I'm not going to rehash what revere said, as far as the description of what happens once a grant application arrives at the NIH and how it winds its way through the Initial Review Group to one of many study sections through programmatic review. In any case, I agree that it's a crappy system--except for all the others. Certainly it has much to recommend it. Junior scientists compete for funds with more senior scientists on a more equal footing than perhaps any other nation in the world, and almost anyone can get a grant if they have compelling evidence and a strong research plan (at least this was the case a few years ago when the paylines weren't so tight). Even better, scientists can request funding for in essence any sort of scientific project. True, the NIH has priorities in what it is looking for and tries to steer funds towards those areas, but still any scientist can propose virtually any sort of health science-related research project and it will be seriously considered for grant funding by a study section composed of experts qualified to evaluate it.

What interested me the most was the proposed change in NIH grant applications described by revere, specifically decreasing the page limit allowed for describing the proposed research from 25 to 15. This was a proposal that I had been unaware of and is apparently part of an effort to decrease the burden of work on those scientists who participate in the NIH study sections. As a still relatively junior faculty member who has participated in two study section meetings thus far, I was truly interested in this proposal. You see, after I got into reviewing the grants for my first meeting, I seriously wondered what on earth I had gotten myself into. Although I'm sure I'll get better at it, the first time around, it took me anywhere from 5-10 hours (or, in one case, even more) to read and review a single grant, and I had eight of them assigned to me, five as primary or secondary reviewer (a full multipage review) and three as discussant (a one page brief review). The study section coordinator was actually going easy on me because it was my first study section; several members had ten to review. Because I was new at this and had only a year ago received my first NIH R01 grant, the importance of what I was doing weighed very heavily on me. My evaluation of the grant would have a large impact on millions of dollars worth of funding and on the careers of eight principle investigators and all the other investigators whose work would be funded (or not funded) by these grants. And because the paylines had fallen so low, I realized that the odds were that only one of these grants would be funded. Consequently, I read each grant twice very carefully before composing my review.

You can see how a shorter grant proposal might be appealing to those sitting on study sections.

Unfortunately, like revere, I'm not sure that this change would do any good. In the two attempts that I made writing an R01 before I finally was funded, the total number of pages in the grant proposal was around 75. The rest was made up of biosketches, budgets, descriptions of facilities, justification for vertebrate animal use, human subjects protection, letters of support, and various other paperwork, all of which a reviewer still has to go through. Of more interest to me is whether cutting the number of pages for the description of the science down to 15 would interfere with my ability to convey my science and research plan adequately.

For me, at least, the answer is probably no. Here's why. In the first five years after I accepted my first faculty position, I ground out between 15 to 20 grant applications to a variety of organizations in a wide variety of formats. In my experience, when I write R01 grants, my tendency is to pad out the text somewhat with detailed descriptions of experimental procedures. Every more senior scientist that I talked to explained to me that, because I was a new investigator who had no track record, reviewers would expect it; yet I found that it detracted from my ability to paint the big picture for them, that I tended to get bogged down in the details, even when describing a five year research project. (In fact, given my tendency towards logorrhea on this blog, I see the same tendency popping up in the longer format.) I've also written grants for the U.S. Army Breast Cancer Research Program. The Army only asks for five pages. That's it. It focuses the mind marvelously, but I would argue that five pages are probably not enough, even for a three year project, which is the duration of the major Army grants, like the Idea Award. Heck, I've even filled out grant applications that are either three, two, or even one page long. I have no idea how a reviewer could possibly evaluate properly research described by only one page (or even three pages), but apparently it's possible. (I just missed getting one of the grants requiring only a two page application.) To me, the sweet spot seems to be between 15-20 pages; so 15 pages wouldn't be that bad. Even so, overall, I'm skeptical that decreasing the page limit would matter that much, as investigators would find ways to get around it; for instance tacking on more publications in the addendum. In any case, it's a suggestion that, if implemented, would probably have little effect other than forcing scientists to focus their ideas more tightly. Indeed, long-winded blogger that I am, I have pointed out on several occasions that for me it's harder to write a tight, focused, short piece than a long rambling piece. The same is true of writing grants.

One other change coming to the NIH review process that revere didn't mention is how study sections are conducted. I happen to be an ad hoc reviewer in a study section that is trying a new way of conducting its thrice-yearly meeting, and we used this new method last month for the very first time. Basically, it's an online meeting format, in which the study section members are expected to post their reviews to a special NIH website by a certain deadline before the study section "meets" virtually and then to log on to read the reviews by other reviewers and participate in an online discussion several times during the period of time of the study section (two days). The format was very much like a typical web-based discussion forum. At the end of that the "meeting" time, we were required to post our scores for each grant, although the NIH still required us to write them down on a form and FAX it to the study section coordinator. The proposed advantages of this system are (1) reviewers don't have to travel; (2) they can supposedly do their review duties from work with less interference with their schedule. Both of these are true, although, because the study section encompassed my normal clinic day, I did have to cancel my clinic anyway, because clinic is to busy to let me spend even a few hours online discussing grants.

Here's my take on this process, now that I've experienced the old way and the new way. The new way is indeed more convenient. Other than some technical glitches in the website, which seemed to have a distressing tendency to go down on the first day, the online review system worked smoothly. I have no doubt that more senior scientists on study sections will find it a boon and make it more likely that they will keep serving on study sections, rather than bowing out as many do as they approach late career. But will it encourage scientists who haven't been on a study section before to agree to serve on a study section?

I have serious doubts about that.

I'll explain. The first problem that I missed immediately was the organization. At the meeting, the chair of the study section goes through each grant on the list one by one and moderates the discussion. In this new format, if there wasn't much detailed discussion of grants that weren't assigned to me, I had a hard time figuring out what the discussion was about, even if I had looked at the abstract of the grant. I suppose that a little self-discipline can help me overcome this problem but it's just not as clean as a real meeting. Another problem is that this new format loses something that's hard to quantify. Scientists serve on study sections for many reasons, but the major reasons that I (and, I daresay, most earlier career scientists who haven't been on study sections before) wanted to be on a study section were (1) to learn the dynamics of what goes on in a study section, so that I know what reviewers are looking for and what constitutes good grantsmanship; (2) to give back to the NIH (in general, you have to be funded by the NIH to be invited to be a reviewer, with few exceptions); and (3) to meet senior scientists and impress them with how brilliant I am. I'm only being partially facetious about (3). In fact, a key benefit of serving on study sections, particularly for early career scientists, is to network, to hang out with senior scientists in a conference room for two days, so that they learn who you are and put a face with your work. The online discussion takes this all away. Each member of the study section becomes nothing but words on a screen. In terms of (1), it's also a bit harder to learn what reviewers look for in a grant, mainly because most of the commentary on the forum seemed to be fairly short.

I have to acknowledge, though, that the very loss of interpersonal dynamics may be an advantage in one sense. Because online each person's input is nothing more than words on a computer screen, it is solely the quality of one's arguments and discussion about the various grants that matters. This is not to be discounted. Strong personalities don't matter nearly as much. No one person can sway the room by force of will. Indeed, I'm a complete natural in such a process, given my more than a decade of experience on Usenet and my nearly two years of blogging. My influence was probably far greater than it would have been in a conventional study section. Even so, something was missing. The human touch, perhaps, but also the opportunity to go out to lunch or dinner with members of the study section and get to know them. You can't get that online, and such networking can pay off in collaborations, invitations to speak, and in so many other ways. In fact, at least one other study section member seemed to echo my thoughts, at least implicitly, because one of the suggestions that was made after the online meeting was finished was that study section members should provide pictures of themselves that could be used as avatars. (He didn't use the word "avatar," but that's basically what he seemed to mean.)

So, in the end, I'm conflicted about this new innovation. I'm sure that the NIH will improve upon it and streamline it as time goes on. It's also clear that the NIH is probably going to move towards online "meetings" for most study sections. I'm just not convinced that the increased convenience and decreased costs (the NIH won't have to pay for travel to and lodging in Washington, D. C. for study section members anymore) will outweigh what is potentially being lost.