Last week's post on a Peer-Reviewed Research icon has generated a tremendous amount of interest, including many very thoughtful comments and an incisive post over on Cabi Blogs. I'll get to Philip's comments in a moment, because they are at the core of what "peer reviewed" means, but first let me update you on the status of the project.
First off, Kevin Z of the fabulous The Other 95% blog offered the best suggestion for a name for this initiative: PR^3 (Peer-Reviewed Research Reporting). Unfortunately, PR3.org was taken, but fortunately, BPR3.org was not. I decided to risk $9 and registered the domain this morning. For now, our little initiative will be known as BPR^3, Bloggers for Peer-Reviewed Research Reporting. Soon I'd like to move this entire discussion to that web site, but right now, we've got a working blog and discussion engine here, so this will have to do.
I've got some more detailed responses to the many, many suggestions offered by commenters below. I've also got a poll to see who's interested in a design contest, so make sure you read (or skip) all the way to the end.
What, exactly, should the icon signify?
I suggested that at a minimum, the blogger should have carefully read the original research report. Two commenters recommended that we should require thoughtful commentary, not just a link or a reprinting of the abstract. I agree, as long as "thoughtful commentary" includes summary. Two commenters thought that blog authors should have academic credentials, but three felt that credentialing was problematic. As someone who doesn't have a Ph.D., I tend to agree with the no-credentials argument. Also, many qualified bloggers blog anonymously for various reasons, and I don't think we want to exclude them. One commenter suggested linking to the original study, while two others said a full citation should be required. Sounds good to me. One commenter asked how to handle a post that covered both mainstream media reporting and peer-reviewed research. I'd say that would be fine, as long as bloggers make it clear in their post when they are talking about peer reviewed research and when they aren't. One commenter suggested that bloggers should have to register on the main website in order to use the icon.
How do we define "peer review?"
Todd O. said that his library has a service where users can look up journals on a list to find out. That sounds good -- can anyone give us a link to such a resource? Another way of handling this, suggested by several commenters, is to define peer review very loosely, then allow users to dispute particular sources on a case-by-case basis. That sounds good to me. Anyone want to give a shot at this sort of "loose" definition?
What about copyright for the icon?
Not a lot of commentary here, but one commenter suggested a license, to allow for policing of its use. I tend to prefer the public domain. That way if the organization doesn't hold together for whatever reason, the resources it has created are freely available for anyone to use. But I'm happy to hear arguments either way.
How should we design the icon?
Three commenters liked the idea of a contest. That sounds good to me, too, as long as we get plenty of submissions. So let's take a little poll to see who's interested.
There were several suggestions on the design as well. One commenter suggested a lasso, while another suggested a magnifying glass. One said the font should be bigger than the one on CogDaily.
Finally, several commenters addressed issues I hadn't thought of yet. There was a lot of interest in some kind of aggregator that would allow readers to spot all blogs sporting the badge. I agree -- although I think RSS would probably work better than trackback for this purpose, given how polluted the trackback system is with spam. If bloggers registered on the central web site, I would think it shouldn't be too hard to create a central RSS feed with only the peer-reviewed posts (does anyone know how to do this?).
Two commenters suggested that all this was already handled by Postgenomic. My problem with Postgenomic is quite simple. I've never gotten the site to work on my Mac. Is it PC-only?
Finally, there were a couple of comments (and a blog post) questioning the validity of linking only to peer-reviewed articles. Aren't there other useful sources for information -- books, "grey literature," and the like? Well, of course there are, but there are also plenty of sources of misinformation. While peer review isn't perfect, I do believe it's a good way to distinguish between useful information and misinformation. Sure, sometimes useful information doesn't get published that way, and sometimes misinformation sneaks in -- and bloggers can play an important role in explaining the difference -- but I still think it's valuable to identify blog posts that are talking about peer-reviewed literature.
A much more pervasive problem, I think, is scholarship by press release, or by talking loudly. Too many blogs amplify the problem by mindlessly quoting these news releases without checking to see whether other scholars in the field believe the research is well-executed. That's why I think BPR^3 is a worthwhile initiative. If someone else wants to create a "grey literature" icon, or any other icon they want, they're welcome to try. And no one is suggesting that bloggers write exclusively about peer-reviewed research -- we'd just like to make it clear when we are (or are not) talking about peer-reviewed work.
- Log in to post comments
For aggregation a few of methods come to mind. One would be to use something like FeedWordPress which can automatically scan feeds and produce an excerpt for linking. The problem with that is that it would aggregate all entries in a feed so bloggers would need to provide a BPR^3 only feed which may not be easy for some bloggers. - Planet something similar, but I've never got to grips with Planet.
Another method would be to produce a group blog, with members listed as authors either set up as a stand-alone blog, or else on Blogger or Wordpress.com. When blogger writes an entry they then write a brief summary for the central site.
Or you could skip the aggregator engine on the website and use Pipes. A pipe could scan a feed for the code for the logo image source name like "brp3.png" and, if it finds it, list it in the aggregated feed. Pipes is open, so the code can be cloned by anyone.
...and finally I've just remembered I wrote a post Build your own web aggregator... If bloggers tagged their posts "brp3" then a technorati tag and tumblr could be used to set up an aggregator.
Aggregation can be done in one of two ways, as I see it. Either the BPR3 site scrapes registered sites or registered sites push data to BPR3.
In the first method, BPR3 would need to continously scrape registered sites and their RSS feeds, sifting out posts that use the icon. This means that updates come at pre-determined intervals (likely a few times a day). It would also require bloggers to include the icon in their RSS summary, or require some tagging scheme. This method is the most bandwidth hungry, for both blogger and BPR3. And it will not scale well if BPR3 ever expands significantly.
The other solution is to use Trackbacks and would be the simplest and most non-invasive. Trackback is a standard feature on most blog software these days and specifically designed to push data to another site.
Moderating this stream of trackback should be fairly simple. Trackback permalink domain is compared to the domain sending the trackback POST. If they match (ie. Trackback is from the source, rules out masquerading) it is then validated against the database of registered domains and posted or discarded accordingly. If you wanted to get really fancy, each blogger could be given an automatically generated trackback URL that includes a unique key which they post to, but I think this would be unescessary.
Regarding copyright, I don't think this needs to be a concern yet. I don't see anyone "abusing" the icon. The icon doesn't symbolize a large publishing organization or something that can bestow legitamecy to a poster. It is merely a group of bloggers that want a means to highlight other good blog posts. Perhaps in the future it may become a problem but now I dont see it being an issue.
The interesting discussion on CABI blogs about the value of scientific research that is not peer-reviewed (NGO, government, etc) raises an interesting point.
Peer-review is one widely-accepted standard for evaluating that research meets at least minimal standards. The proliferation of non-reviewed journals and other sources makes it hard for readers to know the value of sources.
This is not to say that no other research has high quality. It is a standard that is applied in just about all academic disciplines, so the logo / program has the potential to draw together academics of several stripes. It recognizes a characteristic that is easily verifiable.
It would be good if, after this gets rolling, it sparked ideas about how to critically evaluate and mark high quality research from other sources.
Will you allow Universities put the icon on press releases?
Aluns point about using a blog is great idea. Set up a blog at bpr3.org. All members can use the icon and have rights to post the first paragraph, or a 200 word summary linking to their blog, with keywords so people can search the blog for their favorite topic. The added bonus of using a template like Blogger or Wordpress is they create the RSS feed for you, so people can subscribe to it very easily. I believe any blog can use the free feedburner service too.
I agree with Zachary that the icon will not likely be abused. It can be one of those "we'll cross that road when get there" situations. Bloggers will be policing the site anyways by posting to it and reading entries from it. All one needs to do if there is inappropriate content is to email the administrator for review and possible removal. Plus, if we are defining peer-review loosely than it becomes even less of a problem.
I think peer-review should be defined as from any source subject to the scrutiny of another scientist (including social scientists). This may include government reports that have well-known scientists on their panels or committees, journals (no matter how obscure), or any other source where results are given for some sort of observations or analysis. Like I've mentioned before. 90% or more of the time, bloggers are going to posting on well-defined peer-reviewed research. Todd O. or someone with similar resources could put together a master list of peer-reviewed journals by discipline. This would be a good resource in and of itself.
I think its OK to quote from press releases when they interview a study's authors. Then you can get direct quotes from the horses mouth. Of course rehashing a press release is bad journalism no matter the discipline.
Herb, I think most universities will not be interested in having BPR3 badge on their press releases. But you bring up an interesting point on what the difference between a press release and a blog post is when highlighting a certain study.
In my opinion, press releases do not offer an analysis of the study, but seek to point others to the research and provide a public relations mechanism. Press releases are mostly geared towards research published in high impact journals to boost the image of the university. For instance my species descriptions in obscure taxonomic journals will never be heralded by Penn State or even grace the departmental website. When I received a $4000 grant to study anemone systematics from the Census of Marine Life (as a graduate student mind you and sole author/PI), there was no hoopla or fanfare or congratulations from the department head.
Now a blog post will highlight this type of research, potentially. (my new species will be blogged at my site when the study is published... now in review). Most of the papers I blog about have impact factors less then 3 or 4.
Any other thought on this?
Around Christmas time last year I attempted something similar to this concentrating on space policy journal articles at http://polispacepeerreview.wordpress.com/
Unfortunately I ran out of steam after 2 months as my studies got in the way, especially as a good review took me 2-3 hours to write. Here are some of my thoughts about my experience.
1) A good peer-review takes time. Too many bloggers just put up a link to an article and 2 or 3 sentences on their general opinion of the article. While I thank these bloggers for bring stuff to my attention, this isn't enough usually to convince me to read the article. On the flip side, writing too much is a problem I had. Bottom line, there needs to be some sort of feedback put back into the process for the icon to mean anything. (quality control)
2) Categorization/keywords are important. As a blogger, this is probably obvious but this is overlooked by many people. A neat feature to add would be rss feeds by category on BPR3. i.e. I could subscribe to the Evolutionary Biology or High-Energy Physics feed
3) Citation is important. During the time I ran my blog I got a lot of comments via email about citation. Many people liked the fact that the first paragraph of any review contained information on how to get to the journal (important for obscure journals) and a little bit of info about the author. I also included the full citation in the title of the blog post. This was very cumbersome and awkward. I would suggest instead of creating some type of standard in providing a citation at the opening of the review if possible. If you standardize this, it could be very beneficial for any type of aggregator you create for BPR3.org
4) I'd like to second Kevin's take on opening up the definition of peer-reviewed a bit. Many grey lit areas are really heavily peer reviewed and scientifically significant. I am think specifically of items such as National Academy reports or government studies.
Just my $0.02
I don't think it matters whether the original source was peer-reviewed. It is easy to blog ignorantly about peer-reviewed research.
The quality of the original resarch and whether it was peer-reviewed and by who is irrelevant because that will be filtered through the non-peer-reviewed analysis of the blogger.
The icon will be attached to blogposts not papers, therefore, the icon should signify the quality of the blogpost. The icon as currently described signifies nothing about the credentials of the blogger nor about the quality of the blogpost. I don't think it is sensible to put a "peer-reviewed research" icon on non-peer-reviewed blogposts.
About the aggregation, I think it would be best to talk to the developers of Postgenomic to get the problems you mentioned solved. Aggregated blog comments on published works can serve as an additional metric to evaluate the interest and quality of the work. This is why it would be better to concentrate all the efforts into one single place instead of scattering this further.