Monthly Archives: May 2012

Want to write a review? Pick your format of choice!

Last week, I summarized some of the main issues with the recent approaches to post-publication peer review (PPPR), such as comments and different types of altmetrics. The conclusion was that, while these approaches have a certain amount of potential, they usually fail to reach a high quality level, predominantly due to their very short and unstructured nature.

As mentioned in the previous post, most researchers will be writing summaries and reviews of research articles on a regular basis, yet these will usually be kept private. And even if one were to find an incentive that would drive researchers to publish these reviews, they would in most cases be of little value to others. The reason for this is that every researcher has his own particular focus of interest and will write his review accordingly, meaning that there will be a lot of conflict between researchers, particularly ones coming from different disciplines. The solution to the problem, in my opinion, lies in making reviews more systematic even at the personal level.

The need for systematic reviews has so far been relatively well addressed only in the health care section, with the establishment of the Cochrane Collaboration and Cochrane Reviews in the 1990’s. Unfortunately, other disciplines have so far failed in developing a similarly structured format for reviewing its advances. However, I would not necessarily see this as a failure of the disciplines, but as an indication that it can be simply infeasible to agree on any one particular review format for a specific research area.

In fact, there is no reason why smaller fields of study should not be given the opportunity to develop their own review formats that would have a higher chance of being adopted by its scholars because they would provide more value to both review writers and readers, than any generalized review format could ever do (from this perspective, a free-text review is basically as generalized a format as it gets).

In order to facilitate the above, PaperCritic will be rolling out a review template system this Summer that will enable its users to collaboratively define review formats that are most meaningful for their particular field of study. Furthermore, we will also be launching both public and private interest groups, aimed at bringing together scholars from one disciplines and sharing their views on the latest research in the field.

Ultimately, the hope is that having more structured reviews will make it significantly easier for scientists to both produce and consume these, thus making it a much more enjoyable process. Additionally, having a set of established review templates available in one’s field might just provide the needed encouragement for researchers to publish their reviews by alleviating the fear of not having composed the review well enough.

As ever, comments and suggestions are more than welcome as this is really a proposal and work in progress. Other than that, we’re looking forward to bringing the review templating system to you as soon as possible, so keep following us here and on Twitter.

Advertisements

3 Comments

Filed under PaperCritic

Where is all the quality post-publication peer review?

Post-publication peer review (PPPR) is something that we all would like to flourish, yet at the same time seem to struggle to find a proper format for. Arguably the most accessible and widely accepted form of PPPR is writing comments in the form of letters, or directly on a publisher’s website. However, Kent Anderson has recently discussed the problems associated with comments being used for PPPR, the main issue being a lack of substance and quality in such ´reviews´.

In search of new forms of PPPR, a new movement called ´altmetrics´ has established itself recently as a potential front-runner, basically switching the focus from traditional full-text reviews to widespread usage-based metrics, ranging from the more traditional citation counts, to more recent inventions, such as tweets, Facebook likes, etc. Unfortunately though, these metrics suffer from several drawbacks themselves, as discussed by David Crotty in one of his recent posts. Not surprisingly, the main issue identified was that these metrics are extremely hard to interpret, especially since one needn’t be an expert in the field (or even know what an article is about) to retweet its link to others for example.

The big question that remains then is: Does a form of PPPR exist that can provide a sufficient amount of substance, not be too demanding for the reviewer and in fact provide an incentive for the reviewer in itself? To answer this question, I suggest that we stop looking at public metrics for just a second, and focus on the reviewer. This person will in most cases will be a (semi-) professional scientist, meaning that the person will be spending a lot of time reading and evaluating the work of others, looking for gaps that could be addressed in future research.

In effect, every researcher spends half the time writing personal comments and reviews on published books and articles, which may partially end up in the “Introduction”, “State of the art” and “Literature review” sections of the researcher’s articles, books, dissertations, etc. There is already an abundance of quality PPPR out there, our task is to encourage researchers to make (some of) this material public, by providing the right tools and formats.

How can we do this? The short answer is: by providing even more value to the researcher. I’m still mulling over the details somewhat, so I’ll put off the full proposal for a follow up post which should appear later this week. I’d be happy to hear what your thoughts are so far, though I guess the solution is the most interesting aspect to all this.

4 Comments

Filed under PaperCritic