It seems that every second blog post written, talk given, or tweet tweeted on the topics of science and research nowadays is in one way or other concerned with the shortcomings of the established peer review process and the ways in which one could evaluate papers more openly and transparently. For now, the discussions usually get stuck at the crossroads between needing to publish in high impact publications and the fact that these are more often than not rather “old-fashioned” when it comes to their peer review practices. But isn’t it in our hands, as scientists, to change this?
Admittedly, the reviews we get are anonymous and often not full of praise to put it mildly – yet that shouldn’t stop a researcher from wanting to share the review with others, provided there is an easy technical solution for doing that. If you think about it, if the review is very (un-)helpful, it’s in everyone’s interest to share it for the sake of exhibiting how (bad) the reviewers of a particular publication are. If the review reflects really well on the submitted paper, then there seems to be a natural impetus for the scientist to share it. And even if the review is very critical of the paper (in which case scientists tend to switch into full-on denial mode), it’s still in your interest to share the review and collect more opinions on the paper in order to understand if there is really a flaw in it, or if the reviewer was just being unprofessionally picky for some reason.
Going somewhat in this direction RePEc have recently issued a call to editors to submit the reviews that were written by the referees of their journals for open display. Even if the idea is very much biased towards whatever the editors deem as appropriate for showcasing, I applaud the effort. However, the scientists themselves should be also more proactive in this area and just publish the reviews that they obtain!
PS. There might be a slight legal caveat to this idea, so I asked an organizer of an event if he would see a problem in me publishing the reviews I got on submitting for presentation. The answer I got was that while it’s something they probably can’t do anything about legally, the reviewers might get “annoyed if what they wrote appeared on the Internet unexpectedly”. Well (especially considering that this would still be done anonymously) I don’t see a reason why they should be, unless they’re ashamed about what they wrote.
As you might be aware of already, the Open Science Summit 2011 is coming up this weekend, with a host of very exciting speakers and surely an abundance of heated discussions on the theme of making science more open. While the range of topics that will be addressed at this year’s event (see program) is quite broad, there is one in particular that I’m especially looking forward to – the introduction of the ‘social’ aspect to science.
What I’m referring to is that even if one sidesteps the issue of open access, there still remains the trend of science being conducted by ‘closed’ entities, be it individuals, research groups or large-scale collaborations. Regardless of how large any such entity might be (some collaborative projects span tens of universities and hundreds of scholars), it is usually the case that research will still be performed from start to finish within the entity itself and without much feedback from the rest of the community. It is only after a corresponding publication will be published that any kind of ‘social’ involvement can begin. But even that is not the biggest issue!
As can be seen from this comparison between blog posts and letters to the editor, there are easy ways of voicing one’s opinion in the science domain, yet, as noted by the author, where blogging (or any other social approach for that matter) falls short is its value in terms of career building and pumping up one’s CV. Now it would be unreasonable to ask of scientists to ignore their careers and devote their lives to science alone, what we need instead is a way of introducing a structure of rewards for social engagement in science. The idea has been quietly gathering momentum in the recent years, as exemplified by Kathleen Fitzpatrick’s recently published book Planned Obsolescence (see this blog post on Inside Higher Ed for a very interesting interview) and a session on microattribution at this year’s Science Online London Conference. If this trend continues at the current rate, we can be optimistic about a scientist’s level of social engagement becoming a valid professional metric in academic institutions rather sooner than later.
All that being said, if you’re attending the event, make sure you enjoy all the talks, but also make sure to take a look at the different poster/app presentations during the breaks which will be running throughout the weekend. If not, you can still follow all the talks and discussions via a live stream that will be available at http://fora.tv/live/open_science/open_science_summit_2011. Hopefully, the organizers will be able to stream some of the off-stage presentations as well, so you might get a glimpse of PaperCritic there. I will post an update on my Twitter account with more specific details if I find out when our presentation will be streamed.
Since the public launch of PaperCritic was announced one and a half months ago, we have been working hard on making the basics of the app as solid as possible. In particular, with the help of some very valuable comments from our first users, we were able to identify and solve the majority of the low-level usability issues of our site, such as being able to access one’s Mendeley library, being able to easily access a list of own reviews, as well as edit and delete these.
As ever, we are not going to sit and wait for something to happen on its own and so will continue improving our app as best we can. Given the number of very encouraging and positive comments about PaperCritic that we received over the better part of the last two months, we would like to take this opportunity to ask our potential users:
Where to next? What features should we add to our site next in order to make it more attractive and usable for everyone?
The following is a list of the features that were already requested (in no particular order). Which of these do you think should be addressed as soon as possible (and which maybe put on hold for now)? Can you think of something essential that is seemingly missing from the list? Do let us know!
- Email subscriptions: Users should be able to subscribe to reviews on papers that interest them (this is already possible via RSS on a paper basis, though not cumulatively). In addition, let users auto-subscribe to the documents from their Mendeley library.
- Pre-publication review: Note that this is actually possible as Mendeley has an “Unpublished work” document type, but it might be useful if unpublished works and their reviews were more explicitly separated from post-publication reviews on the site.
- Trackback system: While we hope that users will take the time to come to PaperCritic to submit their reviews, we also want to be the hub for all mentions about a paper – allowing for bloggers and tweeters out there to send trackbacks to PaperCritic will allow us to aggregate all related content about a paper and become a central repository of public opinion in scientific publishing.
- API: This is probably the most common request for every app nowadays – there should be an API that would allow publication houses, bloggers and others to fetch every review on a given paper via its Mendeley UUID, DOI or similar identificator.
The list can be clearly extended quite a bit, but these seem to be the most relevant points for now. As mentioned above, if you agree/disagree or have alternative suggestions – do let us know! However, I would like to stress out that this post is intentionally only touching on technical features of the site – attracting users and increasing the impact of post-publication reviews will be discussed shortly.