A proactive approach to making peer review more transparent

It seems that every second blog post written, talk given, or tweet tweeted on the topics of science and research nowadays is in one way or other concerned with the shortcomings of the established peer review process and the ways in which one could evaluate papers more openly and transparently. For now, the discussions usually get stuck at the crossroads between needing to publish in high impact publications and the fact that these are more often than not rather “old-fashioned” when it comes to their peer review practices. But isn’t it in our hands, as scientists, to change this?

Admittedly, the reviews we get are anonymous and often not full of praise to put it mildly – yet that shouldn’t stop a researcher from wanting to share the review with others, provided there is an easy technical solution for doing that. If you think about it, if the review is very (un-)helpful, it’s in everyone’s interest to share it for the sake of exhibiting how (bad) the reviewers of a particular publication are. If the review reflects really well on the submitted paper, then there seems to be a natural impetus for the scientist to share it. And even if the review is very critical of the paper (in which case scientists tend to switch into full-on denial mode), it’s still in your interest to share the review and collect more opinions on the paper in order to understand if there is really a flaw in it, or if the reviewer was just being unprofessionally picky for some reason.

Going somewhat in this direction RePEc have recently issued a call to editors to submit the reviews that were written by the referees of their journals for open display. Even if the idea is very much biased towards whatever the editors deem as appropriate for showcasing, I applaud the effort. However, the scientists themselves should be also more proactive in this area and just publish the reviews that they obtain!

PS. There might be a slight legal caveat to this idea, so I asked an organizer of an event if he would see a problem in me publishing the reviews I got on submitting for presentation. The answer I got was that while it’s something they probably can’t do anything about legally, the reviewers might get “annoyed if what they wrote appeared on the Internet unexpectedly”. Well (especially considering that this would still be done anonymously) I don’t see a reason why they should be, unless they’re ashamed about what they wrote.

Advertisements

7 Comments

Filed under PaperCritic

Looking forward to the Open Science Summit 2011

As you might be aware of already, the Open Science Summit 2011 is coming up this weekend, with a host of very exciting speakers and surely an abundance of heated discussions on the theme of making science more open. While the range of topics that will be addressed at this year’s event (see program) is quite broad, there is one in particular that I’m especially looking forward to – the introduction of the ‘social’ aspect to science.

What I’m referring to is that even if one sidesteps the issue of open access, there still remains the trend of science being conducted by ‘closed’ entities, be it individuals, research groups or large-scale collaborations. Regardless of how large any such entity might be (some collaborative projects span tens of universities and hundreds of scholars), it is usually the case that research will still be performed from start to finish within the entity itself and without much feedback from the rest of the community. It is only after a corresponding publication will be published that any kind of ‘social’ involvement can begin. But even that is not the biggest issue!

As can be seen from this comparison between blog posts and letters to the editor, there are easy ways of voicing one’s opinion in the science domain, yet, as noted by the author, where blogging (or any other social approach for that matter) falls short is its value in terms of career building and pumping up one’s CV. Now it would be unreasonable to ask of scientists to ignore their careers and devote their lives to science alone, what we need instead is a way of introducing a structure of rewards for social engagement in science. The idea has been quietly gathering momentum in the recent years, as exemplified by Kathleen Fitzpatrick’s recently published book Planned Obsolescence (see this blog post on Inside Higher Ed for a very interesting interview) and a session on microattribution at this year’s Science Online London Conference. If this trend continues at the current rate, we can be optimistic about a scientist’s level of social engagement becoming a valid professional metric in academic institutions rather sooner than later.

All that being said, if you’re attending the event, make sure you enjoy all the talks, but also make sure to take a look at the different poster/app presentations during the breaks which will be running throughout the weekend. If not, you can still follow all the talks and discussions via a live stream that will be available at http://fora.tv/live/open_science/open_science_summit_2011. Hopefully, the organizers will be able to stream some of the off-stage presentations as well, so you might get a glimpse of PaperCritic there. I will post an update on my Twitter account with more specific details if I find out when our presentation will be streamed.

Leave a comment

Filed under PaperCritic

PaperCritic basics in place – Where to next?

Since the public launch of PaperCritic was announced one and a half months ago, we have been working hard on making the basics of the app as solid as possible. In particular, with the help of some very valuable comments from our first users, we were able to identify and solve the majority of the low-level usability issues of our site, such as being able to access one’s Mendeley library, being able to easily access a list of own reviews, as well as edit and delete these.

As ever, we are not going to sit and wait for something to happen on its own and so will continue improving our app as best we can. Given the number of very encouraging and positive comments about PaperCritic that we received over the better part of the last two months, we would like to take this opportunity to ask our potential users:
Where to next? What features should we add to our site next in order to make it more attractive and usable for everyone?

The following is a list of the features that were already requested (in no particular order). Which of these do you think should be addressed as soon as possible (and which maybe put on hold for now)? Can you think of something essential that is seemingly missing from the list? Do let us know!

  • Email subscriptions: Users should be able to subscribe to reviews on papers that interest them (this is already possible via RSS on a paper basis, though not cumulatively). In addition, let users auto-subscribe to the documents from their Mendeley library.
  • Pre-publication review: Note that this is actually possible as Mendeley has an “Unpublished work” document type, but it might be useful if unpublished works and their reviews were more explicitly separated from post-publication reviews on the site.
  • Trackback system: While we hope that users will take the time to come to PaperCritic to submit their reviews, we also want to be the hub for all mentions about a paper – allowing for bloggers and tweeters out there to send trackbacks to PaperCritic will allow us to aggregate all related content about a paper and become a central repository of public opinion in scientific publishing.
  • API: This is probably the most common request for every app nowadays – there should be an API that would allow publication houses, bloggers and others to fetch every review on a given paper via its Mendeley UUID, DOI or similar identificator.

The list can be clearly extended quite a bit, but these seem to be the most relevant points for now. As mentioned above, if you agree/disagree or have alternative suggestions – do let us know! However, I would like to stress out that this post is intentionally only touching on technical features of the site – attracting users and increasing the impact of post-publication reviews will be discussed shortly.

8 Comments

Filed under PaperCritic

Introducing PaperCritic – an open publication review tool powered by Mendeley API

We live in a world where our lives are broadcast by Facebook and Twitter, our news consumption is dominated by blogs and our knowledge is defined by Wikipedia articles. Yet somehow, science, which should really be at the front of any kind of advances, remains 20 years adrift in terms of the amount of collaboration in its echelons, not to mention when it comes to opening up to the broader public.

In fact, right now, the only acceptable way of presenting one’s academic work and obtaining critical reviews for it is through the tedious and obscure submit-and-wait process. We at PaperCritic find this way of promoting science severely outdated and simply unacceptable.

Thankfully, ever since Mendeley, the biggest player in the publication management market, has opened the door to its resources with an Open API, it has become possible to construct simple and powerful apps that leverage the Mendeley database in order to promote science and make it more collaborative.

Having been made aware of this, as well as the Mendeley/PLoS API Binary Battle, we have quickly put our hands to work and came up with PaperCritic – an app that offer researchers a way of obtaining feedback for their scientific work, and everybody interested – a way of providing it, in a fully open and transparent environment.

Apart from helping the scientific community, PaperCritic also helps you as a researcher or a science enthusiast organize your publication library even better. You’ve surely already embraced all the neat things Mendeley offers, such as tags, summaries and in-text notes. But what you want is to rate the different aspects of publications, or to write critical reviews of your own. Well, using PaperCritic you can now rate and review any publication just like the big guns at IEEE!

Disclaimer: While PaperCritic is powered by the Mendeley API, the two services are in no way affiliated to one another.

3 Comments

Filed under PaperCritic

Every Blog should have a right to an API!

A couple of days ago, a well known online information warehouse called Fluidinfo (formerly known as FluiDB) has announced that they have created an API for BoingBoing, a popular mainstream blog. Now any common Web user reading this piece of news wouldn’t make much of it, considering there’s not a lot he can do with a bunch of XML-formatted stuff that can be fetched over something called an API. And no one could blame him.

There are, however, people who should be getting particularly excited about this announcement. People who are involved with any kind of data-driven applications, be it personalized news services, recommender tools, semantic apps. Up until now, developers in these areas were severely limited by the unstructured (read: “bloody messed up”) nature of HTML when trying to access one of the richest sources of personal information online – the blogs. Forget Twitter, Facebook and all the rest, if you want to learn something deep and insightful about human opinion on some topic, you need to analyze what they write in their blogs. And up until now, this has been a serious pain in the you know what due to a lack of any structured representation in the blogosphere.

Now, if the news by Fluidinfo can be any indication of things to come, this might just be about to change. While it is still silly to talk about a revolution on the blogosphere, it can take just a number of popular mainstream blogs like ReadWriteWeb, GigaOM, TechCrunch etc. to adopt the offer of Fluidinfo for many others to follow. Because who would want to be left out of the new and sexy apps built on top of a clean API interface where users could be presented with data in the most innovative of ways? All I know is that I wouldn’t! And hopefully sooner or later most blog authors will realize that they’re being left out of something because their, admittedly public, archives are not usable by the newest apps out there.

And so the revolution begins.

4 Comments

Filed under Storyflow

Topify is back with a new team!

It is my pleasure to announce that after almost one and a half years in the wilderness, induced by me finishing my studies and moving to a new country, and my designer fellow opting to take up his own firm full time, Topify is finally back on track. As I have no expertise, skill, or talent in either design or serious marketing, I was reluctant to push this project on without a proper partner on my side. But alas, I have found one in the form of JoyGroup, a small but rapidly growing and ambitious web agency from Almere, Netherlands.

The structure of the project has also slightly changed. We have decided to use the name Topify for our main topic-based aggregation/analysis service, and make it a platform that will be reusable in many different ways. Of course, the first product that we will be launching based on Topify will be a blog-searching service that was the initial goal of mine, but further developments will include a full API, with a bunch of services around it, including a host of Drupal modules!

Watch this space for more announcements in the very near future.

Leave a comment

Filed under Storyflow

Blog search broken? DEAD?! Let’s just call it “hybernating”.

Over the last year, there has been a flurry of stories on the blogosphere about the about the inability of big search players and small startups to create a viable blog search engine. It all gradually started with the ubiquitous niggle at Technorati, the “leading” player in the blog search market, a company that probably spends more time at thinking about the perfect ad placement coordinates than spam reduction and search relevance.. hmm, well, you see what I mean by “ubiquitous niggle”.

But then I just thought it was all meant to encourage Technorati, as we all have this innate need to see Google beaten at any race they’re in, even though we actually just need that to happen to be able to tell ourselves that Google is not the only option and we’re using it at free will (sorry, drifting offtopic..). Anyway, that seemed not to be the case, as the will to see Technorati improve slightly went over into a general sense of frustration, capped off in August 2008 with Mashable’s supposition that Blog Search is Broken.

The main notion in that post was that Technorati has the most potential, but just doesn’t seem to be getting any love from the users, while the newer options like MyBlogLog, BlogCatalog or Wikio are just trying to put collections of blogs and/or blog posts together in a directory-like manner, without offering much of the search dimension.

Since then, at least one startup has sprung to life which seems to be getting a lot of acclaim, namely Twingly. But while the company is doing really good progress, it doesn’t seem to have established itself as a major player yet, and thus we come to March 2009, where, as the legend goes, blog search has finally died. At least according to The Blog Herald’s latest analysis, according to which there is too much spam and irrelevant stuff whenever it comes to blog searching.

Now they are probably completely right on the problem aspect, i.e. there is really a lot of spam-blogs out there, as well as feeds that aren’t even blog feeds or some company’s great idea to index the whole page of a blog post, without even trying to cut out the actual text from it (Go*cough*ogle). As for the solutions, there is much more that needs to be done but also much reason to be optimistic even of the very near future to come up with a decent solution.

First, the problem or spam-blogs, feeds that don’t point to blogs and other basic irrelevancy. Think of just one approach here, the Wikipedia-approach, and you will be rewarded aplenty! Yes, spammers will try to add their spam-blogs, while others will be accidentally saved by the crawlers, but there are users to help us out there. A simple combination of user-voting and admin-monitoring can do wonders here in my opinion, there just needs to be a good base of blogs to start with to attract the crowd and then the snowball will be on its’ way.

A much bigger problem is that blog search engines are still focusing on the old approaches of link-based post ranking, tagging and also the new approach of user-voting. As I pointed out in previous posts, the first is just a tonic for top 100 blogs, the second highly misleading and the third just not applicable for a pure search site (it surely is for a news aggregator or such). What is amiss here are semantic technologies that will mine deeper into the meaning of the blog posts and provide the user with highly relevant stories based on their content and not some keyword-link rank.

Thus, to revive blog search, we have to see that it’s different from the initial Web problem of finding any relevant content in a haystack, it’s about finding the one that makes the most sense. Luckily for all concerned, there are many semantic startups launching and getting good support around the scene and it can’t be long before they spread over to such tasks as blog search. When that day comes (i.e. I stop talking and finally deliver thing thingy called Topify), Jonathan, you will be one of the first to get an invite and have your faith in blog search restored :)

3 Comments

Filed under Storyflow