I posted a link last week to a dialogue written by a former colleague of mine, Sylvain Ribault. Sylvain’s dialogue is a summary of different perspectives on academic publishing. Unlike certain more famous dialogues written by physicists, Sylvain’s account doesn’t have a clear bias: he’s trying to set out the concerns different stakeholders might have and highlight the history of the subject, without endorsing one particular approach as the right one.
The purpose of such a dialogue is to provoke thought, and true to its purpose, the dialogue got me thinking.
Why do peer review? Why do we ask three or so people to read every paper, comment on it, and decide whether it should be published? While one can list many reasons, they seem to fall into two broad groups:
- We want to distinguish better science from worse science. We want to reward the better scientists with jobs and grants and tenure. To measure whether scientists are better, we want to see whether they publish more often in the better journals. We then apply those measures on up the chain, funding universities more when they have better scientists, and supporting grant programs that bring about better science.
- We want published science to be true. We want to make sure that when a paper is published that the result is actually genuine, free both from deception and from mistakes. We want journalists and the public to know which scientific results are valid, and we want scientists to know what results they can base their own research on.
The first set of goals is a product of scarcity. If we could pay every scientist and fund every scientific project with no cost, we wouldn’t need to worry so much about better and worse science. We’d fund it all and see what happens. The second set of goals is more universal: the whole point of science is to find out the truth, and we want a process that helps to achieve that.
My approach to science is to break problems down. What happens if we had only the second set of concerns, and not the first?
Well, what happens to hobbyists?
I’ve called hobby communities a kind of “post-scarcity academia”. Hobbyists aren’t trying to get jobs doing their hobby or get grants to fund it. They have their day jobs, and research their hobby as a pure passion project. There isn’t much need to rank which hobbyists are “better” than others, but they typically do care about whether what they write is true. So what happens when it’s not?
Sometimes, not much.
My main hobby community was Dungeons and Dragons. In a game with over 50 optional rulebooks covering multiple partially compatible-editions, there were frequent arguments about what the rules actually meant. Some were truly matters of opinion, but some were true misunderstandings, situations where many people thought a rule worked a certain way until they heard the right explanation.
One such rule regarded a certain type of creature called a Warbeast. Warbeasts, like Tolkien’s Oliphaunts, are “upgraded” versions of more normal wild animals, bred and trained for war. There were rules to train a Warbeast, and people interpreted these rules differently: some thought you could find an animal in the wild and train it to become a Warbeast, others thought the rules were for training a creature that was already a Warbeast to fight.
I supported the second interpretation: you can train an existing Warbeast, you can’t train a wild animal to make it into a Warbeast. As such, keep in mind, I’m biased. But every time I explained the reasoning (pointing out that the text was written in the context of an earlier version of the game, and how the numbers in it matched up with that version), people usually agreed with me. And yet, I kept seeing people use the other interpretation. New players would come in asking how to play the game, and get advised to go train wild animals to make them into Warbeasts.
Ok, so suppose the Dungeons and Dragons community had a peer review process. Would that change anything?
Not really! The wrong interpretation was popular. If whoever first proposed it got three random referees, there’s a decent chance none of them would spot the problem. In good science, sometimes the problems with an idea are quite subtle. Referees will spot obvious issues (and not even all of those!), but only the most diligent review (which sometimes happens in mathematics, and pretty much nowhere else) can spot subtle flaws in an argument. For an experiment, you sometimes need more than that: not just a review, but an actual replication.
What would have helped the Dungeons and Dragons community? Not peer review, but citations.
Suppose that, every time someone suggested you could train a wild animal to make it a Warbeast, they had to link to the first post suggesting you could do this. Then I could go to that first post, and try to convince the author that my interpretation was correct. If I succeeded, the author could correct their post, and then every time someone followed one of these citation links it would tell them the claim was wrong.
Academic citations don’t quite work like this. But the idea is out there. People have suggested letting anyone who wants to review a paper, and publishing the reviews next to the piece like comments on a blog post. Sylvain’s dialogue mentions a setup like this, and some of the risks involved.
Still, a setup like that would have gone a long way towards solving the problem for the Dungeons and Dragons community. It has me thinking that something like that is worth exploring.
4 gravitons 
Ok, nothing of substance to say. But shout-out to all the D&D players out there!
Cheers,
Guy
Ps. Been reading for years and years. Thanx for all the writing.
LikeLiked by 1 person
It would be interesting to elaborate on how your suggestion could work in practice. If I understand correctly you propose to formalize the peer review work that is done by researchers who rely on an article, taking a citation as a sign that some peer review was performed. Moreover, there is the idea that citing an article means taking some responsibility for its correctness, and committing to draw consequences from future corrections. We might add the possibility of “adversarial citations”, citing an article while denouncing its errors.
The question is: which institution or platform would formalize this? Would we have some sort of preprint server, with extra information on the peer review status of articles, and on the meaning of citations?
I am not sure that this approach would imply renouncing the “first set of concerns”: if citations were fewer and more meaningful, as would have to happen under your scenario, they might become a reasonable basis for research assessment.
LikeLike
Ok, semi-spitballing here:
I was actually thinking of something more like spontaneous review rather than citations, where the reviews sit like “comments” below the original article in the same way that threads sit below the original post in a php forum.
I do like the idea of just having citations themselves serve this purpose, especially for fields where citing a paper really ought to imply reproducing some of that paper’s results. But I think the “activation barrier” is a bit too high currently, as publishing a paper just to argue someone is wrong is something people usually do only when they’re especially passionate about the point/when it’s leading to widespread misunderstandings. Something more comment-like is easier for people to do and thus people would be more comfortable pointing out that something is wrong, even if it’s “minor”. But making it easier and more normal to post short/”trivial” papers would also solve this issue.
In that kind of world, yeah, you’d want something like adversarial citations. I don’t think you’d want to implement them like “negative citations” or “downvotes”. Instead, the idea could be that if you think a paper is wrong then you publish an argument to that effect, with the result that the paper gets flagged as disputed. Ideally that flag would “propagate up” citations, so that every paper citing the disputed paper would also be flagged. Then the original author could either convince the critic that their paper is actually fine, at which point they’d remove the disputed flag and flag their own comment, or be convinced in turn that their paper is actually wrong, at which point the flag goes from disputed to confirmed incorrect.
This of course only works in a field where people are going to be honest and civilized about these things! I think some areas of math and physics could manage it though. Biology would probably have to wait until genuine post-scarcity conditions.
It would also require more granularity of citations. People would need to be able to have different types of citations for things their paper logically depends on vs. “further/historical reading”, and ideally would be able to cite specific parts of papers if they only need a particular point but don’t care if the rest of the paper is correct.
And yeah, I could imagine this all being kept track of on something like a preprint server. I hesitate to say it should be exactly a preprint server, because hosting papers could well be taken care of in a more distributed way, while just the ratings/logical dependencies are indexed centrally. But some sort of giant imaginary web interface yeah.
In such a system you could have fewer and more meaningful citations, especially if you restricted them to the “logical dependency” type. But while I think that could give you a “research productivity” number if you wanted one, it wouldn’t be a particularly good one. It wouldn’t take into account the thing today’s journals seem to care about, where the journal asks reviewers not merely to check if a paper is correct but to check whether it matters according to some (vague, journal-varying) criteria. It certainly doesn’t implement whatever countries are trying to do when they fund science, where they care about the potential of the research to solve problems in the real world and not merely how many scientists end up using it (how do you actually implement something like that? I’m skeptical that there is any smart way, but the standard smart person answer is probably prediction markets). And under scarcity it still means that citations are going to be doing something other than verifying correctness. In particular, it encourages people to write papers that can plausibly be described as logically dependent on their friends’ papers, which if anything would decrease peoples’ incentive to try new things and would gradually water down the intended meaning of the citations.
LikeLike