A Tale of Two Archives

When it comes to articles about theoretical physics, I have a pet peeve, one made all the more annoying by the fact that it appears even in pieces that are otherwise well written. It involves the following disclaimer:

“This article has not been peer-reviewed.”

Here’s the thing: if you’re dealing with experiments, peer review is very important. Plenty of experiments have subtle problems with their methods, enough that it’s important to have a group of experts who can check them. In experimental fields, you really shouldn’t trust things that haven’t been through a journal yet: there’s just a lot that can go wrong.

In theoretical physics, though, peer review is important for different reasons. Most papers are mathematically rigorous enough that they’re not going to be wrong per se, and most of the ways they could be wrong won’t be caught by peer review. While peer review sometimes does catch mistakes, much more often it’s about assessing the significance of a result. Peer review determines whether a result gets into a prestigious journal or a less prestigious one, which in turn matters for job and grant applications.

As such, it doesn’t really make sense for a journalist to point out that a theoretical physics paper hasn’t been peer reviewed yet. If you think it’s important enough to write an article about, then you’ve already decided it’s significant: peer review wasn’t going to tell you anything else.

We physicists post our papers to arXiv, a free-to-access paper repository, before submitting them to journals. While arXiv does have some moderation, it’s not much: pretty much anyone in the field can post whatever they want.

This leaves a lot of people confused. In that sort of system, how do we know which papers to trust?

Let’s compare to another archive: Archive of Our Own, or AO3 for short.

Unlike arXiv, AO3 hosts not physics, but fanfiction. However, like arXiv it’s quite lightly moderated and free to access. On arXiv you want papers you can trust, on AO3 you want stories you enjoy. In each case, if anyone can post, how do you find them?

The first step is filtering. AO3 and arXiv both have systems of tags and subject headings. The headings on arXiv are simpler and more heavily moderated than those on AO3, but they both serve the purpose of letting people filter out the subjects, whether scientific or fictional, that they find interesting. If you’re interested in astrophysics, try astro-ph on arXiv. If you want Harry Potter fanfiction, try the “Harry Potter – J.K. Rowling” tag on AO3.

Beyond that, it helps to pay attention to authors. When an author has written something you like, it’s worth it not only to keep up with other things they write, but to see which other authors they like and pay attention to them as well. That’s true whether the author is Juan Maldacena or your favorite source of Twilight fanfic.

Even if you follow all of this, you can’t trust every paper you find on arXiv. You also won’t enjoy everything you dig up on AO3. Either way, publication (in journals or books) won’t solve your problem: both are an additional filter, but not an infallible one. Judgement is still necessary.

This is all to say that “this article has not been peer-reviewed” can be a useful warning, but often isn’t. In theoretical physics, knowing who wrote an article and what it’s about will often tell you much more than whether or not it’s been peer-reviewed yet.

6 thoughts on “A Tale of Two Archives

  1. Giotis

    For the SYK model there is not even a paper yet, people cite Kitaev’s KITP talk.
    Perelman’s proof of the Poincare conjecture is still unpublished, you can find it only in the arXiv.

    I guess if you have something really important to say all these become mere formalities.

    But for non experts reading a regular paper from an unknown (even known sometimes) author without being able to evaluate it themselves, peer review is important for filtering junk, mistakes or tone down potential grandiose claims.


  2. ohwilleke

    Twitter and the blogosphere play a really important role here for folks who want to read notable new contributions relevant to them without slogging through every single contribution. Follow two or three folks with similar interests with a blog or twitter feed and you get effortless screening of the wheat from the chaff.

    BioRxiv does a pretty decent job of integrating this commentary with the underlying pre-prints; arXiv, not so much.


    1. ohwilleke

      Fortunately, in each case, the median quality is irrelevant. If there is some good stuff in the top 1%, humanity is still much better for it. (And, the median quality of ArXiv can be a bit dubious at times anyway.)

      Liked by 1 person

  3. Xezlec

    This makes physics sound clubby, with “who you know” determining whether your papers are read. I wonder if it would be nice to have some kind of a system with upvotes and downvotes and an algorithm that combines freshness with vote count, like Reddit. That way good stuff could get noticed even if it were by someone obscure.


    1. 4gravitonsandagradstudent Post author

      Within physics, that essentially happens by word of mouth. For outsiders, you can use citations as a rough metric of the same thing. And unlike Reddit, few enough papers go up on arXiv every day that many people will at least skim every abstract.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s