ArXiv Will Ban You for Hallucinated References

5 Replies

Thomas Dietterich, Chair of the Computer Science section of the preprint server arXiv.org, recently clarified the site’s policies towards “hallucinated” citations and other signs of careless use of AI in a post on X. If your paper contains a citation to a paper that doesn’t actually exist, or has other signs you didn’t read it before posting like leftover commentary (the example he gave was “here is a 200 word summary; would you like me to make any changes?”), then you can get banned from the arXiv for one year. Even after that year you’d be on a kind of “probation”, and would need to show that your next few papers had been accepted by peer-reviewed journals first before posting them.

At the risk of saying the obvious, this is a good idea! arXiv isn’t peer review, it isn’t meant to judge the value of the papers it hosts. But it still needs to be a useful place for scientists to post their papers, which is why they try to keep spam and irrelevant content to a minimum. If you don’t actually endorse the content of a paper, you shouldn’t post it in the first place.

That said, the whole existence of hallucinated citations on arXiv feels a little silly. It makes sense for academic journals and preprint servers in other fields. But arXiv was the first site of its kind for a reason. Its users, physicists, mathematicians, and computer scientists, don’t need much hand-holding when it comes to computers. Papers submitted to arXiv aren’t typically written in Word, they’re written in a document-writing language called LaTeX, that lets users make decently-formatted papers without help from a journal. Physicist-written code may be terrible by any reasonable criteria…but it exists, much more universally than for example biologist-written code.

This extends to citations. In my old field, there is a database called INSPIRE that updates automatically from arXiv. Click on a paper, and a handy “cite” link gives you standardized citations in several formats, ready to copy and paste into your LaTeX code. Nearly every citation in my papers is copied from there. The ones that aren’t are either from other fields where I didn’t know of that style of database, or things that haven’t been published (this can be manuscripts in preparation, or personal communications).

All of this, though, feels like a lot less than what the field could be doing. In a world where almost everyone posts their papers to the same website, and almost everyone has at least a rudimentary understanding of programming…why are people still writing citations in free-form text in the first place? Why aren’t citations built in to the submitted papers on arXiv, automatically linked to the papers they cite? Why don’t we have a setup where, except for a small number of “special” citations, every citation is built so that it automatically goes to a real paper, and gives a clear error message if it doesn’t? In short, why are hallucinated citations even possible?

Look, I’m naive, I get that. I believe in automation, not in the modern context of LLMs and other heuristics, but in setting clear procedures and building clear rules. The world doesn’t work that way! The clear rules are always more contentious than you expect, the fuzzy human-led version always the only choice people can agree on.

But still. Citations. There has to be a better system, right?

Make No Mistakes

9 Replies

I’m taking a Danish exam next week, and it’s a big one, a culmination of years learning the language. My classmates are stressed. Despite how much we’ve learned, it feels like we’re always making little mistakes. We write the wrong prepositions, put verbs in the wrong form, or mess up the order of words in a sentence. And while we should have time to check our work, that doesn’t help as much as it should. If we don’t notice a mistake the first time around, what guarantee is there that we notice it on the next read, or the next? Too many checks and we can even end up second-guessing ourselves, “correcting” something that was right to begin with.

It’s given me some sympathy for AI.

Earlier this month, investor Marc Andreessen posted a custom prompt he inputs when using AI, which was immediately mocked.

The silliest instruction, according to many critics, was to “Never hallucinate or make anything up.” It’s similar to a prompt that’s become a meme used to make fun of AI-using “vibe coders”, “Make no mistakes”.

Experts point out that this is just not how AI works. Large language model-powered programs like ChatGPT are inherently random, producing text largely based on its similarity to other text. “Hallucinations” or “mistakes” are an inevitable feature of the technology, and a prompt like Andreessen wrote isn’t a set of instructions the AI will follow without error: it’s just another part of the text the AI is trying to generate.

All that said, telling an AI to “make no mistakes” should have some effect. But it likely won’t be what you want.

The best way I’ve found to understand AI is in terms of stories. Chatbots like ChatGPT take a large language model, a mathematical formula for how words are most likely to appear in a text, and warp it, twisting it to almost always produce one particular kind of text: one half of a dialogue with a fictional AI assistant. This twisted formula determines how the AI responds to your prompts, but these days it also is used behind the scenes, in a kind of structured soliloquy called a “chain of thought”. You can think of the prompts you send to the AI as a preface to those soliloquies, and imagine the AI telling stories of a sort that would typically follow that preface.

So if you tell an AI “make no mistakes” or “do not hallucinate”, you’re making it more likely to generate the kind of story that begins, “the AI was instructed to make no mistakes”.

Let me put it this way, Mr. Amor. The 9000 series is the most reliable computer ever made. No 9000 computer has ever made a mistake or distorted information. We are all, by any practical definition of the words, foolproof and incapable of error. – HAL 9000, “2001: A Space Odyssey”

You’d expect this to affect the chain of thought. For example, the AI might occasionally pause to say “I’m supposed to make no mistakes, so I should check this. What could have gone wrong?” and then list something that plausibly could be wrong with its idea. If this happens often enough, you’ll probably catch some real problems.

But I’m reminded of my classmates, practicing for that Danish exam. We can go over the text again and again, asking if this thing, or that, might be wrong. We can try again and again to use our mental model of the Danish language, seeing if this time it catches a new mistake. But there are things we won’t catch. And if we do it too much, we’ll second-guess ourselves out of the good answers, too.

Ultimately, “make no mistakes” isn’t a great instruction, either for humans or for chatbots. And its use by people like Marc Andreessen has me wondering if they are used to interacting with humans in the same way, as tools that keep making mistakes no matter how many times they’re instructed not to, requiring more and more long-winded instructions and yet continuing to misbehave.

Then again, that may be a mistake on my part.

Bonus Info for “100-year-old assumption about the universe may soon be overturned”

2 Replies

I had a piece up in New Scientist last week (paywalled, sorry!), about a new analysis that suggests the universe is less homogeneous (more “lumpy”) that most cosmologists believe.

The piece was a bit different than my usual. Normally I do what people in the biz call “features”: longer articles about general trends. This was a much more classic “news piece”. The people I interviewed had several papers up in early April, the editors at New Scientist thought they were interesting enough to write about, so I was asked for a short, timely piece with the key takeaways.

That means I didn’t have a ton of space for background info. So if you’d like to know more, this post is for you!

The 100-year old assumption in the title refers to the Friedmann–Lemaître–Robertson–Walker (or FLRW) universe, an idea that first came together in the 1920’s, where cosmologists model the universe as homogeneous and isotropic: the same no matter where, or in which direction, you look. That sounds like a crazy assumption, but on the largest scales we can measure it’s actually mostly fine. Once you’re trying to calculate ripples in the cosmic microwave background or find out how fast distant galaxies are accelerating away, it works surprisingly well to act like the universe is an evenly-mixed soup of matter, radiation, dark matter, and dark energy.

But every assumption in physics has its doubters. The doubters of homogeneity are known as inhomogeneous cosmologists, and I’ve been sympathetic to their complaints for a while now.

I even let an inhomogeneous cosmologist do a guest post on my blog, back in 2019. That post argued something dramatic: that dark energy may not even exist, but that measurements of accelerating expansion may be a consequence of a dramatic lopsidedness in the universe around us.

The people I covered in New Scientist, Asta Heinesen, Tim Clifton, and Sofie Marie Koksbang, are arguing something much less dramatic…but that’s part of what makes it more compelling. Instead of arguing that the universe is dramatically uneven or lopsided, they’re arguing that the universe can still be on average smooth and homogeneous, the soup of galaxies people seem to expect…but still, can’t be fully modeled that way.

This is a tricky distinction to explain, and certainly something I didn’t have space to cover well enough in New Scientist. But let me take a stab at it here:

Any cosmologist will agree that FLRW can’t be the whole story. We know the universe isn’t a perfectly mixed soup: there are galaxies, and stars, and black holes, and they all wiggle the fabric of the universe in different places. When they study the universe as a whole, they’re averaging out all of that, to get the overall behavior, a bit like you could average the number of children in each family to get the average children per family in a country.

But FLRW isn’t just an average, it’s a model of spacetime. Because of that, it has to obey certain equations, called Einstein’s equations. It has to make sense by itself, as the correct answer for how spacetime would behave if it were filled with a uniform soup.

That’s an extra restriction, and that extra restriction can get you in trouble. To continue with the analogy, any real family has a whole number of children. But the average family doesn’t have a whole number of children. When I was born, the average family in the US had around 2.5 children. A lot of cartoons imagined what the half-child looked like.

From the perspective of Heinesen, Clifton, and Koksbang, assuming FLRW is a bit like assuming that the average family must have two children, or three, and can’t possibly have 2.5. Averages don’t have to look like sensible spacetimes, they don’t have to obey the Einstein equations.

In practice, the assumption of FLRW has worked a lot better than assuming that the average family can’t have 2.5 children, and that’s why Heinesen, Clifton, and Koksbang are cautious. They’re not claiming that inhomogeneity can explain everything, all the way to major components of the universe like dark energy. But they do think it can be a good explanation for smaller effects. And as cosmologists worry about smaller and smaller effects, wondering if dark energy changes over time and why the expansion rate of the universe doesn’t match up between different measurements, it can be important to remember that averages aren’t all-powerful. Eventually, they can break down. It’s a more subtle issue than a fractional child. But, as I covered in New Scientist, it may already be happening.

Breakthrough Prize 2026

Leave a reply

Because of last week’s “bonus info” post, I’m only now getting around to commenting on this year’s Breakthrough Prizes in Fundamental Physics. While I don’t comment on them every year, I know enough about several of this year’s winners that I figured a post would be helpful.

For those who haven’t heard of it, the Breakthrough Prizes are a bit like the Nobel, if it was created by a 21st century rich person instead of a 19th century one. They give out more money, and instead of an organization like the Swedish Academy of Sciences they pick winners via a committee of past winners. They’re more flexible in structure than the Nobel, with extra prizes for early-career researchers and a tendency to reward accomplishments that are either entirely theoretical or solid experimental work that doesn’t show a new discovery, both of which are things the Nobel Prize is structured to avoid. They’ve also shown willingness to reward large collaborations, rather than following the Nobel’s informal rule to only give the award to three people at a time.

This last was on display this year in their main award in physics this year, for the muon g-2 collaborations. The award is going to collaborations of scientists and engineers at three different particle colliders, for work done over a span of over fifty years to measure the magnetic properties of the muon. These measurements have shown a tantalizing discrepancy with predictions that inspired many to conjecture new physics. However, in the last few years it’s looked more and more like the discrepancy was due to an imprecise prediction, and better methods seem to be converging to the experimental value. At this point, smart money is that there is no disagreement with the Standard Model here, but as always in science there’s a chance some mystery remains.

The Breakthrough Prize also offered a special, out-of-schedule prize to David Gross. Already a Nobel laureate, Gross had a crucial role in our understanding of the force of quantum chromodynamics that binds protons and neutrons together. He was also a major founding figure in string theory, and since the Breakthrough Prize is more comfortable recognizing theoretical contributions they get to mention this as well. Gross is also known in the community for his personality, which tends to fill up any room he’s in. I can only imagine the conversations that led to Breakthrough’s decision to add a special prize for him this year.

Breakthrough is also adding a new recurring prize, the Vera Rubin New Frontiers Prize, honoring women who make important contributions to physics within two years of their PhD. The prize is a bit smaller than the exiting early-career New Horizons in Physics Prizes, presumably because it goes to even younger researchers. This year’s winner is from my old field, scattering amplitudes. Carolina Figueiredo is part of the latest evolution of the research program behind the amplituhedron. The new framework of “surfaceology” seems like a promising geometry-flavored way to understand particle physics calculations in more realistic theories, and unlike its predecessors may have some practical value eventually as well. Congrats Carolina!

Finally, the New Horizons in Physics Prizes are for impressive early-career researchers. I don’t know much about the first recipient, Benjamin Safdi, who works on searches for axions and axion-like particles, today’s most trendy dark matter candidate. I know a bit more about the work done by Clay Córdova, Thomas Dumitrescu, Shu-Heng Shao, and Yifan Wang, having met several of them in my physics career. They work on what are called generalized symmetries, concepts which go beyond the usual idea of how symmetry is supposed to work by involving more complicated tensors. I saw these crop up a fair bit in talks, but they were distant enough from my area that I never had a particularly clear grasp of what people were doing with them. I know even less about the work of the last three, Dillon Brout, J. Colin Hill, Mathew Madhavacheril, Maria Vincenzi, Daniel Scolnic, and W. L. Kimmy Wu, on cosmological measurements, but I was friends with Mathew in grad school and am impressed that he’s now working on cosmology given how little cosmology research there was at Stony Brook at the time.

Bonus Info for “Quantum ‘Jamming’ Explores the Truly Fundamental Principles of Nature”

4 Replies

I had a new piece in Quanta Magazine last week, about a hypothetical trick in theories beyond quantum mechanics called jamming.

Sometimes, I get science news stories from contacts. Sometimes I see an academic post something cool on X or Bluesky. But when the stories aren’t coming easy, I open up arXiv.org, click on “new”, and start browsing. And occasionally, I spot something cool.

That happened with jamming. I saw the concept mentioned in an abstract, the idea that someone could “jam” quantum entanglement from afar, like you would jam a radio signal. I hadn’t heard of it before. I wanted to know more. And after I talked to Quanta’s editors, they wanted to know more too.

Jamming is not possible under the rules of quantum mechanics we know. Instead, it’s something that could be possible in a kind of super-quantum mechanics, a theory even weirder than the famously weird theory we use today. In my piece for Quanta, I talked about where the idea of jamming comes from, and why it’s spurring discussion in recent years. In this post, I wanted to give some “bonus info” that didn’t fit into the piece.

One theme I didn’t have as much space to explore is causality.

Quantum mechanics famously seems to do weird things with cause and effect. In a double-slit experiment, photons pass one by one through one of two slits in a wall, headed to a photographic screen. No matter how slowly and carefully you send the photons, their distribution on the other end will show interference between the two possible paths, one through each slit, even though each photon only goes through one. It’s as if before hitting the screen, the photons are simultaneously traveling on every possible path, only to pick one in the moment the photon is detected.

Einstein was bothered by this. He imagined a photographic screen so large it would take light years to cross. How could detecting a photon on one side change the possibility of detecting a photon on the other side? That seemed, to him, to require signals traveling faster than light, which in turn would screw up cause and effect, as any way to send a signal faster than light can also, from another perspective, send a signal back in time.

The answer most physicists accept is that no signal can be sent in this way…at least, in the modern sense. Quantum outcomes are random, so while you could imagine that a measurement in one place changes the outcome in another place, your choice to measure has no effect on that distant outcome. You can’t intentionally send a message faster than light. We call that “no-signaling”, and it prevents the paradoxes of time travel.

Jamming obeys similar rules. A jammer (in the story in my article, a magician named Jim) can modify the entanglement between two distant particles, seemingly faster than light. But he can only do this in a way that involves randomness, so that the probabilities for measurement results for each individual particle stay the same. Instead, he can only modify how measurements between the two particles are related, their correlation. And he can only do this if the two particles can only be compared in a region that he can reach without traveling faster than light.

That’s enough to allow Jim to break the security of many quantum cryptography procedures. He can do this for example by mimicking entanglement: quantum cryptography often uses entanglement to verify that a message hasn’t been tampered with. If you can modify correlations from afar, you can make two particles appear to be entangled when actually they’re related by some other rules, which give you access to the secret that others are trying to hide.

Part of what’s still under discussion, is whether that kind of trick is compatible with causality. This depends a lot on how you think causality is supposed to work, and while the people I talked to are trying to get the story straight, they weren’t in agreement yet. In particular, Vilasini and Colbeck seemed to think that there was an important difference between the way that jamming bends causality and the way that ordinary quantum mechanics does, while Eckstein and Ramanathan weren’t so sure.

More broadly, Vilasini and Colbeck have a broader way of thinking about causality that I only barely touched on. Part of that is ways you can think of one event causing another even if no signal can be sent between them. Part of that is time loops, but of a limited kind: loops that can’t cause paradoxes, because they’re loops of causes, but not intentional signals. Vilasini and Colbeck have argued that jamming, if it existed, could be used to set up these kind of limited time loops, in a piece that was covered by New Scientist. It should be emphasized that these are really very limited time loops, for more reasons than one. They’re also limited to being in only one spatial dimension: that is, everyone in the loop has to be lined up in exactly a straight line. And I got the impression they also require everyone to activate their measurement or jamming devices instantly: with any small delay, the loop breaks.

I said even less about Mirjam Weilenmann’s critique, because there were bigger aspects that the researchers still disagreed on when I spoke with them. Weilenmann’s argument looks at what happens when there are multiple jammers, jamming different pairs of entangled particles. I got the impression from her that she felt she had found a contradiction in these examples, where jamming could only work if it broke its essential no-signaling rules. But Eckstein and Ramanathan seemed to think she was describing a scenario where one jammer could cause noise that would disrupt another jammer, “jamming the jammers” in a sense that didn’t cause any fundamental problems, just introduced jammer vs. jammer combat to make the story more interesting. I opted to not say much about this, since it was clear that things weren’t resolved yet. The researchers are still talking, and I look forward to hearing what they conclude when they reach agreement.

I also didn’t say much about tests in the real world. But that is something Eckstein and collaborators are actively exploring. They’re investigating experiments that could show deviations from quantum mechanics in a variety of contexts, from tabletops in university labs to particle colliders. The hope is that some of these strange ideas could actually be tested.

In general, the impression I got was that despite the seeds of this topic being laid thirty years ago, and reintroduced to the field ten years ago…the topic is heating up right now, in a way it hadn’t before. I’m expecting more jamming papers. If they’re cool enough, I may even cover some of them.

A Window on Absolutely Everything

15 Replies

It’s often said that in quantum physics, everything that can happen will happen.

One way this comes up is in something called a path integral, used to calculate the probabilities of quantum events. If you want to find what happens to a particle traveling from point A to point B, you have to add up a contribution for every path, no matter how windy, that goes between A and B. These contributions mostly cancel out, and matter less the further they are from a straight line, so the straight-line path is, for the most part, a good description of what happens. But in principle, all of the other paths matter too.

The same thing happens in quantum field theory, in more elaborate form. Instead of a path from one place to another, the paths are from one configuration of quantum fields to another, via all the different ways fields can in principle interact. We are almost never able to take account of all these possibilities mathematically, so we have to approximate, organizing the interactions into more and more complicated pictures called Feynman diagrams, each with a smaller and smaller effect.

In principle, these diagrams need to contain every single combination of interactions that might result in the end-state we’re interested in. These combinations can have a Rube Goldberg flavor, with one field activating another, which activates another, only to all cancel out in the end. Because of this, any field that exists, any particle no matter how rare, can matter, if only a little.

And from that, physicists can learn something.

Because absolutely everything matters, physicists get to reason about absolutely everything that exists.

The best example involves something called an anomaly. These aren’t the anomalies of experimental physics, unexpected results that have a tendency to go away with better measurements. Instead of something unexpected, a theorist’s anomaly is something impossible.

Anomalies are combinations of particles that, if they were to show up together in a sum of Feynman diagrams, would break the rules that the theory was made with in the first place. If they show up, they’re a sign of an inconsistent theory, one that doesn’t obey its own rules and thus doesn’t make sense.

In order to have a theory without anomalies, different calculations involving different particles need to cancel. For example, it might be that the charge of different particles has to add up to zero. This means that if you’ve only discovered a few particles, and their charges don’t add up to zero, then you know you’re missing one. There is an extra particle there, which you haven’t observed, that together makes charge add up to zero.

This logic actually works! It was used to predict the top quark. Before the top quark was discovered, the list of quarks, electrons, and neutrinos had electric charges that didn’t add up to zero. One particle was missing, with the same charge as the up quark and charm quark. It was found in 1995, after being proposed almost 20 years earlier.

What AI Physicists Are Missing and What They Aren’t

4 Replies

I’ve seen a couple more thoughtful takes on use of LLMs for physics lately. This blog post by Minas Karamis is particularly nice.

He points out something that I’ve said a version of: an AI that must be supervised like a student isn’t very useful, because the main point of student projects isn’t the paper at the end: it’s training the student. If students don’t struggle through all the mistakes of a project, they won’t get the expertise to one day do greater things.

Someone might object that not all suffering is educational. In the 1700’s, Leonhard Euler calculated digit after digit of transcendental numbers by hand. Nobody asks students to do that anymore, and they still seem to turn out alright. Why would using an LLM for science be worse than using a computer for numerical calculations?

In a word: different skills. Programming numerics teaches you some of the same skills as calculating the numbers by hand: skills at being specific about what you mean, aware of the consequences of the details and their implications. Prompting an AI still requires those skills, to check whether the AI’s output is correct. But it’s much worse at teaching them: unlike programming or calculating, when prompting AI, the consequences of your actions aren’t predictable.

For some, though, there is another objection. Sure, using AI reliably might require those skills now. But when it gets better, surely being careful will stop mattering. Surely the AI will end up doing science on its own, and all that training will be as useful as if we trained the students to play football.

I’m skeptical, but not as strongly as some. I think we’re still living in a time when it makes sense to hire scientists, and train people to think, and invest in your retirement.

I don’t think I have any knock-down arguments for that, though. Just some suggestive ones.

One I’ve talked about before is that a lot of the most important parts of thinking aren’t written down. An AI physicist is going to have a hard time replicating the kinds of methods and approaches that people use behind the scenes, but rarely describe or spell out. It will be easier to suss this out over time, as more data accumulates of people working with LLMs and correcting them. But ultimately there isn’t going to be a lot of documentation of this kind of thing.

Another limitation is memory. A mature scientist can draw from experiences across their entire career. For an LLM, any problem it’s solved in the past is by default lost in each new session. People build structures around this, taking notes and reminding the AI when it “wakes up”, or making documents the AI can be prompted to check. But nothing in this vein so far seems to get nearly as wide-scope or powerful as human memory. A scientist career is still the best way we have to build durable, functional expertise.

Finally, there is a question of costs, and efficiency. Here I’m not an expert, and I get the impression the actual experts disagree. I don’t know whether we should expect scaling to hit a wall, but I wouldn’t be that surprised if it did.

There are other common reasons for skepticism that seem more dubious to me. I don’t think AI is inherently worse at creativity just because they’re trained on existing work, though some of the skills we associate with creativity aren’t very well-documented, and thus are hard to train for. I don’t think AI’s randomness or unreliability is a deal-breaker, because human intuition is also random and unreliable: we solve that with tools, and that’s something AI can in principle do as well. I don’t think humans are “more agentic” or something, except in the sense that most AIs are made by companies who need to make them behave in a customer-friendly way. But an agent is just a game-theoretic construct, a way to figure out can win or lose in situations with defined stakes, and anything you can train or engineer to try to win can be modeled by that construct.

Coming from a place of uncertainty, my main appeal to you is to not get hung up on the bad reasons, either yourself, or from the people you’re arguing with. Focus on the best arguments, and see where they take you.

ArXiv to Leave Cornell

Leave a reply

Yes, I’m late to the party on this one.

A few weeks ago, arXiv.org announced that it will be leaving Cornell, the university that currently manages it, and establishing its own nonprofit.

arXiv is a crucial part of the infrastructure for physics, mathematics, computer science, and a few related fields. Researchers post papers to arXiv as what are called “preprints” before the papers are submitted to a journal. In practice, nobody ends up reading the journal versions: the arXiv is free to access, and typically reflects better what the paper’s authors want the paper to look like. So in practice, arXiv is how researchers in these fields communicate, which makes its role enormously important.

If you’re from another field, you might wonder how something like arXiv is financially sustainable. The answer is that it works better than you’d think, but not perfectly. They’ve been supported by philanthropy, in addition to Cornell, and while there have apparently been budget shortfalls and drama behind the scenes, But nonetheless, arXiv has stayed in continuous operation since 1991.

The move to an independent nonprofit is supposed to make it easier for arXiv to get philanthropic funding, which otherwise needed to be filtered through Cornell in ways that were sometimes opaque or didn’t give donors the control they wanted.

While it wasn’t mentioned in the announcements, I suspect another motivation is security. Universities are fixed in place, and that makes them easier to pressure. For an organization that wants to process scientific output in an unbiased way, the link to Cornell represented a vulnerability. It’s not a vulnerability that has mattered yet, and likely didn’t seem like it would ever matter. But it wouldn’t surprise me if they’re more worried now that someone might try to pressure Cornell in order to change how arXiv operates. For critical scientific infrastructure, it’s important to be as independent of those kinds of pressure as possible.

Trust Is a Tree

Leave a reply

Scientists trust what they think they can verify.

In principle, you can work your way through the proof of every mathematical theorem. With enough money and time, you could replicate every experiment. For every expert opinion, you could dig through the literature and find how it was justified.

And while a scientist can’t actually do that for every field, they might be able to for the ones they care about most. In your specialty, you probably can check the logic behind every claim. And you know that enough people try, that you can trust your colleagues’ work.

As a science journalist, most of the time, you can’t do those checks. You don’t even pretend you can. Instead, you build trust, like a tree.

You start with a grounding. A former scientist might trust their former colleagues, people they trusted, as a scientist, to do (and know) good work. A non-scientist has to start somewhere else. They might use prestige, looking up those tenured folks at Harvard or Princeton or Stanford. They might look to who other journalists trusted, scientists who’ve already been in the news. They might track journals or roles, assuming that a publication in Nature, or a position on a national grant committee, has a special meaning.

And if things stopped there, it would be a pretty elitist system. It still can be, and often is. But there is another step, which softens it.

The trust builds.

When I want to know if a paper in an unfamiliar field makes sense, if it’s worth covering, I try to ask someone I trust. Sometimes, they don’t know, and shrug. Other, more useful, times, they don’t know, but they have a suggestion: someone they trust, who can give me the answer.

And so I ask the new person, and now I trust someone more.

And suppose the new person says the new paper is good, and worth covering, good science and all that jazz.

Well, now I can trust its authors too, right?

So when the next paper comes, I now don’t just have that first someone. I have the person they recommended, and the authors of the previous paper.

The trust builds out, and up, like branches on a tree.

The Twitter of Physics

4 Replies

The paper I talked about last week was frustratingly short. That’s not because the authors were trying to hide anything, or because they were lazy. It’s just that these days, that’s how the game is played.

Twitter started out with a fun gimmick: all posts had to be under 140 characters. The restriction inspired some great comedy, trying to pack as much humor as possible into a bite-sized format. Then, Twitter somehow became the place for journalists to discuss the news, tech people to discuss the industry, and politicians to discuss politics. Now, the length limit fuels conflict, an endless scroll of strong opinions without space for nuance.

Physics has something like this too.

In the 1950’s, it was hard for scientists to get the word out quickly about important results. The journal Physical Review had a trick: instead of normal papers, they’d accept breaking news in the form of letters to the editor, which they could publish more quickly than the average paper. In 1958, editor Samuel Goudsmit founded a new journal, Physical Review Letters (or PRL for short), that would publish those letters all in one place, enforcing a length limit to make them faster to process.

The new journal was a hit, and soon played host to a series of breakthrough results, as scientists chose it as a way to get their work out fast. That popularity created a problem, though. As PRL’s reputation grew, physicists started trying to publish there not because their results needed to get out fast, but because just by publishing in PRL, their papers would be associated with all of the famous breakthroughs the journal had covered. Goudsmit wrote editorials trying to slow this trend, but to no avail.

Now, PRL is arguably the most prestigious journal in physics, hosting over a quarter of Nobel prize-winning work. Its original motivation is no longer particularly relevant: the journal is not all that much faster than other journals in its area, if at all, and is substantially slower than the preprint server arXiv, which is where physicists actually read papers in practice.

The length limit has changed over the years, but not dramatically. It now sits at 3,750 words, typically allowing a five-or-six page article in tight two-column text.

If you see a physics paper on arXiv.org that fits the format, it’s almost certainly aimed at PRL, or one of the journals with similar policies that it inspired. It means the authors think their work is cool enough to hang out with a quarter of all Nobel-winning results, or at least would like it to be.

And that, in turn, means that anyone who wants to claim that prestige has to be concise. They have to leave out details (often, saving them for a later publication in a less-renowned journal). The results have to lean, by the journal’s nature, more to physicist-clickbait and a cleaned-up story than to anything their colleagues can actually replicate.

Is it fun? Yeah, I had some PRLs in my day. It’s a rush, shining up your work as far as it can go, trimming down complexities into six pages of essentials.

But I’m not sure it’s good for the field.