Author Archives: 4gravitons

The Mistakes Are the Intelligence

The Bystander Effect for Reviewers

I probably came off last week as a bit of an extreme “journal abolitionist”. This week, I wanted to give a couple caveats.

First, as a commenter pointed out, the main journals we use in my field are run by nonprofits. Physical Review Letters, the journal where we publish five-page papers about flashy results, is run by the American Physical Society. The Journal of High-Energy Physics, where we publish almost everything else, is run by SISSA, the International School for Advanced Studies in Trieste. (SISSA does use Springer, a regular for-profit publisher, to do the actual publishing.)

The journals are also funded collectively, something I pointed out here before but might not have been obvious to readers of last week’s post. There is an agreement, SCOAP3, where research institutions band together to pay the journals. Authors don’t have to pay to publish, and individual libraries don’t have to pay for subscriptions.

And this is a lot better than the situation in other fields, yeah! Though I’d love to quantify how much. I haven’t been able to find a detailed breakdown, but SCOAP3 pays around 1200 EUR per article published. What I’d like to do (but not this week) is to compare this to what other fields pay, as well as to publishing that doesn’t have the same sort of trapped audience, and to online-only free journals like SciPost. (For example, publishing actual physical copies of journals at this point is sort of a vanity thing, so maybe we should compare costs to vanity publishers?)

Second, there’s reviewing itself. Even without traditional journals, one might still want to keep peer review.

What I wanted to understand last week was what peer review does right now, in my field. We read papers fresh off the arXiv, before they’ve gone through peer review. Authors aren’t forced to update the arXiv with the journal version of their paper, if they want another version, even if that version was rejected by the reviewers, then they’re free to do so, and most of us wouldn’t notice. And the sort of in-depth review that happens in peer review also happens without it. When we have journal clubs and nominate someone to present a recent paper, or when we try to build on a result or figure out why it contradicts something we thought we knew, we go through the same kind of in-depth reading that (in the best cases) reviewers do.

But I think I’ve hit upon something review does that those kinds of informal things don’t. It gets us to speak up about it.

I presented at a journal club recently. I read through a bombastic new paper, figured out what I thought was wrong with it, and explained it to my colleagues.

But did I reach out to the author? No, of course not, that would be weird.

Psychologists talk about the bystander effect. If someone collapses on the street, and you’re the only person nearby, you’ll help. If you’re one of many, you’ll wait and see if someone else helps instead.

I think there’s a bystander effect for correcting people. If someone makes a mistake and publishes something wrong, we’ll gripe about it to each other. But typically, we won’t feel like it’s our place to tell the author. We might get into a frustrating argument, there wouldn’t be much in it for us, and it might hurt our reputation if the author is well-liked.

(People do speak up when they have something to gain, of course. That’s why when you write a paper, most of the people emailing you won’t be criticizing the science: they’ll be telling you you need to cite them.)

Peer review changes the expectations. Suddenly, you’re expected to criticize, it’s your social role. And you’re typically anonymous, you don’t have to worry about the consequences. It becomes a lot easier to say what you really think.

(It also becomes quite easy to say lazy stupid things, of course. This is why I like setups like SciPost, where reviews are made public even when the reviewers are anonymous. It encourages people to put some effort in, and it means that others can see that a paper was rejected for bad reasons and put less stock in the rejection.)

I think any new structure we put in place should keep this feature. We need to preserve some way to designate someone a critic, to give someone a social role that lets them let loose and explain why someone else is wrong. And having these designated critics around does help my field. The good criticisms get implemented in the papers, the authors put the new versions up on arXiv. Reviewing papers for journals does make our science better…even if none of us read the journal itself.

Why Journals Are Sticky

9 Replies

An older professor in my field has a quirk: every time he organizes a conference, he publishes all the talks in a conference proceeding.

In some fields, this would be quite normal. In computer science, where progress flows like a torrent, new developments are announced at conferences long before they have the time to be written up carefully as a published paper. Conference proceedings are summaries of what was presented at the conference, published so that anyone can catch up on the new developments.

In my field, this is rarer. A few results at each conference will be genuinely new, never-before-published discoveries. Most, though, are talks on older results, results already available online. Writing them up again in summarized form as a conference proceeding seems like a massive waste of time.

The cynical explanation is that this professor is doing this for the citations. Each conference proceeding one of his students publishes is another publication on their CV, another work that they can demand people cite whenever someone uses their ideas or software, something that puts them above others’ students without actually doing any extra scientific work.

I don’t think that’s how this professor thinks about it, though. He certainly cares about his students’ careers, and will fight for them to get cited as much as possible. But he asks everyone at the conference to publish a proceeding, not just his students. I think he’d argue that proceedings are helpful, that they can summarize papers in new ways and make them more accessible. And if they give everyone involved a bit more glory, if they let them add new entries to their CV and get fancy books on their shelves, so much the better for everyone.

My guess is, he really believes something like that. And I’m fairly sure he’s wrong.

The occasional conference proceeding helps, but only because it makes us more flexible. Sometimes, it’s important to let others know about a new result that hasn’t been published yet, and we let conference proceedings go into less detail than a full published paper, so this can speed things up. Sometimes, an old result can benefit from a new, clearer explanation, which normally couldn’t be published without it being a new result (or lecture notes). It’s good to have the option of a conference proceeding.

But there is absolutely no reason to have one for every single talk at a conference.

Between the cynical reason and the explicit reason, there’s the banal one. This guy insists on conference proceedings because they were more useful in the past, because they’re useful in other fields, and because he’s been doing them himself for years. He insists on them because to him, they’re a part of what it means to be a responsible scientist.

And people go along with it. Because they don’t want to get into a fight with this guy, certainly. But also because it’s a bit of extra work that could give a bit of a career boost, so what’s the harm?

I think something similar to this is why academic journals still work the way they do.

In the past, journals were the way physicists heard about new discoveries. They would get each edition in the mail, and read up on new developments. The journal needed to pay professional copyeditors and printers, so they needed money, and they got that money from investors by being part of for-profit companies that sold shares.

Now, though, physicists in my field don’t read journals. We publish our new discoveries online on a non-profit website, formatting them ourselves with software that uses the same programming skills we use in the rest of our professional lives. We then discuss the papers in email threads and journal club meetings. When a paper is wrong, or missing something important, we tell the author, and they fix it.

Oh, and then after that we submit the papers to the same for-profit journals and the same review process that we used to use before we did all this, listing the journals that finally accept the papers on our CVs.

Why do we still do that?

Again, you can be cynical. You can accuse the journals of mafia-ish behavior, you can tie things back to the desperate need to publish in high-ranked journals to get hired. But I think the real answer is a bit more innocent, and human, than that.

Imagine that you’re a senior person in the field. You may remember the time before we had all of these nice web-based publishing options, when journals were the best way to hear about new developments. More importantly than that, though, you’ve worked with these journals. You’ve certainly reviewed papers for them, everyone in the field does that, but you may have also served as an editor, tracking down reviewers and handling communication between the authors and the journal. You’ve seen plenty of cases where the journal mattered, where tracking down the right reviewers caught a mistake or shot down a crackpot’s ambitions, where the editing cleaned something up or made a work more appear more professional. You think of the journals as having high standards, standards you have helped to uphold: when choosing between candidates for a job, you notice that one has several papers in Physical Review Letters, and remember papers you’ve rejected for not meeting what you intuited were that journal’s standards. To you, journals are a key part of being a responsible scientist.

Does any of that make journals worth it, though?

Well, that depends on costs. It depends on alternatives. It depends not merely on what the journals catch, but on how often they do it, and how much would have been caught on its own. It depends on whether the high standards you want to apply to job applicants are already being applied by the people who write their recommendation letters and establish their reputations.

And you’re not in a position to evaluate any of that, of course. Few people are, who don’t spend a ton of time thinking about scientific publishing.

And thus, for the non-senior people, there’s not much reason to push back. One hears a few lofty speeches about Elsevier’s profits, and dreams about the end of the big for-profit journals. But most people aren’t cut out to be crusaders or reformers, especially when they signed up to be scientists. Most people are content not to annoy the most respected people in their field by telling them that something they’ve spent an enormous amount of time on is now pointless. Most people want to be seen as helpful by these people, to not slack off on work like reviewing that they argue needs doing.

And most of us have no reason to think we know that much better, anyway. Again, we’re scientists, not scientific publishing experts.

I don’t think it’s good practice to accuse people of cognitive biases. Everyone thinks they have good reasons to believe what they believe, and the only way to convince them is to address those reasons.

But the way we use journals in physics these days is genuinely baffling. It’s hard to explain, it’s the kind of thing people have been looking quizzically at for years. And this kind of explanation is the only one I’ve found that matches what I’ve seen. Between the cynical explanation and the literal arguments, there’s the basic human desire to do what seems like the responsible thing. That tends to explain a lot.

Grad Students Don’t Have Majors

The Machine Learning for Physics Recipe

HAMLET-Physics 2024

1 Reply

Back in January, I announced I was leaving France and leaving academia. Since then, it hasn’t made much sense for me to go to conferences, even the big conference of my sub-field or the conference I organized.

I did go to a conference this week, though. I had two excuses:

The conference was here in Copenhagen, so no travel required.
The conference was about machine learning.

HAMLET-Physics, or How to Apply Machine Learning to Experimental and Theoretical Physics, had the additional advantage of having an amusing acronym. Thanks to generous support by Carlsberg and the Danish Data Science Academy, they could back up their choice by taking everyone on a tour of Kronborg (better known in the English-speaking world as Elsinore).

This conference’s purpose was to bring together physicists who use machine learning, machine learning-ists who might have something useful to say to those physicists, and other physicists who don’t use machine learning yet but have a sneaking suspicion they might have to at some point. As a result, the conference was super-interdisciplinary, with talks by people addressing very different problems with very different methods.

Interdisciplinary conferences are tricky. It’s easy for the different groups of people to just talk past each other: everyone shows up, gives the same talk they always do, socializes with the same friends they always meet, then leaves.

There were a few talks that hit that mold, and were so technical only a few people understood. But most were better. The majority of the speakers did really well at presenting their work in a way that would be understandable and even exciting to people outside their field, while still having enough detail that we all learned something. I was particularly impressed by Thea Aarestad’s keynote talk on Tuesday, a really engaging view of how machine learning can be used under the extremely tight time constraints LHC experiments need to decide whether to record incoming data.

For the social aspect, the organizers had a cute/gimmicky/machine-learning-themed solution. Based on short descriptions and our public research profiles, they clustered attendees, plotting the connections between them. They then used ChatGPT to write conversation prompts between any two people on the basis of their shared interests. In practice, this turned out to be amusing but totally unnecessary. We were drawn to speak to each other not by conversation prompts, but by a drive to learn from each other. “Why do you do it that way?” was a powerful conversation-starter, as was “what’s the best way to do this?” Despite the different fields, the shared methodologies gave us strong reasons to talk, and meant that people were very rarely motivated to pick one of ChatGPT’s “suggestions”.

Overall, I got a better feeling for how machine learning is useful in physics (and am planning a post on that in future). I also got some fresh ideas for what to do myself, and a bit of a picture of what the future holds in store.

Why Quantum Gravity Is Controversial

27 Replies

Merging quantum mechanics and gravity is a famously hard physics problem. Explaining why merging quantum mechanics and gravity is hard is, in turn, a very hard science communication problem. The more popular descriptions tend to lead to misunderstandings, and I’ve posted many times over the years to chip away at those misunderstandings.

Merging quantum mechanics and gravity is hard…but despite that, there are proposed solutions. String Theory is supposed to be a theory of quantum gravity. Loop Quantum Gravity is supposed to be a theory of quantum gravity. Asymptotic Safety is supposed to be a theory of quantum gravity.

One of the great virtues of science and math is that we are, eventually, supposed to agree. Philosophers and theologians might argue to the end of time, but in math we can write down a proof, and in science we can do an experiment. If we don’t yet have the proof or the experiment, then we should reserve judgement. Either way, there’s no reason to get into an unproductive argument.

Despite that, string theorists and loop quantum gravity theorists and asymptotic safety theorists, famously, like to argue! There have been bitter, vicious, public arguments about the merits of these different theories, and decades of research doesn’t seem to have resolved them. To an outside observer, this makes quantum gravity seem much more like philosophy or theology than like science or math.

Why is there still controversy in quantum gravity? We can’t do quantum gravity experiments, sure, but if that were the problem physicists could just write down the possibilities and leave it at that. Why argue?

Some of the arguments are for silly aesthetic reasons, or motivated by academic politics. Some are arguments about which approaches are likely to succeed in future, which as always is something we can’t actually reliably judge. But the more justified arguments, the strongest and most durable ones, are about a technical challenge. They’re about something called non-perturbative physics.

Most of the time, when physicists use a theory, they’re working with an approximation. Instead of the full theory, they’re making an assumption that makes the theory easier to use. For example, if you assume that the velocity of an object is small, you can use Newtonian physics instead of special relativity. Often, physicists can systematically relax these assumptions, including more and more of the behavior of the full theory and getting a better and better approximation to the truth. This process is called perturbation theory.

Other times, this doesn’t work well. The full theory has some trait that isn’t captured by the approximations, something that hides away from these systematic tools. The theory has some important aspect that is non-perturbative.

Every proposed quantum gravity theory uses approximations like this. The theory’s proponents try to avoid these approximations when they can, but often they have to approximate and hope they don’t miss too much. The opponents, in turn, argue that the theory’s proponents are missing something important, some non-perturbative fact that would doom the theory altogether.

Asymptotic Safety is built on top of an approximation, one different from what other quantum gravity theorists typically use. To its proponents, work using their approximation suggests that gravity works without any special modifications, that the theory of quantum gravity is easier to find than it seems. Its opponents aren’t convinced, and think that the approximation is missing something important which shows that gravity needs to be modified.

In Loop Quantum Gravity, the critics think their approximation misses space-time itself. Proponents of Loop Quantum Gravity have been unable to prove that their theory, if you take all the non-perturbative corrections into account, doesn’t just roll up all of space and time into a tiny spiky ball. They expect that their theory should allow for a smooth space-time like we experience, but the critics aren’t convinced, and without being able to calculate the non-perturbative physics neither side can convince the other.

String Theory was founded and originally motivated by perturbative approximations. Later, String Theorists figured out how to calculate some things non-perturbatively, often using other simplifications like supersymmetry. But core questions, like whether or not the theory allows a positive cosmological constant, seem to depend on non-perturbative calculations that the theory gives no instructions for how to do. Some critics don’t think there is a consistent non-perturbative theory at all, that the approximations String Theorists use don’t actually approximate to anything. Even within String Theory, there are worries that the theory might try to resist approximation in odd ways, becoming more complicated whenever a parameter is small enough that you could use it to approximate something.

All of this would be less of a problem with real-world evidence. Many fields of science are happy to use approximations that aren’t completely rigorous, as long as those approximations have a good track record in the real world. In general though, we don’t expect evidence relevant to quantum gravity any time soon. Maybe we’ll get lucky, and studies of cosmology will reveal something, or an experiment on Earth will have a particularly strange result. But nature has no obligation to help us out.

Without evidence, though, we can still make mathematical progress. You could imagine someone proving that the various perturbative approaches to String Theory become inconsistent when stitched together into a full non-perturbative theory. Alternatively, you could imagine someone proving that a theory like String Theory is unique, that no other theory can do some key thing that it does. Either of these seems unlikely to come any time soon, and most researchers in these fields aren’t pursuing questions like that. But the fact the debate could be resolved means that it isn’t just about philosophy or theology. There’s a real scientific, mathematical controversy, one rooted in our inability to understand these theories beyond the perturbative methods their proponents use. And while I don’t expect it to be resolved any time soon, one can always hold out hope for a surprise.

Toy Models

7 Replies

In academia, scientists don’t always work with what they actually care about. A lot of the time, they use what academics call toy models. A toy model can be a theory with simpler mathematics than the theories that describe the real world, but it can also be something that is itself real, just simpler or easier to work with, like nematodes, fruit flies, or college students.

Some people in industry seem to think this is all academics do. I’ve seen a few job ads that emphasize experience dealing with “real-world data”, and a few people skeptical that someone used to academia would be able to deal with the messy challenges of the business world.

There’s a grain of truth to this, but I don’t think industry has a monopoly on mess. To see why, let’s think about how academics write computer code.

There are a lot of things that one is in-principle supposed to do to code well, and most academics do none of them. Good code has test suites, so that if you change something you can check whether it still works by testing it on all the things that could go wrong. Good code is modular, with functions doing specific things and re-used whenever appropriate. Good code follows shared conventions, so that others can pick up your code and understand how you did it.

Some academics do these things, for example those who build numerical simulations on supercomputers. But for most academics, coding best-practices range from impractical to outright counterproductive. Testing is perhaps the clearest example. To design a test suite, you have to have some idea what kinds of things your code will run into, what kind of input you expect what the output is supposed to be. Many academic projects, though, are the first of their kind. Academics code up something to do a calculation nobody has done before, not knowing the result, or they make code to analyze a dataset nobody has worked with before. By the time they understand the problem well enough to write a test suite, they’ve already solved the problem, and they’re on to the next project, which may need something totally different.

From the perspective of these academics, if you have a problem well-defined enough that you can build a test suite, well enough that you can have stable conventions and reusable functions…then you have a toy model, not a real problem from the real world.

…and of course, that’s not quite fair either, right?

The truth is, academics and businesspeople want to work with toy models. Toy models are well-behaved, and easy, and you can do a lot with them. The real world isn’t a toy model…but it can be, if you make it one.

This means planning your experiments, whether in business or in science. It means making sure the data you gather is labeled and organized before you begin. It means coming up with processes, and procedures, and making as much of the work as possible a standardized, replicable thing. That’s desirable regardless, whether you’re making a consistent product instead of artisanal one-offs or a well-documented scientific study that another team can replicate.

Academia and industry both must handle mess. They handle different kinds of mess in different circumstances, and manage it in different ways, and this can be a real challenge for someone trying to go from one world to another. But neither world is intrinsically messier or cleaner. Nobody has a monopoly on toy models.

The “Who” of Fixing Academic Publishing

2 Replies

I was on the debate team in high school. There’s a type of debate, called Policy, where one team proposes a government policy and the other team argues the policy is bad. The rules of Policy debate don’t say who the debaters are pretending to be: they could be congresspeople, cabinet members, or staff at a think tank. This creates ambiguity, and nerds are great at exploiting ambiguity. A popular strategy was to argue that the opponents had a perfectly good policy, but were wrong about who should implement it. This had reasonable forms (no, congress does not have the power to do X) but could also get very silly (the crux of one debate was whether the supreme court or the undersecretary of the TSA was the best authority to usher in a Malthusian dictatorship). When debating policy, “who” could be much more important than “what”.

Occasionally, when I see people argue that something needs to be done, I ask myself this question. Who, precisely, should do it?

Recently, I saw a tweet complaining about scientific publishing. Physicists put their work out for free on arXiv.org, then submit that work to journals, which charge huge fees either to the scientists themselves or to libraries that want access to the work. It’s a problem academics complain about frequently, but usually we act like it’s something we should fix ourselves, a kind of grassroots movement to change our publication and hiring culture.

This tweet, surprisingly, didn’t do that. Instead, it seemed to have a different “who” in mind. The tweet argued that the stranglehold of publishers like Elsevier on academic publishing is a waste of taxpayer money. The implication, maybe intended maybe not, is that the problem should be fixed by the taxpayers: that is, by the government.

Which in turn got me thinking, what could that look like?

I could imagine a few different options, from the kinds of things normal governments do to radical things that would probably never happen.

First, the most plausible strategy: collective negotiation. Particle physicists don’t pay from our own grants to publish papers, and we don’t pay to read them. Instead, we have a collective agreement, called SCOAP3, where the big institutions pay together each year to guarantee open access. The University of California system tried to negotiate a similar agreement a few years back, not just for physicists but for all fields. You could imagine governments leaning on this, with the university systems of whole countries negotiating a fixed payment. The journals would still be getting paid, but less.

Second, less likely but not impossible: governments could use the same strategies against the big publishers that they use against other big companies. This could be antitrust action (if you have to publish in Nature to get hired, are they really competing with anybody?), or even some kind of price controls. The impression I get is that when governments do try to change scientific publishing they usually do it via restrictions on the scientists (such as requiring them to publish open-access), while this would involve restrictions on the publishers.

Third, governments could fund alternative institutions to journals. They could put more money into websites like arXiv.org and its equivalents in other fields or fund an alternate review process to vet papers like journal referees do. There are existing institutions they could build on, or they could create their own.

Fourth, you could imagine addressing the problem on the job market side, with universities told not to weigh the prestige of journals when considering candidates. This seems unlikely to happen, and that’s probably a good thing, because it’s very micromanagey. Still, I do think that both grants and jobs could do with less time and effort spent attempting to vet candidates and more explicit randomness.

Fifth, you could imagine governments essentially opting out of the game altogether. They could disallow spending any money from publicly funded grants or universities on open-access fees or subscription fees, pricing most scientists out of the journal system. Journals would either have to radically lower their prices so that scientists could pay for them out of pocket, or more likely go extinct. This does have the problem that if only some countries did it, their scientists would have a harder time in other countries’ job markets. And of course, many critics of journals just want the journals to make less obscene profits, and not actually go extinct.

Most academics I know agree that something is deeply wrong with how academic journals work. While the situation might be solved at the grassroots level, it’s worth imagining what governments might do. Realistically, I don’t expect them to do all that much. But stranger things have gotten political momentum before.

At Quanta This Week, With a Piece on Vacuum Decay

2 Replies

I have a short piece at Quanta Magazine this week, about a physics-y end of the world as we know it called vacuum decay.

For science-minded folks who want to learn a bit more: I have a sentence in the article mentioning other uncertainties. In case you’re curious what those uncertainties are:

Gamma ( $\gamma$ ) here is the decay rate, its inverse gives the time it takes for a cubic gigaparsec of space to experience vacuum decay. The three uncertainties are from experiments, the uncertainties of our current knowledge of the Higgs mass, top quark mass, and the strength of the strong force.

Occasionally, you see futurology-types mention “uncertainties in the exponent” to argue that some prediction (say, how long it will take till we have human-level AI) is so uncertain that estimates barely even make sense: it might be 10 years, or 1000 years. I find it fun that for vacuum decay, because of that $\log_{10}$ , there is actually uncertainty in the exponent! Vacuum decay might happen in as few as $10^{411}$ years or as many as $10^{1333}$ years, and that’s the result of an actual, reasonable calculation!

For physicist readers, I should mention that I got a lot out of reading some slides from a 2016 talk by Matthew Schwartz. Not many details of the calculation made it into the piece, but the slides were helpful in dispelling a few misconceptions that could have gotten into the piece. There’s an instinct to think about the situation in terms of the energy, to think about how difficult it is for quantum uncertainty to get you over the energy barrier to the next vacuum. There are methods that sort of look like that, if you squint, but that’s not really how you do the calculation, and there end up being a lot of interesting subtleties in the actual story. There were also a few numbers that it was tempting to put on the plots in the article, but turn out to be gauge dependent!

Another thing I learned from those slides how far you can actually take the uncertainties mentioned above. The higher-energy Higgs vacuum is pretty dang high-energy, to the point where quantum gravity effects could potentially matter. And at that point, all bets are off. The calculation, with all those nice uncertainties, is a calculation within the framework of the Standard Model. All of the things we don’t yet know about high-energy physics, especially quantum gravity, could freely mess with this. The universe as we know it could still be long-lived, but it could be a lot shorter-lived as well. That in turns makes this calculation a lot more of a practice-ground to hone techniques, rather than an actual estimate you can rely on.

4 gravitons

Stories about physics from someone who's been there