Category Archives: Life as a Physicist

Showing Me Your Amateur Breakthrough Is Evidence That It’s Wrong

Leave a reply

Ok, people who read my post on the Anthropic science blog: we need to talk.

It’s understandable. You saw me write about AI being used to tackle a serious, publishable physics problem. Naturally, some of you want to share that you accomplished something even more impressive with AI, something the scientific journals and arXiv moderators won’t acknowledge. Maybe you’re hoping I’ll write about it, maybe you want me to check your math. Maybe you just think it’s neat.

But here’s the thing: most of the time, I’m not the right person to send those ideas to. And the fact that you didn’t figure that out is a bad sign.

Outside of the movies, scientists are specialists. Not just “physics” or “particle physics” or “theoretical particle physics”, but so specific that they often focus on particular methods, or are geared around particular goals. For any idea you propose, if it’s good, there will be a small list of people who will be excited to see it and able to evaluate it. They’ll be people who are trying to accomplish the same thing, or people who used the methods you’re using for other things. Those people aren’t hard to find, if your idea is in a field you’re actually interested in: they’re the people who write papers on the subject. If you find the topic interesting, you’ll have read their papers, and know who to reach out to.

So even back when I was active as a physicist, there was only a small list of ideas where I’d be the right person to talk to. Now, though, as I mentioned in the Anthropic post, I’m not a scientist anymore. I haven’t been following the field in enough detail to actually verify anything in detail, even on topics I used to research. Part of the punchline of the post was that I don’t have a clear idea of what’s possible in the field right now. So even if you’ve figured out something in scattering amplitudes, you should reach out to someone who’s still researching the subject, a few of whom were even mentioned in the post!

Now, it might be you’re not interested in someone checking your work, and instead you’re writing to me in my capacity as a science journalist, because you feel like you did something newsworthy. But as a science journalist, I’m not supposed to evaluate your work by myself. I’m supposed to reach out to an independent expert, and if I can’t find any who will speak positively about your result, then the editors won’t publish it. So if you think you did something big enough to merit news coverage, the first thing to do is once again, reach out to a scientist who actually has the expertise to evaluate your work, and get them excited about it. Then you two can contact me together!

If you don’t know who to contact, that’s a sign you don’t read papers in the field. It makes it more likely you managed to convince yourself of something nonsensical, because you don’t have the expertise to evaluate it. It’s a pretty clear sign that I shouldn’t trust you or your AI methods about the result.

It’s also just plain bad for you. If you produce a paper without ever wanting to read any, it suggests your priorities are in a dangerous place: that you’re trying to do something that sounds cool and important over doing something you actually care about.

I’m coming from personal experience here.

As an academic, I spent years telling myself that not enjoying reading papers or going to talks was just part of the job. Then it really cost me something, and I realized the job wasn’t worth my superficial desire to be a physicist. So I left.

Then, I spent a year trying to get a job in industry, and was told the same thing. They wanted to hire people who cared about the job. The finance guys wanted to hire people who read books about finance. The data scientists wanted to hire people who did projects in data science. They wanted, if not passion, then at least grace. Someone who could take to the job easily, and naturally.

Now, what I do for a living is something I was doing once a week, for no more incentive than a vague sense of personal obligation, for over a decade. And I’m doing a lot better.

You want to do something impressive with AI? You think you’re a whiz with Claude Code, that your custom workflow is immune to AI hallucinations?

Then use it for something else. Pick something you actually enjoy reading about, not just the pop news, but the real technical literature. Something you don’t need the AI to explain to you. Something that matters to you, in a way that actually counts.

At the very least, it’ll work out better than writing to me.

Data Comes From Papers

Leave a reply

There’s a lovely resource that I suspect most non-physicists don’t know about. It’s called the Particle Data Group, and its interactive site PDGLive.

If you want to know the most up-to-date information on a subatomic particle, PDGLive has your back. From their home page, you can click on familiar particles like photons (\gamma), electrons (e) or gluons and find the best info experiments have provided on properties like their mass and charge. It’s a great place to get an authoritative number for any particle physics property you’re interested in.

Of course, scientists don’t just accept one authoritative answer for anything, even whether Pluto is a planet. That’s why each measurement in PDGLive comes with an arrow you can expand to see a table of past measurements, which you can compare to.

Each measurement comes with a link to a source. And each source is not an experiment webpage, or some entry in a database. It’s a document, a publication, a paper.

In fact, PDGLive itself is just a website version of a paper, the Particle Data Group’s “Review of Particle Physics”. Click enough times, and you’ll find sections of a pdf on the site, explaining their reasoning for picking this experiment over that, emphasizing this or the other thing.

People talk about papers as how academics persuade one another, or how they show off and establish credit. But papers are also just a way to organize data. Each time an experiment figures something out, all of their reasoning and procedures are summarized together with the numbers they got. And anyone who reports those numbers to you will include a link, so you can go back, and check where the number came from.

It probably feels a bit weird, all of these numbers and technical details bottoming out in an archaic practice of writing down words for other human beings. But it means that all of the richness of the process is there somewhere, linked together and collated by the same social forces that keep track of credit, all in one navigable whole.

So when you run into a number, spare some thought for where it came from. You can probably find out.

Better Bounds

Leave a reply

I swear this isn’t turning into an AI blog. But did you see the one about the Riemann hypothesis?

Someone at Anthropic did something I’m sure they’re all tempted to do, and tried to use an internal version of their Claude AI system to prove the most famous open conjecture in mathematics. It didn’t work, to be clear, and I get the impression they didn’t expect it to. But out of six hundred or so fruitless tries, one attempt did prove a new bound. Previously, mathematicians had been able to prove that at least 41.6% of the zeroes of the Riemann zeta function satisfied the Riemann hypothesis. Now, the new proof shows that at least 67.2% satisfy it.

Anthropic’s press release is impressively careful. As someone who’s had to think about how to write content that both excites the public and doesn’t piss off experts too much, they do an admirable job walking that line. They even say, straight-out, “We don’t expect that the techniques Claude used will lead to proving the Riemann hypothesis.”

Bounds are like that, sometimes.

I should know. Physicists also find bounds.

Physics has its own conjectures with the fame of the Riemann hypothesis. Dark matter might be made of detectable particles. Protons could decay. There might be extra dimensions, or magnetic monopoles, or cosmic strings. General relativity might be subtly wrong.

It would be an amazing achievement to demonstrate any of these things. But most physicists won’t manage that. Instead, they bound them.

Physicists compete to get better bounds, excluding unusual possibilities with greater and greater care. They find evidence that dark matter can’t be of a specific mass with a specific charge, so the next experiment has to look somewhere else, or find evidence that general relativity holds to even greater precision, so any deviation must be even smaller. Some work to improve experiments with better and better bounds. Others analyze data from older experiments, or find under-appreciated consequences of known facts, and can get even better bounds.

Bounds aren’t typically newsworthy (though occasionally they make it through), so most people don’t hear about them. If you read the news, you hear about positive claims much more often than negative ones: evidence for something new, not evidence that our current knowledge holds. But the nature of physics is that most work supports the status quo. Most work improves bounds.

Do bounds lead, with time, to the positive claims? Sometimes, but not always. Often, bounds are just bounds. They’re attempts to use the methods physicists have to learn something new about the world. Even if the new fact is just “don’t look here”.

Should You Read What You Cite? That Depends

Leave a reply

When arXiv announced it would ban people for hallucinated citations, that is citations of papers that don’t exist, the discussion online got sidetracked by the question of whether academics actually read the papers they cite. Some people proudly insisted that any good scholar always reads every paper they reference, others argued that was ridiculous.

As always, the answer is never that simple. In certain fields, it is enormously important to read the papers you cite if you want to do solid, careful, scholarly work. In others, it’s entirely irrelevant.

It mostly comes down to what citations are for. And luckily, I’ve already written a post about that.

So let’s go through the citation motivations I mention in that post.

First, some citations are about respecting priority, feeding the system by which academics get credit for having an idea first. The incentive system of academia depends on getting this more or less right, but that doesn’t mean every academic has to check things at every step of the way. Besides, if you get this wrong, you’ll find out quickly. Submit a paper to a preprint server like arXiv, and you’ll be sure to get emails telling you that some obscure Soviet researcher figured it all out first.

Other citations are about substantiating claims. These are the most important to get right. Here, you really ought to have read, if not the whole paper, at least the full justification for the claim you’re making. You can have some leeway if the methods are unfamiliar enough, for example a complicated experiment you can’t understand all the details of. Science and technology do require some trust. But you should have at least a sense of where things could go wrong, and why.

Citations to provide context are a different beast. Here, you’re trying to tell a reader where your ideas come from. You can’t show them the conversations you have with your colleagues, the things they value and get you excited about. So you have to show them papers instead. But the papers aren’t the thing you read, they’re just a convenient proxy.

Finally, citations do sometimes just exist to follow social conventions. And yeah, you don’t have to read these, just like you don’t have to say how you’re doing when someone asks you how you’re doing. They’re the academic equivalent of social white lies, and should be taken roughly as seriously, both by their supporters and detractors.

Doing Things Well Is an International Activity

4 Replies

In the US, funding agencies seem to be increasingly opposed to an often inevitable feature of good science: international collaboration. Scientists have been told by officials at the National Institutes of Health that they need to remove mention of foreign collaborators from progress reports, or that they need to avoid such collaborations to begin with. At NASA, officials have told scientists that rather than just avoiding funding work in China, they should actively avoid collaborating with Chinese researchers. And a recently introduced bill would make that restriction more explicit.

I have a general policy against discussing concrete political issues on this blog, so I’m not going to dig into the details of who’s doing what here, how far it’s going or how novel it is. That policy extends to the comments. If you mention specific laws, politicians, or political parties, I will delete your comment.

I do want to say something more general, though. I think people often underestimate just how important international collaboration is.

I’ve talked before about how scientific specialization spreads scientists around the world. Scientists want to work with people who work on their specific interests, and there are often only a few people that fit that description. So people move across the world, creating centers of expertise.

More than that, though, essentially any activity, done well, is done internationally. The better you want to perform, the more likely it is that the best collaborator will be someone in another country.

People don’t notice this as much as they could, because they’re used to the exceptions. Popular art is often siloed by language and cultural references. Sports are intentionally set up as competitions between regions and nations, and militaries compete as a practical necessity. But without those exceptions, international competition wins out. The best doctor, the best classical musician, and the best businessperson for a job can’t be expected to come from one country or another. Those fields, like science, are international.

When that internationalism is weak, it’s a warning sign. Without that drive to succeed on an international stage, scientists get lazy. There are countries with a history of academic cronyism, where universities were run more on interpersonal politics than scholarly merit, cozy fiefdoms where prominent academics dole out positions. To combat this, policymakers work to make their research systems more international. They explicitly ask about international collaborations and participation in international conferences in grant applications, not to discourage them, but to encourage them: to reward academics who show merit on the international stage and break up lazy patronage networks.

It worries me that it sounds like some US policymakers want to do the opposite. People are increasingly worried about bias and groupthink in the sciences, and increasingly mad that scientists could be wasting the public’s money to maintain a cushy lifestyle. International collaboration is how you hold scientists to account, how you force them to compete and show their merit. If you drop that, academia is going to get a whole lot worse.

ArXiv Will Ban You for Hallucinated References

7 Replies

Thomas Dietterich, Chair of the Computer Science section of the preprint server arXiv.org, recently clarified the site’s policies towards “hallucinated” citations and other signs of careless use of AI in a post on X. If your paper contains a citation to a paper that doesn’t actually exist, or has other signs you didn’t read it before posting like leftover commentary (the example he gave was “here is a 200 word summary; would you like me to make any changes?”), then you can get banned from the arXiv for one year. Even after that year you’d be on a kind of “probation”, and would need to show that your next few papers had been accepted by peer-reviewed journals first before posting them.

At the risk of saying the obvious, this is a good idea! arXiv isn’t peer review, it isn’t meant to judge the value of the papers it hosts. But it still needs to be a useful place for scientists to post their papers, which is why they try to keep spam and irrelevant content to a minimum. If you don’t actually endorse the content of a paper, you shouldn’t post it in the first place.

That said, the whole existence of hallucinated citations on arXiv feels a little silly. It makes sense for academic journals and preprint servers in other fields. But arXiv was the first site of its kind for a reason. Its users, physicists, mathematicians, and computer scientists, don’t need much hand-holding when it comes to computers. Papers submitted to arXiv aren’t typically written in Word, they’re written in a document-writing language called LaTeX, that lets users make decently-formatted papers without help from a journal. Physicist-written code may be terrible by any reasonable criteria…but it exists, much more universally than for example biologist-written code.

This extends to citations. In my old field, there is a database called INSPIRE that updates automatically from arXiv. Click on a paper, and a handy “cite” link gives you standardized citations in several formats, ready to copy and paste into your LaTeX code. Nearly every citation in my papers is copied from there. The ones that aren’t are either from other fields where I didn’t know of that style of database, or things that haven’t been published (this can be manuscripts in preparation, or personal communications).

All of this, though, feels like a lot less than what the field could be doing. In a world where almost everyone posts their papers to the same website, and almost everyone has at least a rudimentary understanding of programming…why are people still writing citations in free-form text in the first place? Why aren’t citations built in to the submitted papers on arXiv, automatically linked to the papers they cite? Why don’t we have a setup where, except for a small number of “special” citations, every citation is built so that it automatically goes to a real paper, and gives a clear error message if it doesn’t? In short, why are hallucinated citations even possible?

Look, I’m naive, I get that. I believe in automation, not in the modern context of LLMs and other heuristics, but in setting clear procedures and building clear rules. The world doesn’t work that way! The clear rules are always more contentious than you expect, the fuzzy human-led version always the only choice people can agree on.

But still. Citations. There has to be a better system, right?

ArXiv to Leave Cornell

Leave a reply

Yes, I’m late to the party on this one.

A few weeks ago, arXiv.org announced that it will be leaving Cornell, the university that currently manages it, and establishing its own nonprofit.

arXiv is a crucial part of the infrastructure for physics, mathematics, computer science, and a few related fields. Researchers post papers to arXiv as what are called “preprints” before the papers are submitted to a journal. In practice, nobody ends up reading the journal versions: the arXiv is free to access, and typically reflects better what the paper’s authors want the paper to look like. So in practice, arXiv is how researchers in these fields communicate, which makes its role enormously important.

If you’re from another field, you might wonder how something like arXiv is financially sustainable. The answer is that it works better than you’d think, but not perfectly. They’ve been supported by philanthropy, in addition to Cornell, and while there have apparently been budget shortfalls and drama behind the scenes, But nonetheless, arXiv has stayed in continuous operation since 1991.

The move to an independent nonprofit is supposed to make it easier for arXiv to get philanthropic funding, which otherwise needed to be filtered through Cornell in ways that were sometimes opaque or didn’t give donors the control they wanted.

While it wasn’t mentioned in the announcements, I suspect another motivation is security. Universities are fixed in place, and that makes them easier to pressure. For an organization that wants to process scientific output in an unbiased way, the link to Cornell represented a vulnerability. It’s not a vulnerability that has mattered yet, and likely didn’t seem like it would ever matter. But it wouldn’t surprise me if they’re more worried now that someone might try to pressure Cornell in order to change how arXiv operates. For critical scientific infrastructure, it’s important to be as independent of those kinds of pressure as possible.

The Twitter of Physics

4 Replies

The paper I talked about last week was frustratingly short. That’s not because the authors were trying to hide anything, or because they were lazy. It’s just that these days, that’s how the game is played.

Twitter started out with a fun gimmick: all posts had to be under 140 characters. The restriction inspired some great comedy, trying to pack as much humor as possible into a bite-sized format. Then, Twitter somehow became the place for journalists to discuss the news, tech people to discuss the industry, and politicians to discuss politics. Now, the length limit fuels conflict, an endless scroll of strong opinions without space for nuance.

Physics has something like this too.

In the 1950’s, it was hard for scientists to get the word out quickly about important results. The journal Physical Review had a trick: instead of normal papers, they’d accept breaking news in the form of letters to the editor, which they could publish more quickly than the average paper. In 1958, editor Samuel Goudsmit founded a new journal, Physical Review Letters (or PRL for short), that would publish those letters all in one place, enforcing a length limit to make them faster to process.

The new journal was a hit, and soon played host to a series of breakthrough results, as scientists chose it as a way to get their work out fast. That popularity created a problem, though. As PRL’s reputation grew, physicists started trying to publish there not because their results needed to get out fast, but because just by publishing in PRL, their papers would be associated with all of the famous breakthroughs the journal had covered. Goudsmit wrote editorials trying to slow this trend, but to no avail.

Now, PRL is arguably the most prestigious journal in physics, hosting over a quarter of Nobel prize-winning work. Its original motivation is no longer particularly relevant: the journal is not all that much faster than other journals in its area, if at all, and is substantially slower than the preprint server arXiv, which is where physicists actually read papers in practice.

The length limit has changed over the years, but not dramatically. It now sits at 3,750 words, typically allowing a five-or-six page article in tight two-column text.

If you see a physics paper on arXiv.org that fits the format, it’s almost certainly aimed at PRL, or one of the journals with similar policies that it inspired. It means the authors think their work is cool enough to hang out with a quarter of all Nobel-winning results, or at least would like it to be.

And that, in turn, means that anyone who wants to claim that prestige has to be concise. They have to leave out details (often, saving them for a later publication in a less-renowned journal). The results have to lean, by the journal’s nature, more to physicist-clickbait and a cleaned-up story than to anything their colleagues can actually replicate.

Is it fun? Yeah, I had some PRLs in my day. It’s a rush, shining up your work as far as it can go, trimming down complexities into six pages of essentials.

But I’m not sure it’s good for the field.

Most Academics Don’t Choose Their Specialty

4 Replies

It’s there in every biography, and many interviews: the moment the scientist falls in love with an idea. It can be a kid watching ants in the backyard, a teen peering through a telescope, or an undergrad seeing a heart cell beat on a slide. It’s a story so common that it forms the heart of the public idea of a scientist: not just someone smart enough to understand the world, but someone passionate enough to dive in to their one particular area above all else. It’s easy to think of it as a kind of passion most people never get to experience.

And it does happen, sometimes. But it’s a lot less common than you’d think.

I first started to suspect this as a PhD student. In the US, getting accepted into a PhD program doesn’t guarantee you an advisor to work with. You have to impress a professor to get them to spend limited time and research funding on you. In practice, the result was the academic analog of the dating scene. Students looked for who they might have a chance with, based partly on interest but mostly on availability and luck and rapport, and some bounced off many potential mentors before finding one that would stick.

Then, for those who continued to postdoctoral positions, the same story happened all over again. Now, they were applying for jobs, looking for positions where they were qualified enough and might have some useful contacts, with interest into the specific research topic at best a distant third.

Working in the EU, I’ve seen the same patterns, but offset a bit. Students do a Master’s thesis, and the search for a mentor there is messy and arbitrary in similar ways. Then for a PhD, they apply for specific projects elsewhere, and as each project is its own funded position the same job search dynamics apply.

The picture only really clicked for me, though, when I started doing journalism.

Nowadays, I don’t do science, I interview people about it. The people I interview are by and large survivors: people who got through the process of applying again and again and now are sitting tight in an in-principle permanent position. They’re people with a lot of freedom to choose what to do.

And so I often ask for that reason, that passion, that scientific love at first sight moment: why do you study what you do? It’s a story that audiences love, and thus that editors love, it’s always a great way to begin a piece.

But surprisingly often, I get an unromantic answer. Why study this? Because it was available. Because in the Master’s, that professor taught the intro course. Because in college, their advisor had contacts with that lab to arrange a study project. Because that program accepted people from that country.

And I’ve noticed how even the romantic answers tend to be built on the unromantic ones. The professors who know how to weave a story, to self-promote and talk like a politician, they’ll be able to tell you about falling in love with something, sure. But if you read between the lines, you’ll notice where their anecdotes fall, how they trace a line through the same career steps that less adroit communicators admit were the real motivation.

There’s been times I’ve thought that my problem was a lack of passion, that I wasn’t in love the same way other scientists were in love. I’ve even felt guilty, that I took resources and positions from people who were. There is still some truth in that guilt, I don’t think I had the same passion for my science as most of my colleagues.

But I appreciate more now, that that passion is in part a story. We don’t choose our specialty, making some grand agentic move. Life chooses for us. And the romance comes in how you tell that story, after the fact.

How Much Academic Attrition Is Too Much?

4 Replies

Have you seen “population pyramids“? They’re diagrams that show snapshots of a population, how many people there are of each age. They can give you an intuition for how a population is changing, and where the biggest hurdles are to survival.

I wonder what population pyramids would look like for academia. In each field and subfield, how many people are PhD students, postdocs, and faculty?

If every PhD student was guaranteed to become faculty, and the number of faculty stayed fixed, you could roughly estimate what this pyramid would have to look like. An estimate for the US might take an average 7-year PhD, two postdoc positions at 3 years each, followed by a 30-year career as faculty, and estimate the proportions of each stage based on proportions of each scholar’s life. So you’d have roughly one PhD student per four faculty, and one postdoc per five. In Europe, with three-year PhDs, the proportion of PhD students decreases further, and in a world where people are still doing at least two postdocs you expect significantly more postdocs than PhDs.

Of course, the world doesn’t look like that at all, because the assumptions are wrong.

The number of faculty doesn’t stay fixed, for one. When population is growing in the wider world, new universities open in new population centers, and existing universities find ways to expand. When population falls, enrollments shrink, and universities cut back.

But this is a minor perturbation compared to the much more obvious difference: most PhD students do not stay in academia. A single professor may mentor many PhDs at the same time, and potentially several postdocs. Most of those people aren’t staying.

You can imagine someone trying to fix this by fiat, setting down a fixed ratio between PhD students, postdocs, and faculty. I’ve seen partial attempts at this. When I applied for grants at the University of Copenhagen, I was told I had to budget at least half of my hires as PhD students, not postdocs, which makes me wonder if they were trying to force careers to default to one postdoc position, rather than two. More likely, they hadn’t thought about it.

Zero attrition doesn’t really make sense, anyway. Some people are genuinely better off leaving: they made a mistake when they started, or they changed over time. Sometimes new professions arise, and the best way in is from an unexpected direction. I’ve talked to people who started data science work in the early days, before there really were degrees in it, who felt a physics PhD had been the best route possible to that world. Similarly, some move into policy, or academic administration, or found a startup. And if we think there are actually criteria to choose better or worse academics (which I’m a bit skeptical of), then presumably some people are simply not good enough, and trying to filter them out earlier is irresponsible when they still don’t have enough of a track record to really judge.

How much attrition should be there is the big question, and one I don’t have an answer for. In academia, when so much of these decisions are made by just a few organizations, it seems like a question that someone should have a well-considered answer to. But so far, it’s unclear to me that anyone does.

It also makes me think, a bit, about how these population pyramids work in industry. There there is no overall control. Instead, there’s a web of incentives, many of them decades-delayed from the behavior they’re meant to influence, leaving each individual to try to predict as well as they can. If companies only hire senior engineers, no-one gets a chance to start a career, and the population of senior engineers dries up. Eventually, those companies have to settle for junior engineers. (Or, I guess, ex-academics.) It sounds like it should lead to the kind of behavior biologists model in predators and prey, wild swings in population modeled by a differential equation. But maybe there’s something that tamps down those wild swings.