Sometimes, when I write a post about AI, I’ve been sitting on an idea for a long time. I’ve talked to experts, I’ve tried to understand the math, I’ve honed my points and cleared away clutter.
This is not one of those times. The ideas in this post almost certainly have something deeply wrong with them. But hopefully they’re interesting food for thought.
My first dumb idea: instruction tuning was a mistake.
I’m drawing the seeds of this one from a tumblr post by nostalgebraist, someone known for making a popular bot trained on his tumblr posts in the early days before GPT became ChatGPT.
AIs like ChatGPT are based on Large Language Models, insanely complicated mathematical formulas that predict, given part of a text, what the rest of that text is likely to look like. In the early days, this was largely how they were used. Loosely described nostalgebraist’s bot, called nostalgebraist-autoresponder, began with a list of tumblr posts and asks and determines what additional posts would best fit in.
If you think about it, though, ChatGPT doesn’t really work like that. ChatGPT has conversations: you send it messages, it sends you responses. The text it creates is a dialogue, with you supplying half the input. But most texts aren’t dialogues, and ChatGPT draws on a lot of non-dialogue texts to make its dialogue-like responses.
The reason it does this is something called instruction tuning. ChatGPT has been intentionally biased, not to give the most likely completion to a task in general, but to give completions that fit this dialogue genre. What I didn’t know until I read nostalgebraist’s post was that this genre was defined artificially: AI researchers made up fake dialogues with AI, cheesy sci-fi conversations imagining how an AI might respond to instructions from a user, and then biased the Large Language Model so that rather than giving the most likely text in general, it gives a text that is more likely to look like these cheesy sci-fi conversations. It’s why ChatGPT sounds kind of like a fictional robot: not because sci-fi writers accurately predicted what AI would sound like, but because AI was created based on sci-fi texts.
For nostalgebraist, this leads into an interesting reflection of how a sci-fi AI should behave, how being warped around a made-up genre without history or depth creates characters which act according to simple narratives and express surprising anxiety.
For myself, though, I can’t help but wonder if the goal of dialogue itself is the problem. Dialogue is clearly important commercially: people use ChatGPT because they can chat with it. But Large Language Models aren’t inherently chatbots: they produce plausible texts, of any sort you could imagine. People seem to want a machine that can, for example, answer scientific questions as part of a conversation. But most competent answers to scientific questions aren’t conversations, they’re papers. If people stuck with the “raw” model, producing excerpts of nonexistent papers rather than imitating a dialogue with a non-existent expert, wouldn’t you expect the answers to be more accurate, with the model no longer biased by an irrelevant goal? Is the need to make a sell-able chatbot making these AIs worse at everything else people are trying to use them for?
I’m imagining a world where, instead of a chatbot, OpenAI built an “alternate universe simulator”. You give it some context, some texts or parts of texts from a universe you made up, and it completes them in a plausible way. By imagining different universes, you can use it to answer different questions. Such a gimmick would get fewer customers, and fewer investors, it would probably do worse. But I have to wonder if the actual technology might have been more useful.
My second idea is dumber, to the point where I mostly know why it doesn’t work. But thinking about it might help clarify how things work for people unused to AI.
I saw someone point out that, unlike something like Wikipedia, AI doesn’t give you context. You shouldn’t trust Wikipedia, or a source you find on Google, blindly. If you want to, you can look through the edit history on Wikipedia, or figure out who wrote a page you found on Google and how. If ChatGPT tells you something, by default you don’t know where that knowledge came from. You can tell it to search, and then you’ll get links, but that’s because it’s using Google or the like behind the scenes anyway. You don’t know where the model is getting its ideas.
Why couldn’t we get that context, though?
Every text produced by a Large Language Model is causally dependent on its training data. Different data, different model, different text. That doesn’t mean that each text draws from one source, or just a few sources: ChatGPT isn’t copying the training data, at least not so literally.
But it does mean that, if ChatGPT says something is true, you should in principle be able to ask which data was most important in making it say that. If you leave a piece of data out of the training, and get similar answers, you can infer that the response you got doesn’t have much to do with that piece of data. But if you leave out a text in training, and now ChatGPT gives totally different responses to the same question…then there’s a pretty meaningful sense that it got the information from that source.
If this were the type of non-AI statistical model people use in physics, this would be straightforward. Researchers do this all the time: take one experiment out of the data, see how their analysis changes, and thereby figure out which experiments are most important to check. One can even sometimes calculate, given a model, where you should look.
Unfortunately, you can’t do this with ChatGPT. The model is just too big. You can’t calculate anything explicitly about it, the giant mathematical formulas behind it are so complicated that the most you can do is get probabilities out case by case, you can’t “unwind” them and see where the numbers come from. And you can’t just take out sources one by one, and train the model again: not when training takes months of expensive computer time.
So unlike with the previous idea, I understand even on a technical level why you can’t do this. But it helped me to be able to think about what I would like to do, if it were possible. Maybe it helps you too!
4 gravitons 
Re: your second idea — yes, that would be very informative if it were feasible, and yes, it’s not feasible in practice with large models.
The ML data attribution literature refers to it as “leave-one-out (LOO) retraining,” and it’s the intractable gold-standard which that literature tries to approximate with cheaper methods.
Unfortunately, I’m not aware of any cheap/feasible approximate methods with convincing theoretical backing.
The method that everyone talks about is called “influence functions,” which (roughly speaking) treats the optimum found by training without the data point as a small perturbation of the optimum found with the data point. Solving for the perturbed optimum to first order requires taking 1st and 2nd derivatives of the loss function, which is more expensive than a single gradient descent step (since gradient descent only needs the 1st derivative), but less expensive than an entire run of gradient descent.
Influence functions can be shown to approximate LOO retraining under some unrealistic/idealized assumptions. But in the general case, those assumptions don’t hold.
In this interesting paper, the authors ask “OK, so if influence functions don’t approximate LOO retraining in the general case, what other thing are they approximating?” The answer turns out to be this gnarly-looking formula which the authors call the “proximal Bregman response function” (PBRF).
But I’ve never seen anyone present a convincing case for why I should want to approximate the PBRF, or care about it at all – it’s not some pre-existing object of interest, it’s just the thing that influence functions happen to be approximating. (Nevertheless, I still see papers going ahead and computing influence functions as if they were approximations of LOO retraining, with maybe an apologetic caveat about the PBRF somewhere in an intro/conclusion/appendix. Maybe I’m missing something here, IDK.)
Anyway, just thought you might find that interesting!
LikeLike
Thanks, this is interesting indeed!
LikeLike
These are good ideas, but I wouldn’t write off chatbots for original science. Papers date from the pre-digital era and could use some updates. An unlimited Q&A with the author sounds nice! (Especially for embarrassing undergraduate level follow-up questions.)
I suspect CS people might start doing this before anyone else. They already sometimes build websites to accompany their papers, with complex online-only visualizations (e.g. sliders), an easy Pareto improvement over the status quo.
LikeLike
I’m not sure instruction tuning helps with that kind of thing! I think instead of a weird layered thing where you’ve got an LLM fine-tuned to act like ChatGPT acts, and then you ask it to roleplay as a dead paper author, you could just prompt an LLM with the beginning of a text that looks like a transcript of a Q&A with the author. (People sometimes publish historical papers together with transcripts from conferences, so that’s not even that unusual a format.) It would sometimes make up “your side of the conversation”, but while that’s disconcerting for a lay user it shouldn’t be a big problem for a scientist. Sometimes it might even suggest good questions to ask!
LikeLike
The training material for an LLM was the Internet, so ideally it can look up sources, refine its response and give you links. You just need to tell it to.
For the first part about instruction tuning, that probably has loads of room for improvement. Specifically for technical tasks like proving a theorem or writing code that passes given tests. We still use general purpose massive models for most tasks, while smaller fine tuned models could give better results far cheaper and faster.
LikeLike
I don’t think LLMs “come with” a complete representation of their training data in the way you’re imagining. It’s generalizing/interpolating the internet, certainly, but that doesn’t mean it can directly retrieve things like that. The impression I had is that when you ask a chatbot to look up sources, it’s actually using a traditional search engine. Otherwise it can give you links, but it will give you the links that are statistically likely to show up in an article about the topic, or links that resemble them. In addition to fake links, they can easily be links to things that weren’t in the training data.
LikeLike
Certainly, but it can be instructed to find links to verify what it says. Most of what it says should, if true, be discoverable on the public Internet, so if it can’t find sources to back up what it says that’s likely incorrect.
I’m imagining a flow like:
generate response -> search the internet for sources -> modify the response to only include sourced information -> add links to sources like a Wikipedia article.
LikeLike
Yeah, the pair “search the internet for sources -> modify the response to only include sourced information” is the question-begging step here. If you had a reliable algorithm to determine which text summaries of a source were accurate, you wouldn’t need the LLM in the first place! You can certainly have it only include information likely to come after, for example, the text “[text of the source], which shows that,” but not all text of that form is actually going to be inferred from that text. You can still easily have the LLM be drawing a conclusion that it came to due to some other piece of evidence that isn’t mentioned in the source.
LikeLike