Simulators Lesswrong

[Benita Vandervoort]

Infact, although the *output* tokens are myopic, autoregressive transformers are incentivised to compute activations at early sequence positions that will make them better at predicting tokens at later positions. This may also have indirect impacts on the actual tokens output at the early positions, although my guess would be this isn't a huge effect.

It's true that gradients can flow to causally visible tokens and modify weights to serve future predictions. This does break a type of narrow myopia, but I don't find that break very concerning.

That's a pretty heavy constraint. There often aren't many degrees of freedom for earlier tokens to shift the prediction to serve a later prediction without harming the current one. A well-calibrated predictor can create self-fulfilling or self-easing prophecies while managing locally low loss only when the context permits it.

Another angle: suppose there are two possible predictions P0 and P1 which are estimated to be equivalently and maximally probable and are distinct. We'll assume P1 would make a future token prediction easier.

Despite P1 having a long term advantage, the model cannot simply choose to bias P1's probability up as that would worsen the current prediction's loss in expectation. In order for P1 to be preferred against local pressure, the future expected benefit must swamp the local influence.

Under what circumstances can that occur? In offline training, the model isn't consuming its own outputs. It's being calibrated against an external ground truth. Biasing up P1 isn't helpful during training, because making the future easier to predict in an autoregressive context doesn't reduce loss at training time. Matching the distribution of the input does. In this context, a bias in prediction is almost certainly from undertraining or a lack of capability.

(The other options would tend to be esoteric stuff like "the model is extremely strong to the point of simulated agents managing a reflective gradient hacking attempt of some sort," which doesn't seem to be an obvious/natural/necessary outcome for all implementations.)

In other words, the manner in which GPT-like architectures lack myopia is that the model can learn to predict the future beyond the current token in the service of predicting the current token more accurately.

There's probably some existing terminology for this split that I don't know about, but I might call it something like myopic perception versus myopic action. GPT-likes do not have myopic perception, but they do have myopic action, and myopic action is the important one.

I don't think it has to be in service of predicting the current token. It sometimes gives lower loss to make a halfhearted effort at predicting the current token, so that the model can spend more of its weights and compute on preparing for later tokens. The allocation of mental effort isn't myopic.

As an example, induction heads make use of previous-token heads. The previous-token head isn't actually that useful for predicting the output at the current position; it mostly exists to prepare some handy activations so that induction head can look back from a later position and grab them.

That's an important nuance my description left out, thanks. Anything the gradients can reach can be bent to what those gradients serve, so a local token stream's transformation efforts can indeed be computationally split, even if the output should remain unbiased in expectation.

In general I'm skeptical that the simulator framing adds much relative to 'the model is predicting what token would appear next in the training data given the input tokens'. I think it's pretty important to think about what exactly is in the training data, rather than about some general idea of accurately simulating the world.

I think that the main value of the simulators framing was to push back against confused claims that treat (base) GPT3 and other generative models as traditional rational agents. That being said, I do think there are some reasons why the simulator framework adds value relative to "the model is doing next token prediction":

My guess is you have a significantly more sophisticated, empirical model of LMs, such that the simulators framework feels like a simplification to your empirical knowledge + "the model is doing next token prediction". But I think the simulator framework is valuable because it incorporates additional knowledge about the LM task while pushing back against two significantly more confused framings. (Indeed, Janus makes these claims explicitly in the simulators post!)

I'm not sure this is that important, or that anyone else actually thinks this, but it was something I got wrong for a while. I was thinking of everything that happens at sequence position n as about myopically predicting the nth token.

One was someone saying that they thought it would be impossible to train the model to distinguish between whether it was doing this sort of hallucination vs the text in fact appearing in the prompt, because of an argument I didn't properly understand that was something like 'it's simulating an agent that is browsing either way'. This seems incorrect to me. The transformer is doing pretty different things when it's e.g. copying a quote from text that appears earlier in the context vs hallucinating a quote, and it would be surprising if there's no way to identify which of these it's doing.

if you give it a prompt with some commands trying to download and view a page, and the output, it does things like say 'That output is a webpage with a description of X', when in fact the output is blank or some error or something.

if you give it a prompt with some commands trying to download and view a page, and in reality those commands would return a blank output or some error or something, it does things like say 'That output is a webpage with a description of X'.

(In retrospect this seems like a pretty uncharitable take on something anyone with a lot of experience with language models would find a problem. My guess is that at the time I was spending too much time thinking about what you were saying looked like in terms of my existing ontology and what I would have expected to happen, and not enough on actually making sure I understood what you were pointing at).

Second, I'm not fully convinced that this is qualitatively different from other types of hallucinations, except in that they're plausibly easier to fix because RLHF can do weird things specifically to prompt interactions (then again, I'm not sure whether you're actually claiming it's qualitatively different either, in which case this is just a thought dump). If you prompted GPT with an article on coffee and ended it with a question about what the article says about Hogwarts, the conditional you want is one where someone wrote an article about coffee and where someone else's immediate follow-up is to ask what it says about Hogwarts.

But this is outweighed on the model's prior because it's not something super likely to happen in our world. In other words, the conditional of "the prompt is exactly right and contains the entire content to be used for answering the question" isn't likely enough relative to other potential conditionals like "the prompt contains the title of the blog post, and the rest of the post was left out" (for the example in the post) or "the context changed suddenly and the question should be answered from the prior" or "questions about the post can be answered using knowledge from outside the post as well" or something else that's weird because the intended conditional is unlikely enough to allow for it (for the Hogwarts example).

Put that way this just sounds like it's quantitatively different from other hallucinations, in that information in the prompt is be a stronger way to influence the posterior you get from conditioning. And this can allow us a greater degree of control, but I don't see the model as doing fundamentally different things here as opposed to other cases.

Relatedly, I've heard people reason about the behavior of current models as if they're simulating physics and going from this to predictions of which tokens will come next, which I think is not a good characterization of current or near-future systems. Again, my guess is that very transformative things will happen before we have systems that are well-understood as doing this.

I'm not entirely sure this is what they believe, but I think the reason this framing gets thrown around a lot is that it's a pretty evocative way to reason about the model's behaviour. Specifically, I would be pretty surprised if anyone thought this was literally true in the sense of modelling very low-level features of reality, and didn't just use it as a useful way to talk about GPT mechanics like time evolution over some learned underlying mechanics, and to draw inspiration from the analogy.

For current and near-future models, I expect them to go off-distribution relatively quickly if you just do pure generation - errors and limitations will accumulate, and it's going to look different from the text they were trained to predict. Future models especially will probably be able to recognize that you're running them on language model outputs, and seems likely this might lead to weird behavior - e.g. imitating previous generations of models whose outputs appear in the training data. Again, it's not clear what the 'correct' generalization is if the model can tell it's being used in generative mode.

I agree with this. But while again I'm not entirely sure what Janus would say, I think their interactions with GPT involve a fair degree of human input on long simulations, either in terms of where to prune / focus, or explicit changes to the prompt. (There are some desirable properties we get from a relaxed degree of influence, like story "threads" created much earlier ending up resolving themselves in very unexpected ways much later in the generation stream by GPT, as if that was always the intention.)

3a8082e126