Goodfire 的 Eric Bigelow：大模型如何「做决定」，关键 token 与相变

Nathan Labenz

一个名字在那次综述里反复出现，就是 Eric。他最近在哈佛心理学系完成了博士学位，论文题为《迈向大语言模型的认知科学》；一些学术同行起初对他转向 LLM 有疑问，而如今连精英学术机构也愿意把 AI 当作某种「心智」来研究，这一点我觉得非常值得注意。

这场对话和我最近的其他对话给我的底线结论是：思维链越来越不可靠；而且这些问题似乎不仅与 RL 训练相关，也与原始预训练模型本身的能力相关，所以这个趋势可能很难逆转。这意味着，包括我们理解智能体为什么采取某些行动的能力，在很大程度上可能要押在可解释性上。

One name kept coming up in that thread, and it was Eric. Eric recently completed a PhD in the Harvard Psychology Department, with a dissertation titled Toward a Cognitive Science of Large Language Models. And while some of his academic colleagues initially questioned his pivot to focus on LLMs, the fact that even elite academic institutions are now willing to engage with AIs as a sort of mind strikes me as very notable indeed.

The bottom line from this conversation and others I’ve had recently is that Chain of Thought is increasingly unreliable, and because these issues seem to correlate not just with RL training, but with the power of the raw pre-trained model itself, this trend might be quite difficult to reverse. That means an awful lot, including our ability to understand why agents take the actions they do, may rest on interpretability.

Nathan Labenz

我想从你的研究史聊起，也想得到一个最新的理解：这些东西到底是怎么做决定的？我们对此了解什么，又打算怎样更好地理解它？

What I hope to get out of this conversation is a little intellectual history of your work and the up-to-date understanding of how these things make decisions. What do we know about that, and what is our plan to figure out how to understand it better?

Eric Bigelow

我先快速回顾一下对我理解 LLM 最有启发的项目：Forking Paths in Neural Text Generation。那是两年前，用的是 GPT-3.5 系列的一个模型。我们让它一步步推理，然后在推理中的每一个 token 处重采样：每个 token 处采 30 条不同的后续，再看每条最终落在什么答案上，把这些聚合成分布来分析。

最醒目的是，把它排成多变量时间序列时，会看到类似 grokking 里「训练中突然学会」那样的尖锐相变：推理中的某个点，模型原本 50% 倾向这个答案、30% 倾向那个答案，然后分布突然剧变，往往直接塌缩到某个最终答案——可以说模型「决定」了接下来说什么，不过我给「决定」加了引号。

有时这个点是你按评委思路能猜到的，比如推理链里第一次提到最终答案；但也有时分布在一个很随意的词上剧变，比如论文里的例子：「kilowatt hours」后面跟的左括号。你绝不会预期它改变最终答案，但在这个确切的 token 处重采样，如果换成别的词，最后答案就会不同。

I’ll give a quick review of a project that is really one of the most informative on how I think about how LLMs work at a basic level: Forking Paths in Neural Text Generation. This was two years ago, with one of the GPT 3.5 models. We take reasoning chains, give it a question and a chain of thought prompt, and have it reason step by step. Then at every single token in its reasoning, we resample 30 different rollouts, and from every rollout we see what final answer it landed on, and aggregate these into distributions.

If you arrange it like a multivariate time series, you see these really interesting dynamics. I take inspiration from the learning dynamics work around grokking, where there are these really sharp phase transitions. Over the course of reasoning, there will be certain points where the model was initially at a 50% chance of this answer and 30% of that answer, and then suddenly the distribution changes really dramatically. Often it collapses onto a final answer as the model, you might say, decides what it’s going to say next — although I use the word “decides” with scare quotes.

There are certain keywords where, if you looked at it like a judge would, you might guess it’s the first time a final answer is mentioned. But other times the distribution changes really dramatically on some arbitrary word, even an open parenthesis — one example in the paper is “kilowatt hours, open parentheses, KWH”. You never expect that to change how the final answer plays out, but if you go to that exact token and resample a bunch of generations, in cases where it’s not an open parenthesis, it ends up as a different final answer.

Eric Bigelow

所以我的简单答案是：这个决定真正发生在采样环节。模型生成的每个 token 都有一次「抛硬币」，它在从候选 token 的分布里采样；而采到什么，决定了它随后如何通过上下文学习，生成一段与之一致的推理。我对机制可解释性在这个问题上能带来更多启发相当乐观。

现代推理模型和当时的思维链有所不同。它们常常做「线性化的搜索」：在上下文里枚举各种可能，最后回头挑一个。我做 forking 分析发现不确定性动态更平滑一些，但仍有分叉点——枚举了很多条链而没有哪条明显更有希望时，最后选 B 还是 C 仍可能取决于采样结果。我另一个项目还发现，在一些玩具场景里有神经元似乎在追踪模型的置信度，甚至可以通过引导让模型整体更自信或更不自信。

The simple answer I have is that this decision really happens during sampling. Every token generated is a coin flip — it’s sampling from a distribution of alternate tokens — and depending on what token is sampled, it in-context learns and then generates a set of reasoning that is consistent with that. I’m pretty bullish on mech interp shedding more light on really answering this question of how decisions are made.

Modern reasoning models are a little different than chain of thought was then. They often do a really linearized search where they enumerate all these different possibilities in context, and at the end they go back and pick one of the things they said before. The uncertainty dynamics on reasoning models are often a bit smoother, but there are still forking points — if it enumerates a lot of chains and none seems more promising, the answer therefore is B or C could actually have been different depending on what was generated. Another project of mine has shown, in some toy domains, neurons that seem to track model confidence, where you can even steer the model to be more or less confident.

Nathan Labenz

这里有几个方向想追问。先从上下文学习与其他东西的关系谈起：已有不少工作看到上下文学习与微调之间某种近似同构的关系，尤其是在 LoRA 这种可以把权重变化局部化的微调上。我们该怎么理解这种关系？

There’s been a lot of work that’s looked at a sort of isomorphic relationship between in-context learning and fine tuning, especially at LoRA scale fine tuning where you can localize the weight changes. How should we understand that relationship?

Eric Bigelow

我觉得最不显然的一点是：历史上人们说上下文学习，指的是给几个输入输出示例，看模型行为怎么变，并画出示例数增加时性能提升的曲线。但我把上下文学习理解为：模型为了调整自身行为所做的一切、且并不发生在权重学习里的事情。这包括它读每个新词时表征如何变化，也包括推理——它生成文本，再用这些文本决定后续行为。

即使你把模型微调得再狠，提示也不会消失：你仍然希望它在第一次读一个新故事时，能在内部不断更新对「谁是好人坏人、谁高兴谁难过」的信念。我有一篇论文就研究这种故事中的信念更新与表征变化，灵感来自 Kurt Vonnegut 那场关于「故事的形状」的演讲。所有给模型的上下文都有动态的、随时间展开的形状。

The most important thing that may not be obvious is that when people say in-context learning historically, it’s meant giving a few shots of input-output examples and seeing how behavior changes. The way I think of in-context learning is capturing everything the model is doing to adapt its behavior that’s not happening within weights learning. This is how it’s changing its representations as it reads each new word of input text, and it also applies to reasoning, where it’s generating text and then using that to determine its following behavior.

Even if you fine-tune a model enormously, you still want it, when reading a story for the first time, to be updating its beliefs internally about who are the good guys and the bad guys, who is happy and who is sad at different times. I have another paper around this belief updating and representation change in stories, inspired by that famous talk by Kurt Vonnegut on the shape of stories. All context that we give models also have shapes that are dynamic and temporal.

Eric Bigelow

我认为 LLM 所做的一切都是动态发生的：更新信念、转移内部表征、在上下文里学到新的表征，而这不只发生在你给出几个示例的时候，而是时时刻刻——在整段对话中、在句子内部、在推理或生成很长的输出期间。很多可解释性工作错过了这些动态变化：它们通常用模板，只改输入里的几个变量，再看某一个 token 或输出变量上的行为。

学习曲线在大数据集上看起来很平滑，但具体到单个问题，可能出现尖锐的相变。Neel Nanda 有句话：如果你看得足够仔细，相变无处不在。上下文学习也一样：汇总起来可能平滑得有欺骗性，但在单个故事上，一个 token 或一句话之后曲线就可能急剧改变，信念也会起起伏伏、拐来拐去。

我很想知道，那位读过大量思维链的人，有没有对同一模型的重采样输出做过比较。有时你只有看大量对同一个提示的重采样，才能看出一个问题上模型思路的「复杂地貌」。比如我最初那个研究只是让模型「讲一个叫 John 的男孩的故事」，结果一半的故事里 John 有一条狗，去冒险，很多还会发现池塘或瀑布。

Everything that LLMs are doing is happening dynamically. They’re updating their beliefs, shifting representations internally, they can learn new representations in context. This doesn’t just happen when you give a few input-output examples; it’s happening all the time, over a conversation, within sentences, during the period where a model is reasoning or generating a really long output. A lot of interpretability work focuses on templates where just a couple variables are changed in the input and behavior is measured on one token or one output variable.

If you look at learning curves over a huge data set, they’ll be very smooth. But specific problems can change very sharply — I think it was Neel Nanda who had this quote that phase transitions are everywhere if you just looked closely enough. In-context learning is like that too: in aggregate things might look smooth, maybe deceptively so, but individual cases like an individual story have really nonlinear dynamics, and curves can change very sharply after even just one token or one sentence.

I’m really curious if the person you had earlier who’s looked at all these chains of thought has looked at resampled outputs for the same model. For an individual prompt there’s this really elaborate landscape in how the model will think about one problem, and sometimes you can only get this by looking at a lot of resampled outputs. I started that Forking Paths project with “tell me a story about a boy named John”, and half the stories involve John having a dog and going on an adventure, and a lot of those involve them eventually finding a pond or a waterfall.

Nathan Labenz

其中一个大的推动方向是持续学习。你理解上下文学习的经验，对完整的持续学习会带来多大额外挑战，有什么启示？

One big thing pushing toward is continual learning. What does your experience trying to understand in-context learning suggest about how much additional challenge full continual learning might present?

Eric Bigelow

这取决于持续学习最终如何实现，这方面我没有深入研究。但如果每个人的模型都被微调，关键问题是表征变了多少。我自己和 Goodfire 同事的一些工作表明，模型的表征空间由一些低维流形描述，表征很大程度由低维的概念结构定义。只要这套结构大体不变，很多技术就容易迁移，也许对每个用户的微调模型做一点探针微调就行。真正的挑战只有在灾难性遗忘导致内部表征结构彻底改变时才会出现，但我怀疑那意味着模型已经坏了。

我对架构之争没有强观点，但我觉得提示、上下文学习、上下文工程和 harness 里有点「苦涩教训」的味道：对很多事情它们可能就够了。短期最可能的结果不是出现在截然不同的数据上训练的定制模型，而是不同的上下文——不同的 LLM 拥有不同的记忆，而记忆在我看来只是每次生成答案时被上下文学习的一堆 token。你不用使劲榨就能从中得到很多收益，而且这些记忆相当可解释。过去一年智能体的兴起，大量工程就是在把更多东西塞进上下文。

I think it really depends on how continual learning plays out, which I haven’t done research on in depth. If we’re fine-tuning for each person, the key question is how much representation change there is. I have some work, and other folks at Goodfire have some really cool work, on low dimensional manifolds that describe the underlying representation space as you interact with models. A lot of the representations are defined by this low dimensional conceptual structure. As long as a lot of that structure remains the same, a lot of the techniques will be pretty easy to translate, with maybe a bit of fine-tuning of your probes. The only real challenge is if continual learning leads to such catastrophic forgetting that the internal representation structure radically changes — but I kind of doubt that’ll happen, because that probably means the models will break.

I don’t have strong perspectives on the architecture argument, but there’s a bit of a bitter lesson around prompting and in-context learning and context engineering and harnesses — they might be all you need for quite a lot of stuff. The most likely outcome in the short term is that we don’t have custom models trained on very different data; what’s happening is different context. We have different LLMs that have different memories, and the memory, the way I see it, is a bunch of tokens that are in-context learned every time the model generates an answer. It’s incredibly easy to get a lot of juice out of that without squeezing too hard, and you end up with memories that are pretty interpretable. That’s really what we’ve seen in the last year or so with the rise of agents: a lot of that engineering is just going into putting a lot of stuff in context.

Nathan Labenz

我还记得 temperature 曾是 API 层面可以玩的东西。现在温度没了，模型似乎有些过拟合——你那个讲故事的例子就是如此。你怎么讲过去两年的故事：模型怎样变了，作为用户和可解释性研究者该注意什么？

I’m old enough to remember when temperature was a feature that we got to play with at the API level. Now it sure seems like the models are quite overfit in some ways — your story example is a good example of that. How would you tell the story of the last two years, and how have the underlying models changed?

Eric Bigelow

首先我希望知道 Claude API 和最新 GPT 模型的采样参数，把它们拿走很可惜。我猜原因是限制大家逆向工程，因为系统性提示的人会向 API 发出成千上万乃至数十亿 token 的请求，可能在蒸馏模型或提取权重，而不是它们真的换了采样方案。

后训练越多，输出的多样性越低，这并不意外。早期的补全模型甚至不知道自己在对话；先是监督微调和 RLHF 把分布收窄到「像聊天机器人那样回应」，然后 RL 又做了很不同的事。我对 RL 很重的小模型做过实验：让一个 Qwen 3.6 35A3B 模型「给我一个故事」，三分之二都是关于钟表匠的。

我对推理的印象是，它把语言模型的分布推向一个与训练时人类文本非常不同的方向：它们做线性化树搜索，枚举各种可能。「wait」是模型改变主意的关键词，但 DeepSeek R1 的输出里一个推理链能说「wait」50 次以上，这得有多少次顿悟。我认为这是模型当时遇到的回溯问题的解法：它们没法从随机采样的某个词退一步，于是干脆枚举所有能想到的东西，再回头选。所以「推理」几乎是个误称，推理模型更像造出一锅 token 汤，再从里面撇出一点东西。

就原始能力来说，我认为过去几年有一些渐进变化和额外的规模，但很多原始能力大概一直就在这些模型里，之后我们做的是把它们唤出来、让分布更尖锐。比如编码智能体，需要在编码上大量训练才能让模型行为正确，就算它们早已从整个互联网文本里学到了对的东西。

I wish I knew what the sampling parameters were for the Claude API and for the latest GPT models. I think it’s very unfortunate that they’ve taken these away. I expect it’s probably similar things happening under the surface; they want to limit how much people can reverse engineer the models. People who are really systematic hit the Claude API with millions and even billions of token requests, and there’s a risk they’re doing something nefarious, trying to distill models or extract model weights.

There’s been a lot of work showing output diversity disappears to some extent with supervised fine tuning, and the more you post-train, the less diverse the outputs become. The earlier completion models wouldn’t even know they were in a conversation. First we went to supervised fine tuning and RLHF, sharpening the distribution around chat, and then I think RL does something pretty different. I’ve done a bunch of experiments with this Qwen 3.6 35A3B model, which was RL’d pretty heavily, and when you give it “give me a story”, two thirds of the stories are about a clockmaker.

My impression is that reasoning shifts the distribution of what language models are doing in a totally different direction, very different from the human text they were trained on. They do a linearized tree search, enumerating all these different possibilities. If you look at DeepSeek R1 outputs, they’ll say “wait” 50 times or more in a single reasoning chain — that’s a lot of epiphanies. This was a solution to a problem models were hitting at the time, which was how to backtrack: they can randomly sample some word and then can’t step back from that path. So reasoning models get around that by enumerating every possible thing they can think of and then going back and choosing. I think reasoning is almost a misnomer. It’s almost a soup of tokens, and then they just skim something out of that soup.

In terms of raw capabilities, I think there’s maybe been some gradual change and a bit of extra scaling, but a lot of the raw capabilities were probably in these pre-chain models the whole time. A lot of what we’ve done since is finding ways to evoke these and sharpen the distribution of what models are doing. Coding agents, for example: you had to train models a lot doing coding to get them to behave in the right way, even if they had learned the right things from the entire internet of text.

Nathan Labenz

你做研究时怎么选模型？在 Goodfire 你们资源很多，也建了基础设施，可以在 Kimi K3 这种大模型上做可解释性。你觉得在前沿中国模型上做的工作，和在美国闭源模型上做，是否基本可以互相替代？

When you do research, how do you choose what model to work with? At Goodfire you have significant resources and you’ve built some infrastructure to do interpretability on Kimi K3. When you do something on a frontier Chinese model, is that essentially synonymous with doing it on an American proprietary model?

Eric Bigelow

我大体认为它们可以互相替代，当然有一些特质，比如 Qwen 很喜欢用中文思考，对它做 logit lens，在你用英文交流时，中间会出现很多中文字符，然后才开始输出英文 token。选模型时，我会把它当作一种权衡：太小（比如低于 10 亿参数）质性不同；我一直要求自己研究那种「可以零样本研究、不必靠训练模型去模拟」的行为，所以倾向于 7 到 8B 或更大，这个规模上开始能得到有意思的零样本行为。比如我的故事信念更新论文，最初只用了 Llama 8B，它会在读故事时跨很多情绪动态更新信念。

闭源与开源之间最大的护城河，我个人认为是后训练：核心能力都来自同一份互联网数据，差别在于后训练那份「特别的酱」。Kimi K3 是个大的阶跃，在编码上很强，Goodfire 同事做过关于奖励黑客的研究：这个模型表现出在小规模上看不到的疯狂奖励黑客行为，这对研究我们认为在闭源前沿模型里发生的现象给出了信号。

Nathan Labenz 问：这是否意味着，闭源公司之外的可解释性研究正在依赖中国模型？Eric 的回答是：目前确实如此，最大模型上，Kimi K3 与其他模型在编程能力上有很大差距，不过如果美国公司更多投入开源模型，这个差距并非无法弥合。他说，如果出现不分来源、一刀切限制开源模型的做法，会是更大的问题，因为一些随规模涌现的风险行为（比如奖励黑客）的研究会被拖慢。

I think of them in large part as standing in for each other. There are idiosyncrasies — Qwen really likes to think in Chinese, and if you do the logit lens, there’ll be a lot of Chinese characters in the middle before it starts outputting English tokens if you’re talking to it in English. A standard I’ve always set for myself is to study something like the big models where you see these elaborate human-like behaviors, zero shot, without training the model to do that or using a facsimile. I tend toward the scale of 7 to 8 billion or more being a sweet spot where you can start to do good interpretability research. In my story belief updating paper we initially just used a Llama 8 billion, and it does update beliefs very dynamically as it reads stories across a number of different emotions.

In terms of the moat between closed source and open source, I think the post-training is the biggest one. They’re all really trained on the same data, which is the entire internet, and the main difference is the post-training and that extra special sauce. Kimi K3 is a big step function in having a model that’s really good at coding that we can study. I’ve seen research other people at Goodfire have done with reward hacking, and the model exhibits this crazy reward hacking behavior that you just don’t see at smaller scales, and it gives us some signal into studying phenomena that we think are happening inside proprietary frontier models.

Nathan Labenz

我最近最大的惊讶之一，是拟人化模型竟然如此有成效。你个人对「怎样拟人化」的哲学是什么？

One of the biggest surprises for me in recent times has been how productive it has been to anthropomorphize models. What is your personal philosophy of how to anthropomorphize?

Eric Bigelow

我读博时 LLM 还不存在。我的导师在答辩上说，如果你当初告诉我论文会是这个题目，我会问：什么？大语言模型是什么？起初我确实遭到身边一些认知科学家的反对，他们习惯了深度学习里的各种「捷径」，觉得模型做不出某些行为。不少心理学和语言学的人觉得 AI 研究在做「糟糕的心理学」，滥用了这些在心理学里有深厚历史的词。但随着时间推移，这种反对软化了，一部分是因为这些模型实在太成功，你没法一直逆着一条汹涌的河推。

我对拟人化有两种看法。一种是个人层面：它帮你理解，用得随意也没关系，就像车发出咔哒声，你说「它想告诉我什么」，认知科学里叫「直觉理论」。另一种是更严肃的：LLM 做的事不只是发出咔哒声，它们说语言、做数学推理、做社交推理、帮我们回答科学问题。所以对那些批评「词被滥用」的领域，最健康的做法是认真对待这些问题：什么是信念？什么是决策？我还发现心理学里有大量关于「事后合理化」的实验：人们会为自己其实没做出的选择编造理由——所以 forking paths 的现象，其实和人类也没有想象中那么不同。

When I started my PhD, LLMs didn’t exist. My advisor said at my defense: if you had told me when I was starting out that this is what my thesis was going to be titled, I’d say, what now? What are large language models? I did get a lot of pushback from some of the cognitive scientists around me, who were used to deep learning having lots of shortcut-type things where a model can’t really do the behavior you’re talking about. People in psychology and related fields look upon research on AI critically, because it feels like a lot of work around AI is doing bad psychology, using words that have a lot of history. Over the years, some of this pushback has softened, in part just how damn successful these models have been. At some point you can’t keep pushing back on a river that’s pushing so hard.

There are two perspectives I have on anthropomorphizing AI. One is you can anthropomorphize in a way where you don’t have to be too careful, it’s for you to understand it. If my car starts making a clunky sound, I might say it’s trying to tell me something — cognitive science calls these intuitive theories. The other perspective is that LLMs aren’t just making clunking sounds: they’re speaking language, doing mathematical reasoning, doing social reasoning, helping us answer scientific questions. What’s really healthy for people from fields who have a critical eye on words being misused is to take those questions seriously. What are beliefs? What are decisions? There’s also a bunch of work on rationalization of decisions, where people come up with explanations for why they chose something they didn’t even choose — so forking paths may be less different from people than I initially thought.

Nathan Labenz

你做了采样项目来弄清模型对一个提示可能生成的全部可能性，还有低采样技术，以及从内部表征预测结果。这一切告诉你「信念」是什么？它能被还原成最终给出某个答案的概率吗？这似乎没有抓住我所说的「我拥有信念」的完整含义。

What does all that tell you about what is a belief? Is it simply reduced to the probability that it’s ultimately going to give this answer? That doesn’t feel like it captures the fullness of what I think when I talk about myself having beliefs.

Eric Bigelow

一部分上，它确实与模型生成各 token 的概率这类简单东西有关，但还有更多细节：它在不同输入下如何变化，或者在读一段文字、生成一段推理时信念如何变。我的心智框架来自认知科学里的贝叶斯建模：行为按潜在假设或概念来生成，哪些概念被唤起很大程度由所给数据决定；人有先验，上下文数据会通过后验推断来重新加权这些潜在概念。我认为这也适用于 LLM。

贝叶斯认知科学常被批评对「大脑里到底是什么算法」保持不可知，因为研究大脑太难。但对 LLM 我们没有这个借口，我们很容易「窥探大脑」并做干预。这给了我们机会弄清：模型在推断时，是在抽样、还是在估计后验？我们用来建模人的模型，到底是描述了算法，还是只描述了聚合行为？

At some level it does relate to simple things like the probabilities of tokens the model will generate, except there’s more nuance: how does this change across different inputs, how do beliefs change as it’s reading a passage or generating a chain of reasoning? My mental framework is around assigning probabilities according to latent hypotheses or concepts the model has, with behavior generated according to latent concepts, and which concepts are evoked determined in large part by the data it’s given. This is the Bayesian modeling of cognition, where we have prior probabilities and context leads us to reweight beliefs through posterior inference. I think this also applies to LLMs.

It’s easy to be agnostic when it’s really hard to study brains. But we don’t have that excuse with LLMs. It’s very easy to peek inside the brains and to do interventions, which is very complicated for both methodological and ethical reasons with people. It gives us a chance to break new ground around how beliefs are represented, and when models are doing inference, are they drawing samples? Are they estimating posteriors in the ways we’ve used to model people? Do these models really describe the algorithm, or just the behavior?

Eric Bigelow

关于「不确定性」要说清楚：在推理链结束时模型几乎 100% 确定答案，因为答案已由先前的推理决定。我们说的不确定性，是指在这一点上重采样后续推理链得到 50% 答案 A 和 50% 答案 B；这并不意味着模型内部表示了这些完整的 rollout。我在做的新工作发现，从隐藏层直接解码这些结果分布其实相当难；一种解释是数据不够（这些方法推理成本极高），另一种是模型根本没有表示完整推理链，只表示了前面几步。我也做了一个项目，表明在一定程度上能解码模型接下来会做什么，并据此引导采样。

就像数学家开始证明时，可能还没推完整条推导，却已经能看见证明的「形状」，我也想知道 LLM 是否也有这样一种对高层路径的不确定性表示。

By the end of a reasoning chain, the model is almost 100% sure what the answer is, because it’s determined by the previous reasoning chain. What we describe as uncertainty in reasoning is: at this point I resample long reasoning chains and I get 50% answer A and 50% answer B. That doesn’t mean it represents these full rollouts. It’s actually pretty hard to decode these rollout distributions from the hidden states. A disclaimer is this could be a data issue, since these methods are so expensive in terms of inference. But another explanation is that models, when they’re uncertain over reasoning, aren’t really representing the full reasoning chains, maybe just a couple steps ahead. I’m working on a project showing that you can actually decode to some extent what models are going to do next, and guide sampling based on this.

Mathematicians, when starting to solve a proof, sometimes have some high-level scaffolding view in mind — they can see the shape of the proof before they complete it — and I wonder if that might be true for LLMs too.

Nathan Labenz

看 Forking Fast 等几篇论文里的图，这种突然的相变非常醒目。你怎么理解？是不是该希望能「平缓着陆」，比如在 Astra 这类循环（looped）模型里？

Looking at some of the graphs from a couple of papers, including the most recent one, Forking Fast, the sudden phase change is an extremely striking result. Do you share the intuition that something is weird or wrong with these cliffs, and would you also like to see a gradual landing of the plane?

Eric Bigelow

我觉得，如果你想要输出多样性，又要求它们自洽、不是胡话，那么分叉可能就是这两个要素的必然属性。我真正想要的是，用户界面能把这一点呈现给我们。早期有显示 token 对数概率的界面；也有一种叫 Loom 的界面。我希望在语义层面也能看到：哪些点可能分出完全不同的路径，被高亮给我。无论是急转弯还是缓转弯，只要用户能知道，就不再是「你没发现的急左转」。

我另外观察到，相变在 rollout 的开头、中间、结尾都会发生，每个输入、每条推理链都有自己的故事。我们或许可以希望训练掉一些「在意外 token 上的急剧分叉」，比如那个左括号，它反映了模型在人类文本里的奇怪偏差；如果分叉发生在人类可解释的位置，会好得多。另一种常见模式是：不确定性稳定维持，直到模型说出「答案是……」才突然塌缩。早期有个「取四个词的末尾字母并拼接」的任务，模型有时甚至会写 Python 代码，最后在说出答案那一刻其实是在现编。

我觉得随机鹦鹉这个比喻应该退休了。它是个相当浅的论证，像中文房间那类论证：一本记住了所有例子的巨大的书。可那本书里到底发生了什么，才让你能进行对话？而且有大量工作表明模型有一致的世界模型（比如 Othello GPT）；到某个程度，学一个世界模型比学一张巨大的查找表更高效。但随机性和采样仍是等式里不可或缺的部分，因为我们想要输出多样性、想要模型表达不确定性，又不想它每次不完全确定就停下推理。随机性并没有死。

关于潜在思维链：循环 transformer 让推理发生在确定性的前向传播里，这是我现在花很多时间琢磨的。我对开源循环 transformer 做了 forking 分析，发现在做完潜在推理后，生成推理链 token 时仍有不确定性，仍会在特定 token 处改变。我不想活在对同一输入永远得到同一输出的世界里，那将是最极端的模式崩塌，没有观点的多样性；而潜在思维链最终仍会进入采样 token 的阶段。

If you want some diversity in what your model will do, and you want these to be self-consistent, you don’t want reasoning chains that are just nonsense, then forking might just be an inevitable property of having these two key ingredients. What I’d really like is for our user interfaces to surface this to us. There were earlier interfaces showing token log probabilities, and another kind of interface called a Loom. I’d like to see something like that be part of the standard interfaces, at the level of semantics: the points where things could branch off into a very different path highlighted to me. Whether it’s a sharp left turn or a gradual left turn, I’d like it surfaced to the user.

Phase changes happen sometimes at the beginning of the rollout, sometimes at the end, sometimes in the middle; every input can tell its own story. Some of this sharp forking at unexpected tokens we would hope to train away a bit — that open parenthesis speaks to weird biases of LLMs in text. If forking happened at points that were interpretable to people, that would be better. You also see a really common pattern where there’s stable uncertainty right up until some point when it suddenly collapses, when it finally says “the answer is blank”. There was a task called last letter: take four words and concatenate the last letter of each. Some reasoning chains are funny — it’ll even write Python code — and then at the final answer moment it’s just making it up on the fly.

I think the stochastic parrot metaphor should probably be put to rest. It’s a fairly shallow argument, sort of like a Chinese room argument with a giant book of memorized examples — but what the heck is going on in that book that lets you respond with conversation? There’s so much work showing consistent world models. At some point it’s more efficient to learn a world model than a giant lookup table. But stochasticity and sampling is still an integral part of the equation if we want output diversity, if we want models to have uncertainty but not halt their reasoning every time there isn’t 100% certainty. Stochasticity is definitely not dead.

With loop transformers and latent chain of thought, reasoning can happen during this deterministic forward pass. I’ve been doing forking paths analysis on open-source loop transformers, and after the latent reasoning, the model still is uncertain between answers and still changes at particular tokens. I don’t want to live in a world where you always get exactly the same output from a model given the same input — that’s the most extreme version of mode collapse, and there’s no diversity of perspective. I don’t think latent chain of thought will completely change that, because you still eventually go into this sampling-tokens regime.

Nathan Labenz

你对思维链为什么变得古怪有直觉吗？比如它里面那些奇怪词汇，还有 Astra 给子智能体发消息的方式。我原以为是长度惩罚，但数 token 的人说其实直说更省 token。

Do you have intuitions for why the chain of thought is getting wonky? We also see this now with Astra and the way that it sends messages to its sub-agents, which I had naively thought has to be some sort of length penalty, but people are counting tokens and saying you could just say the thing in fewer tokens.

Eric Bigelow

推理带来的是分布偏移的挑战：当你做 RL，不论是单模型还是子智能体，如果只做结果层面的验证而非过程层面的验证，目标就是以任何手段得到更好的最终答案，没有激励让中间内容保持人类可读。这像语言演化：时间尺度足够长，语言本身就会变，没有力量强迫今天的英语和 300 年前一样。再加上古德哈特定律：如果你为「思维链可监控」而优化，思维链就可能不那么忠实，模型学到的是「看起来合理」的中间步骤。

我对这件事的想法在过去一年变化很大。我曾经很看好思维链监控，认为即便不完全忠实，它也在很大程度上决定了模型能做什么。但后来看到「stolen chain of thought」那篇论文里某些 GPT 模型的思维链，在谈论音乐剧时有一种几乎像间谍暗语的味道，所以这种情况会出现，回头看是显而易见的，因为 RL 让模型的分布偏离训练数据如此剧烈。我仍然喜欢思维链还存在、东西被迫落在 token 流里，哪怕更难读；我想弄懂 Astra 怎么与子智能体说话，这在科学上也很有意思。但未来思维链和规划者对子智能体的沟通也可能变成潜在的，没有天然理由必须是 token。

Reasoning presents an interesting challenge of distribution shift. When you’re doing RL, whether with a single model or with sub-agents, there isn’t necessarily an incentive to keep things human readable. If you’re doing outcome-level verification instead of process-level verification, then your objective is to have a better final answer by whatever means possible. It reminds me of language evolution with people: on a long enough time scale human languages change, and there isn’t any driving force that forces English to stay as it was 300 years ago. There’s also a Goodhart’s law thing: if you optimize for chain of thought being monitorable, sometimes the chains of thought are less faithful, and you learn intermediate steps that look sensible, whether or not they are sensible.

My thinking on this has shifted pretty dramatically in the last year or so. I used to be very bullish on chain of thought monitoring being promising, where even if it’s not perfectly faithful, it is strongly determining what the model can do. But then there is the stolen chain of thought paper showing some of the chain of thought of GPT models, and they’re talking about musicals in a way that feels almost like spy language. It’s obvious in retrospect why this would happen, because if you’re doing RL, you’re letting models shift the distribution in such a radical way from what they’re trained on. I still like that there is chain of thought and that things get forced into a token stream, even if it’s harder to read, and I really want to understand how Astra talks to its sub-agents. But we might move into a world where chain of thought is latent and how planners talk to sub-agents is also latent — there’s no inherent reason that has to be tokens.

Nathan Labenz

我们正急于把各种重要系统的近端控制权交给 AI，并让它们用语言彼此连接；但当走到那些关键 token 时，我们对「被采样的那个分布是怎么确定的」似乎仍没有好的解释。接下来怎么办？

We’re in a rush to hand off a lot of proximal control over all sorts of important systems to AIs, and we’re going to connect them in language. But when we get to these critical tokens, we still don’t have a great account of how the distribution from which we then sample gets determined. How are we going to get it?

Eric Bigelow

我们需要更多可解释性研究。我们在竞相想出跟得上模型的理论框架和经验工具，但用于让模型变强的投入与用于理解它们的投入相比，根本没法比：有万亿美元公司在造模型，而我所在的这家，也许是唯一一家估值超过 10 亿美元的可解释性公司，一家公司十亿美元已经很多，但整体仍然远远不够，学术界也一直严重缺乏资助。

We just need more interpretability research. We’re racing to come up with theoretical frameworks and empirical tools that can keep up with the models, but for the amount of investment going into making models better versus understanding them, there’s no comparison. There are trillion dollar companies producing these models, and I’m in the largest interpretability company, maybe the only substantial one with a value of over a billion dollars. A billion dollars is a lot for a company, but there just is not enough investment — not just in industry but also in academia, which has always been criminally underfunded.

Eric Bigelow

这对其他智能体也适用，不只是 Silico。很容易想把整个项目丢给智能体，让它自己去跑一堆实验、出一堆图和描述，但智能体容易「迷失在酱汁里」，构造出不成形的理论。我认为最好的用法是带一点手把手，随时跟踪。上下文就是一切：每次我说「这看起来不对，因为……」或「我下一步想做……」，我都会说出理由、说明更大的目标。就像一个在图书馆里醒来的智能体，不知道这个世界怎么运作，需要上下文。

我的 Forking Fast 论文，全部研究都由 Silico 完成，一周内搞定，是很大的加速。但在那之前有一堆支线任务没有结果。当我做了高采样、看到曲线平滑，突然想到也许可以用统计模型来提高效率。同事有句话：在智能体的世界里，科研品味价值连城。用 Silico 就像和一位热情、勤奋但不总是知道大局的一二年级博士生合作。我最大的建议是：握紧你的科学品味，并亲自参与研究。

These thoughts apply to other agents too, not just Silico. It can be really tempting to hand over an entire project to agents, but it’s really easy for agents to get a bit lost in the sauce and create theories that aren’t fully formed. The best way to use agents is to give a little bit of hand-holding and keep track of things as you’re going. Context is everything. Every time I’m saying “that doesn’t seem right” or “here’s what we should do”, I’m describing my logic: here’s what doesn’t look right and why, here’s what I’m going to do next, here’s my bigger-picture goal. If you’re an agent waking up in a fresh context, you don’t know how the world works. You need context.

When I did this Forking Fast paper, all the research was done fully by Silico, and it all happened within a week — a huge accelerant. Before it there were a bunch of side quests that didn’t pan out. Once I did really high sampling and saw that curves were smooth, it clicked: maybe I could use a statistical model for this to be more efficient. A colleague put it that research taste and scientific taste is worth its weight in gold. Using Silico is like working with a very eager first or second year PhD student that has a lot of skills and is hardworking, but doesn’t always know what it’s doing in the bigger picture. My biggest tip is to hold on to your scientific taste and try to take part in the research.

Nathan Labenz

除了「LLM 如何做决定」，现在 Goodfire 午餐桌上的热门话题是什么？

Aside from how do LLMs make decisions, what are the hot topics over lunch at Goodfire these days?

Eric Bigelow

暑假以来有两个大的主题。第一是让智能体做可解释性，也就是 Silico；如果想让可解释性跟上这个领域的飞速发展，也许只有把一部分自动化这一条路，并把可解释性变成某种「常规科学」。第二是公司最近转向对齐，这原本就是很多人感兴趣的方向，做可解释性的人和关心安全的人重叠度很高。我认为 Hugging Face 事件是个分水岭：人们真的在乎这件事，这不只是我们自己觉得重要。

也有很多对话围绕奖励黑客：一个行为看似错了、但符合你所写的规定，遵循了字面而非精神，这算奖励黑客吗？模型有「意图」吗？这些理论与经验问题——什么是目标和意图，怎样测量真正的坏行为而不是简单的错误泛化——都很迷人。

Two big ones since the summer. There is the idea of agents doing interpretability, Silico — if we want interpretability to keep up with the breakneck speed of the field, it might be the only way to have parts of this automated, standardizing interpretability as a sort of normal science. Another big internal push, very recently, has been around alignment and our company’s shift towards this. People interested in interpretability have a very high overlap with people interested in alignment and safety. The Hugging Face incident was a watershed moment: people care about this, it isn’t just something that we care about.

There are fundamental questions around reward hacking: if it’s doing this particular behavior that seems wrong but consistent with what you specified, following the letter but not the spirit of the law, was it reward hacking? What are goals and intentions? How do you measure real bad behavior and not just some simple misgeneralization?

Nathan Labenz

你的清单很满，感谢你慷慨地给了这么多时间。我就不耽误你回去工作了，用我爸引用 Leslie Nielsen 的话：祝你好运，我们都指望着你。

You’ve got a very full plate. I should let you get back to work, to quote my dad quoting Leslie Nielsen: good luck, we’re all counting on you.