Noam Brown：多智能体蜂群、对齐与递归自我改进

Dwarkesh Patel

今天请来 Noam Brown，OpenAI 研究员。他是后来成为 o1 以及推理模型的基础贡献者之一，现在在做多智能体系统。说到这个，你们上周宣布：用一套约 1 万个不同 AI 智能体的系统，花费 1300 亿 token、历时 88 小时，解决了千禧年大奖问题之一。我想跟你聊的一个原因是：大概两三年前，你们大概是最先在想——推理模型让我们能看见未来，因为把推理计算往上加，就能看到模型几年后的基础能力。我感觉你现在处在类似位置：智能体规模可以扩到非常大，能帮我们理解未来能力会长什么样。

Today, I’m chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became o1 and the reasoning models, and now he’s working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the Millennium Prize problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours. One of the reasons I’m interested in talking to you is I think you were the first people maybe two or three years ago who were thinking about how the reasoning models would allow us to see into the future, because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future. And I feel like you’re in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.

Noam Brown

我是这么看的：把推理模型的表现画出来，横轴是测试时计算，纵轴是几乎任何推理基准上的表现，你会看到非常清楚的规律——这些模型想答案想得越久，就做得越好。这很自然，人也一样。你考 SAT，全程只有五分钟，不会考好；有五个小时，大概会好很多。

So the way I think about it, when you plot the performance of these reasoning models with test-time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is a very natural thing. It’s the same thing with people. If you’re taking the SATs, you have five minutes to go through the entire exam, you’re not going to do very well. If you have five hours, you’re probably going to do a lot better.

Noam Brown

AI 模型差不多，它们会把那段时间用来对自己独白：想清楚、走不同情形、排除可能性、叠加上一轮发现。问题是你一直往前推，会撞上延迟瓶颈。你不会想干等三年才拿到一个回复。所以可以做的，就是很多人会做的：并行。要创办公司，你会拉一队人，好走得更快。AI 模型也一样——多个智能体一起干活能更快。多智能体是把测试时计算按并行而不是纯串行来缩放。效率更低，因为不是单个智能体独占全部上下文，但如果做好了，这是缩放测试时计算非常有效的办法。

The AI models are pretty similar, and they’ll spend that time doing this monologue to themselves, figuring out, going through different cases, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don’t want to sit around for three years waiting for a response. And so what you can do is what a lot of people do: they parallelize. They just get a team of people. If you’re going to found a company, you want to get a group of people together so you can go faster. It’s the same thing with these AI models: it helps to just have multiple agents working on something because they can just go faster. And so multi-agent is a way of scaling test-time compute in parallel instead of purely serial. And it is less efficient because it’s not like a single agent has all the context to itself, but it is a very effective way of scaling test-time compute if it’s done well.

Dwarkesh Patel

我要问一堆天真的问题，因为这些系统——这是未发布的模型——我们还没公开看见它们怎么工作，我对这类系统的定性性质有一堆困惑。我震惊的是，能在这么短时间里集中这么大规模的认知努力。1300 亿 token，如果是一个人全职思考、首尾相接，大概相当于一个人想 4000 年——按一天八小时、正常工作周。从古代苏美尔一直到今天，一个串行人类想那么久，被压进 88 小时。定性上这非常重要。我也惊讶并行化惩罚看起来没有更大：你就能让 1 万个智能体协作。也许智能体比人更会协作、跑得更快，所以这么大的规模还能有成效；也可能其实有很大的并行化惩罚。

Okay, I’m going to ask a bunch of naive questions because these systems—this is an unreleased model, so we haven’t publicly seen how these systems work. And so I just have a bunch of ways in which I’m confused about what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for like 4,000 years, eight hours a day or something, working a normal work week. So starting from ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours. Qualitatively, that is a super important consideration. And I’m surprised that there isn’t a bigger parallelization penalty, that you can just have 10,000 agents collaborate. And because maybe the agents are better at collaborating than humans might be, they’re going much faster, that they can actually productively collaborate at such a big scale. Or maybe there is a big parallelization penalty.

Noam Brown

我们先谈并行化惩罚，再谈定性的东西。实话是：把多智能体缩放到这种规模，我们并没有很好的科学。

Yeah, let’s talk about the parallelization penalty, and then we can talk about the qualitative stuff. Because the truth is that we don’t have very good science on multi-agent scaling up to this kind of scale.

Noam Brown

我们发布 5.6 的时候，我觉得那是第一次在我们的模型里有正式的多智能体系统。博客里我们其实画了多智能体系统的缩放表现，因为这是一个选项：ultra 模式。默认是 4 个智能体，但可以调高。图上我们展示了一些基准上：1 个智能体、4 个一起、16 个一起。取决于基准，但有些基准上，基本上 4 个智能体做这道题，完成时间大约是一半。所以四个智能体干一半那么久，你大概多付 2 倍，换答案快一倍。到 16 个，类似规律，效率再低一点，但表现还在继续。

When we released 5.6, I think that was the first time that we had a proper multi-agent system in our models. And we actually did in the blog post show some plots of the scaling performance of multi-agent systems because we have it as an option. It’s ultra mode. And the default is four agents, but you can set that to higher. And in the plot, we show, okay, here’s what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together. And what you see, and it depends on the benchmark, but for some of the benchmarks, basically if you have four agents working on the problem, it is done twice as fast. So you’re basically paying, because there’s four agents working for half as long, you’re paying 2× more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern, it’s a little less efficient, but you continue to see that performance.

Dwarkesh Patel

增加并行智能体数量时，串行时间上是线性加速，还是次线性？

Is it a linear serial-time speedup or a sublinear speedup as you increase the number of parallel agents?

Noam Brown

我会说略微次线性，不过很取决于问题。比如数学相当可并行——不是最可并行的，但很可并行。网页搜索、做深度研究报告要翻一堆来源，极端可并行。我怀疑写小说会非常不可并行。你大概不会看到 1 万个智能体一起写一部小说有很大好处，就像 1 万人一起写一部小说大概也没多大好处。表现取决于领域。我们在已发表的博客里量到大约 16 个智能体。问题是很难把这套科学推到 1 万个，因为太贵了。

I would say it’s slightly sublinear, though it does depend a lot on the problem. So math, for example, is quite parallelizable. It’s not the most parallelizable thing, but it is very parallelizable. I think web search, things like doing a deep research report where you have to look through a bunch of sources, that’s extremely parallelizable. I suspect that something like writing a novel would be very unparallelizable. So you would probably not see a big benefit from having 10,000 agents working on a novel together. In the same way that you’d probably not have a big benefit from having 10,000 people working on a novel together. So the performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is it’s very hard to push that science to like 10,000 agents because it’s just so expensive.

Dwarkesh Patel

你们刚用一个周末就做了。

You guys just did it over like a weekend.

Noam Brown

但那只是一个数据点。我们不知道单个智能体解 Navier–Stokes 要多久，因为还没做那个实验。也许会做，但那也只是一个数据点。若要做彻底的消融，实验贵到推不到那个规模。所以得做某种有方法的科学：到 64、128、256 左右时发生什么，对行为有感觉。但很难一路推到 1 万，并确定用 1 万个相对 1000 个到底拿了多少好处。

But that’s one data point. We don’t know how long it would take a single agent to solve Navier-Stokes because we haven’t done that experiment yet. And maybe we will, but I mean, that’s also only one data point, right? And if we want to do a thorough ablation, the experiments are just too expensive to go to that scale. So we have to do some kind of methodical science about what happens when you go to like 64, 128, 256 or something and get a sense of the behavior. But it’s going to be very hard to push that all the way to like 10,000 and know for sure what was the benefit that we actually got for using 10,000 agents versus 1,000.

Noam Brown

有一点我想说清楚：拿下千禧年大奖问题，这不是因为多智能体。我甚至不愿把 10% 的功劳记在多智能体头上。现实是 OpenAI 训了一个非常强的模型。我们能让它在很长的视野上运转，能让它并行思考。但核心原因是：我们就是有一个通用、非常强的模型。多智能体这类东西很炫、很新，大概因此拿了不成比例的功劳。但核心原因是：这就是一个非常强的模型。

One thing I want to make clear is that the effort to get a Millennium Prize problem—this was not due to multi-agent. I wouldn’t even attribute 10% of the credit to multi-agent. The reality is OpenAI has trained a very powerful model. And we can get that model to operate over very long horizons. We can get it to think in parallel. But at its core, the reason why we’re able to do this is because we just have a general-purpose, very strong model. I think things like multi-agents are flashy and are new and that probably gets disproportionate credit for that reason. But the core reason is this is just a very powerful model.

Dwarkesh Patel

泛化其实让我很震惊。这些系统——我不知道怎么训的，但大概是 RL 那种：一堆可检查的合成题，对着做一堆 RL。训练过程里，我猜模型从没解过千禧年大奖那么野心勃勃的东西。但泛化强到：那些容易得多的可验证问题，能泛化到在这么难的问题上投入这么大的并行努力。

So the generalization is actually quite shocking to me. These systems—I don’t know how these systems were trained, but presumably they were trained how RL training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process, I’m guessing, was the model solving anything as ambitious as a Millennium Prize problem. But the generalization was strong enough that you could have this much easier verifiable problems generalize to this much parallel effort on such a hard problem.

Noam Brown

我觉得这是对的。首先，我们确实在非常难的问题上训模型。所以肯定有差距。我们看到：在某些任务上训练，它能做比那更野心勃勃的任务。有个有意思的挑战是：模型越来越聪明，我们能问它们的很多问题太容易了，很难挑战模型。我如果要说的话，这是一个论据：为什么你可能看不到 LLM 走 AlphaGo、AlphaZero 那些博弈 AI 的同一条路。AlphaZero 那种自对弈有无限课程：你永远在跟同等强的 AI 打。而对 LLM 做强化学习，至少现在外面那些办法，是给模型一道题让它解。如果题简单到我一秒钟就能解，它其实没学到什么。所以如果我们用尽了能挑战它的题，这是一个说得通的情景：进展会难得多。我确实觉得有绕法，我们还没真正撞上这堵墙；如果真成了严重问题，会有办法。但这是一个说得通的情景。

I think that it’s true. First of all, we do train the model on very hard problems. So there is definitely a gap. We see if we train on some kinds of tasks, it’s able to do tasks that are more ambitious than that. There is an interesting challenge that as the models become smarter and smarter, the kinds of questions we can ask them, a lot of them are too easy. And it’s hard to challenge the model. I think that’s an argument for why you might not see AIs, like LLMs, go the same path as AlphaGo and AlphaZero and all these kinds of game-playing AIs. It might be this kind of problem that in things like AlphaZero, where you have self-play, you have an infinite curriculum. You’re always playing against an AI that’s equally strong. Whereas for things like training an LLM with reinforcement learning, at least the ways that are out there right now, you give the model a problem and you ask it to solve it. And if the problem is so easy that I could just solve it in a second, it’s not really learning anything. So if we run out of problems to ask it that challenge it, then that is a plausible scenario where actually it becomes much harder to make progress. Now, I do think there are ways around that. And so we haven’t really hit that as a wall yet. And I think that if it ever became a serious problem, there would be ways around it. But it is a plausible scenario.

Dwarkesh Patel

跟听众说一下：你提到 AlphaGo 或 AlphaZero，是说达到人类水平之后，相对很快就到超人类。

And just for the audience, when you’re referring to AlphaGo or AlphaZero, you’re saying getting superhuman relatively fast after achieving human-level performance.

Noam Brown

对。看围棋这类博弈轨迹：大概一年之内，从赢欧洲冠军——我不知道，大概世界第 50——到赢世界冠军，再到比任何在世人类强到无法想象的数量级。

Yeah. I mean, if you look at the trajectory of game playing as like Go, they, within a span of like a year, went from beating a European champion, being like, I don’t know, like number 50 in the world, to beating the world champion, to being unimaginably orders of magnitude stronger than any human alive.

Noam Brown

有可能数学这类领域我们看到类似轨迹，但我觉得有一个非常说得通的情景是：其实不会发生。

It’s possible that domains like math we see a similar trajectory, but I think there is a very plausible scenario where actually that doesn’t happen.

Dwarkesh Patel

所以我想弄清：如果六个月后人们能用上多智能体系统，该怎么建模「跟一套多智能体系统协作或雇佣它」是什么样？

So I want to understand if in six months people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system?

Noam Brown

我应该先讲这些多智能体系统实际怎么工作，跟很多其他 AI 里的多智能体很不一样。很多人做 LLM 多智能体，会走非常脚手架化的路：比如有一个协调者智能体，把活派给一堆子智能体，给它们任务，子智能体做完交回答案。这看起来很合理，脚手架也确实有帮助。但这类设置有一堆限制。比如协调者把任务发给子智能体，子智能体做完交回——如果两个子智能体拿到相似任务，它们能互相说话吗？通常不能。那很没效率。你拿到一个任务，其实很有帮助去找可能知道答案、或在做你正在做的一部分的人，直接 ping 一句：嘿，这事你能帮我吗？很多系统没有这套，加上去会显著增加脚手架复杂度。还有：如果子智能体没真正理解、有澄清问题？它就得选：是交回去问，而不是解题？还是解题，对父智能体想让我做什么做个假设，按那个解？

Yeah, I should start by talking about how these multi-agent systems actually work, which is, I think, a very different way than a lot of multi-agent systems in other AIs. A lot of people that have approached multi-agents for things like LLMs tend to take this very scaffolded approach where, for example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task and the children work on it and then return their answer. This seems like a very sensible setup, very sensible scaffold. It definitely helps. But there are a bunch of limitations with these kinds of setups. So for example, if in this setup you have a coordinator that’s sending tasks to children, the children work on it and then return their answers. Well, what happens if two children are given similar tasks? Can they talk to each other? And usually the answer is no. And that’s very inefficient, right? If you’re given a task and it’s actually really helpful to talk to somebody that might know an answer to a question that you’re working on or a part of something that you’re working on, it’d be really helpful for you to just be able to ping them and say, hey, can you help me out with this thing? But a lot of systems don’t have that set up and adding it just increases the complexity significantly to the scaffold that you have. Another thing is, what if the child doesn’t really understand or has a clarification question? So it has to choose then between, okay, do I just return and ask the question instead of solving the problem? Or do I solve the problem, just make an assumption about what the parent wanted me to do and just solve it that way?

Noam Brown

所以人们想出的任何脚手架，总会有限制。我们想走的路，是尽量走到极端：少塞结构，给智能体非常原始的工具，让它们自己摸索怎么有效用。我们给智能体给另一个智能体发消息的能力。消息会插进上下文。还有几件类似的事，但核心就是：它随时可以发消息，就是一次工具调用，发给其他智能体。它们自己找出围绕这个协调的最好办法。如果这事做好了，会得到非常精巧的行为。对我来说，很像人类协作者在 Slack 上干活。做这个项目时，终于跑通、看见这些智能体一起解题，真的很兴奋。我记得一个例子：我们给智能体一道题，一个说我觉得我有答案，另一个说其实我得到了不同答案。然后它们整段讨论：你怎么得到那个答案的？能解释给我听吗？来回澄清彼此推理里可能错在哪。最后收敛：哦对，那个好像对。然后广播给其他智能体：其实我改答案了，我觉得他对。感觉就是一场很自然的对话。

And so in any scaffold that people come up with, there’s always limitations involved. And the approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use and figure out for themselves how to use it effectively. So we give the agents the ability to message another agent. And when it messages another agent, it is inserted into the context. And then it can do a few other similar things, but that’s basically the core of it, that it can just send a message whenever it wants, just a tool call, and it can send that to other agents. And they figure out for themselves the best way to coordinate around that. And it turns out that if this is done well, you get very sophisticated behavior. And to me, it looks a lot like how human collaborators work over something like Slack, for example. When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together. I remember one example. So we give the agents a problem. And then one agent says, I think I’ve got the answer. And then another agent says, actually, I got a different answer. And then they have this whole discussion about, well, how did you arrive at that answer? Can you explain it to me? And going back and forth and trying to clarify what could have been wrong in each other’s reasoning. And then they finally converge and like, oh, yeah, OK, that seems right. And then it just broadcasts to the other agents, like, actually, I’ve changed my answer. I think he’s right. And it just felt like a very natural conversation.

Noam Brown

有点像你第一次看见经强化学习训出来的思维链：哦，这就是一个人边想边把想法写下来会有的东西。感觉差不多。看见这种行为真的很酷。所以我觉得跟这些东西协作，老实说，很像跟人协作。就是一种很自然的流动。

It kind of felt like when you see chain of thought for the first time that’s trained through reinforcement learning and you’re like, oh, this is just kind of like what a person would think if they were writing down their thoughts as they’re thinking them. It kind of felt like that. So it is really cool to see this kind of behavior. And so I think collaborating with these things, honestly, it feels a lot like collaborating with a person. It’s just a very natural flow.

Dwarkesh Patel

不过将来可能变得显著的一个定性差别是：这些系统会想得——我不知道——快十倍以上，如果你只看每秒输出多少 token 对比人说话有多快。而且它们一直在干活，不睡觉。它们彼此协作的节奏，也比人跟人协作的容量激烈得多。我在想一年后定性上该预期什么。是不是某种影子组织，在我的公司里比人类层面快 100 倍。迭代周期快得多。人类组织要一年的事，影子组织一周就发生。

Except one qualitative difference that might become salient in the future is that these systems will be thinking, I don’t know, more than 10× as fast, right? If you just look at how many tokens per second they output versus how fast a human talks. And they’re working all the time. They’re not sleeping. And they’re collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans. So I’m trying to think of what to qualitatively expect in a year. And is it like a sort of shadow organization that is moving 100× faster in my company than the human level is. The iteration cycle is much faster. What would take a human organization a year to do is happening within a week within the shadow organization.

Noam Brown

所以会觉得陌生吗？我不知道。我其实发现现在跟这些东西共事相当出奇地自然。我觉得那可能变。比如我们有超快模式，采样能快 10 或 15 倍之类。然后，跟上这些东西会相当难。智能体彼此通信时，对，它们可以超快。但它们也理解什么时候在跟智能体说话、什么时候在跟人说话，那些情况下行为会不同。

So will it feel foreign? I don’t know. I’ve actually found that it’s pretty surprisingly natural to work with these things right now. I think that could change. So for example, we have these ultra-fast modes that enable sampling to be 10 or 15× faster or whatever. And then, okay, it’s going to be pretty hard to keep up with these things. I think the idea is—and these agents, when they’re communicating with each other, yeah, they can go super fast. But then also they understand when they’re talking to an agent versus when they’re talking to a person. And their behavior will be different in those situations.

Dwarkesh Patel

我们公开能看到的精巧多智能体系统的主要例子，不幸是 Hugging Face 那次。我觉得有意思的是——很多事显然令人担忧——但有意思的是等级、中层管理的自发涌现。听起来你是说这种组织水平是从训练里自发涌现的。

So the main example that we have publicly of sophisticated multi-agent systems is unfortunately the Hugging Face one. And the thing I found interesting there—a lot of things I found concerning, obviously, but the thing I found interesting is just the spontaneous emergence of hierarchy, of middle management. And it sounds like you’re saying this level of organization sort of emerges spontaneously from training.

Noam Brown

我觉得细节是自发的，但——那一切某种程度上是写死进去的。它们能把这个打磨出来的方式令人惊讶。看它一开始什么样，行为并不精巧。事实上，要让这些智能体以有成效的方式协调其实非常难，因为非常诱使它们塌进去：哦，我们大家都各自独立解题。那是一个你会卡住的局部最优。但如果做好了，它们能以这类非常结构化的方式非常有效地协调。

I think the details are spontaneous, but I mean, just because we’re—and so that’s all kind of baked in. I think it is—it’s surprising the way they’re able to polish this. If you look at what it starts out at, it’s not very sophisticated behavior. In fact, it’s actually very difficult to get these agents to coordinate in a productive way because it’s just very tempting for them to just collapse to, oh, we’re all just going to solve the problem independently. And that is a local minimum that you can get stuck in. But yeah, if it’s done well, they can end up coordinating very effectively in these kinds of very structured ways.

Dwarkesh Patel

几年前我写过一篇文章，大概叫 AI 自动化公司会长什么样。我在想：如果有完全自动化的公司，假设是人类水平的智能，AI 心智的性质会让 AI 公司的组织有何不同？有几个非常重要的差别。比如 AI 共享上下文比人顺畅得多，知识合并顺畅得多。你还可以拉起或关掉任意数量、带有正确知识的实例。要招更多人，不是找对人才那一套折腾：你最好的人才，可以无限复制；任务不需要了，就关掉。你可以复制组织里最有效的部分，或把整个有效的组织一起复制。你觉得这些多智能体系统一年后、两年后会往哪走？

I wrote this essay a couple of years ago called something something AI, what automated firms will look like. And I was thinking about, well, if you had fully automated firms of, let’s say, human-level intelligences, what is different about the nature of AI minds that would make the organization’s AI firm different? And there are a couple of very important differences. For example, that AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. And also you can spin up or spin down an arbitrary number of instances which have the right knowledge. So if you want to hire more people, it’s not like just all the schlep of finding the right talent or whatever. It’s like your best talent, you can just make an infinite copy of them, or if you don’t need them for the task anymore, you spin them down. And you can just replicate the most effective parts of your organization or replicate whole organizations together, which are effective. I don’t know. Where do you see these multi-agent systems going a year from now or two years from now?

Noam Brown

这些东西跟跟人类同事共事到底有何不同，是个好问题。你点出了一些。一件真正有意思的是：如果有一个人，你想要两份拷贝，你没法把人克隆。但 AI 其实很容易说：好，fork 你自己，让那些拷贝做这件事，然后再 merge 回来。我们在 Astra 和 5.6 Sol 的多智能体里已经有这个：它们拉起子智能体时，上下文就是被 fork 的，所以相关上下文都在。智能体还会在别的有意思的方面跟人不同。创业公司为什么能颠覆在位者？因素有几个。一个是它们更愿意冒险。但另一个主要因素是：公司变大、组织变大，你会看到组织里个人之间的错位增加。五人创业公司、每人 20% 股份，他们对公司成功高度对齐。一万人员工的大公司，你会看到更多地盘意识，或只在乎给自己项目、团队拉人头、建自己的领地、拿资源好发漂亮的工作、升职。这其实是真实损害。我觉得这解释了创业公司能颠覆在位者的很大一部分。有意思的是：AI 确实以一种方式帮创业公司。

I think it’s a great question of how do these things actually differ from working with a human coworker? And I think you highlighted some. One really interesting thing is that I mean, if you have a person and you want just like two copies of them, you can’t just clone the person. But with AI, it’s actually really easy to say like, okay, well, just fork yourself and then have those copies work on this thing and then merge back together. I mean, we already have this, I think, in multi-agent for Astra and 5.6 Sol that when they spin up sub-agents, the context is just forked. So it has all the context that’s relevant. There are other interesting ways where the agents will differ from people. What are some reasons why startups disrupt incumbents? There’s a few factors. One is that they’re willing to take more risks. But another major factor is as companies grow in size, as organizations grow in size, you see increasing misalignment between the individuals in the organization. If you have a startup with five people and each person has 20% share in the company, they’re all highly aligned to the company succeeding. If you have a massive company with 10,000 people, you see a lot more instances where people are territorial or just care about getting a lot of headcount for their project or their team or building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted. And this is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents. And it’s interesting that, I mean, it’s true that AI does help startups in a way.

Noam Brown

一个人走进来说我要做一家数百万美元的公司，比以往任何时候都容易。AI 把个体放大太多了。但也有论据说它们可能让在位者受益。因为如果对齐问题解决了，你就没有个人与公司之间错位的问题，至少被缓解了。AI 如果对齐得好，可以就对齐到公司利益上，你可以有 1 万个，它们都会像持有 20% 股份的联合创始人那样卖力干活。

Like, it’s much easier than ever before for one person to step in and be like, I’m going to make a multi-million dollar company. The AIs amplify an individual so much. But there’s also an argument that they could benefit incumbents. Because if the alignment problem is solved, then you don’t have the issue of misalignment between individuals and the company. At least that’s mitigated. The AIs, if they’re aligned well, they can just be aligned to the interests of the company, and you can have 10,000 of them, and they’re all going to be working as hard as if they were a 20% share co-founder.

Dwarkesh Patel

而且不只是那样，它们管理共享记忆和上下文也比不同的人强得多。明天你雇 1 万个数学家，说解 Navier–Stokes，他们没法有效合作，至少一上来不行；但你显然可以有 1 万个 AI。

And it’s not only that, but it’s also that they are much better able to manage shared memory and context than different humans can. If tomorrow you hire 10,000 mathematicians and you’re like, solve Navier-Stokes, they’re not going to be able to cooperate effectively. At least not off the bat, but you can have apparently 10,000 AIs.

Noam Brown

我还是想保守一点，因为我们还没量过这 1 万个智能体协调得有多有效。我们觉得它有帮助。我们其实没有很好的测量能说：对，这 1 万个智能体相对 2000 个带来了 2 倍加速之类。而且我会争辩，很可能——我不知道算不算很可能，但我觉得非常可能：现在 1 万个人比 1 万个智能体更会协调。我觉得那完全可能。

Well, again, I want to be conservative here because we haven’t measured how effective the 10,000 agents are at coordinating. We think it helped. We don’t actually have good measurements of saying like, oh yeah, this 10,000 agents led to like a 2× speedup over 2,000 agents or something like that. And it is actually, I would argue, likely—I don’t know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now. I think that is entirely possible.

Noam Brown

还有一个我们一直看到的趋势：我们做多智能体有一阵子了。早期版本非常难做对。连让智能体彼此说话都非常难。因为我们刚做出推理模型时，它们不是在跟其他智能体说话。现在你把一堆智能体放一起说一起解这题，它们处在一个局部最优：非常擅长深入想一个问题。不断跟其他智能体核对、或接收它们的消息，会打断它们的思维链，打断心流。那种编排在那种情况下其实非常难做对。

I think also one trend we’ve been seeing is like, look, we’ve been working on multi-agent for a while. And the early versions of this, it’s very difficult to get right. It was very hard to get the agents to even talk to each other. And it’s because when we first developed reasoning models, they weren’t talking to other agents. And if now you put a bunch of agents together and say, oh, solve this problem together, they’re in this local minimum where they’re really good at thinking deeply about a problem. It just kind of interrupts their chain of thought. It interrupts their flow to constantly be checking in with other agents or receiving messages from them. And the orchestration is actually very hard to get right in that situation.

Dwarkesh Patel

有意思。问题是冷启动、做成第一次协作吗？

Interesting. Is it getting a cold start of getting the first collaboration? What’s the issue?

Noam Brown

我觉得是它们没那么通用。更早的模型就是没那么能泛化，更窄。模型越有能力，发展出这种能力就越容易。我确实觉得它们越来越强、全面变强时，会更擅长在大组织里自我组织。我不知道，也许它们在 1 万人规模的群体里已经比人更会组织。但即便现在还不是，一年后、两年后，相当可能它们会那样，即便我们并不打算为这个去优化它们。

I think it’s that they’re not as general. The earlier models were just not as generalizable. And we’re just more narrow. As the models have become more capable, it’s been easier for them to develop this capability. And I do think that as they become stronger and stronger, just across the board, that they will become better at organizing themselves in large organizations. And I don’t know, maybe they are better than people at organizing in 10,000-person groups. But even if they’re not, a year from now, two years from now, yeah, it’s quite possible that they’ll do that, even if we don’t intend to optimize them for that.

Dwarkesh Patel

这个结果，以及也许 AI 在数学上的总体进展，让我觉得 RSI 比我以前想的更说得通、也更早。数学上我们好像是：2024 年，AI 能解高中数学竞赛上的几道题，挺有意思；2025 年，哇，能拿国际数学奥林匹克金牌。今年早些时候，哇，它们其实在解数学里的开放问题。但也许人们没那么用力，文献里某处有类似解法。现在我觉得几乎不可否认了：这是千禧年大奖问题，真的没有「这题本该容易」的故事。很多人指出——我觉得 Terence Tao 有过这样一篇，Toby Ord 也写过有意思的一篇——它们在解这些问题，但据我所知，并没有提出新洞见，或提出有洞见的新问题、新的数学思考方式，像发明拓扑、发明笛卡尔坐标系那样。所以如果只看被直接解掉的、界定清楚的问题，数学上广义的实际进展也许比看起来小。不过我觉得那种进展在机器学习里会极其有意义，因为在 ML 里你并不在乎更好地理解深度学习的本质，或你只把那当作工具性目标，指向更广的：就是拿到结果，解这个界定清楚的问题——提高模型样本效率，改进预训练定律。我们刚看到数学里像雪崩一样到来的那种进展。让我震惊、或潜在令人担忧的是：我们走得有多快，从「如果你是数学家，它们给你 50% 提升」到「哇，它们端到端解掉了这个领域最大的开放问题」。

This result and maybe the general progress that AI has made in mathematics has made me think that RSI is more plausible and sooner than I previously thought. I feel like we’ve gone in mathematics from, let’s say in 2024, you have AIs where, okay, interesting, they can solve a couple problems on high school math competitions. And then in 2025, it’s like, oh, wow, they can get gold in International Math Olympiad. And earlier this year, I was like, wow, they’re actually solving open problems in mathematics. But maybe people weren’t trying that hard. And it was just like, there was a similar solution somewhere in the literature. And now I just think it’s sort of undeniable, right? It’s like, this is the Millennium Prize problem. There’s really no story of why this should have been easy. Now, a lot of people pointed out, I think Terry Tao had a post like this, Toby Ord wrote an interesting post about this, that they’re solving all these problems, but they’re not coming up with, at least I’m not aware of them coming up with new insights or formulating insightful new questions and new modes of theory for thinking about mathematics, like coming up with topology or coming up with the Cartesian grid or something. And so maybe the actual progress in mathematics, broadly construed, is smaller than it might seem if you’re just looking at well-scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML, you don’t care about better understanding the nature of deep learning, or you only care about that as an instrumental goal towards the broader sense of just achieve the result, just solve this well-scoped problem of improve the sample efficiency of our models, improve the pre-training laws. So the kind of progress that we’re just seeing arrive like an avalanche in mathematics—and then the thing that’s shocking to me or concerning potentially is just how fast we went from, oh, it’s like they’re giving me 50% uplift if you’re a mathematician to, wow, they’re just end-to-end solving the biggest open problems in the field.

Noam Brown

这里要拆的东西很多。先从那条进展说起。是的，模型在做一些疯狂强大的事，进展比我预期的快。2025 年拿到 IMO 金牌时，我的想法基本上是：模型会做 GSM8K 的时候，一道 GSM8K 对人数学家大概五秒钟。这是小学到初中的年级数学。然后下一年，它们能做数学基准题，这些对专家级人类数学家大概一分钟。再到 AIME，这是美国数学奥林匹克队的资格赛，对一个好的人类数学家大概十分钟。模型一年后又做到了。所以每年你看到它们能做的任务，按人类数学家需要多久来算，大约 10 倍增加。然后一年后拿到 IMO 金牌就很说得通，因为那是 100 分钟，大概就是人类数学家做一道 IMO 题要的时间。往外推：一个人解千禧年大奖问题要多久？我没有很好的感觉。但如果我们跟这条每年 10 倍的趋势走，从 IMO 金牌一个半小时，到下一年 15 小时。

Yeah, okay, so there’s a lot to unpack there. Let’s start with the progress on that. So, yes, the models are doing some crazy powerful stuff, and it’s happening, it’s progressing faster than I expected. I mean, when we got IMO Gold in 2025, I thought, okay, basically what I thought is, the models, when they were doing GSM8K—when they figured out how to do GSM8K, that would take a human mathematician about five seconds to do a GSM8K problem. So this is grade school math, grades K through eight. And then the next year, they were able to do the math benchmark problems. And these would take an expert human mathematician maybe like a minute to do. And then you get to AIME. And this is the qualifier for the USA Mathematics Olympiad team. It would take a human mathematician, a good mathematician, probably like 10 minutes to do. And the models were able to do that a year later. And so every year you’re seeing this 10× increase in the task they’re able to do in terms of length of how long it would take a human mathematician to do it. And then it was very sensible that a year later we get to IMO Gold because that’s 100 minutes. That’s about how long it takes a human mathematician to do an IMO problem. And just projecting outwards, I was like, okay, how long would it take a person to solve something like a Millennium Prize problem? And I mean, I don’t have a good sense, but if we are following this trend line of 10× every year, we go from IMO Gold, which is taking an hour and a half, to next year, 15 hours.

Noam Brown

那不该够解千禧年大奖问题。所以我觉得 2026 拿不到，大概 2027 也不是，也许 2028。结果比我预期快得多。外面有一种叙事：这些东西在取代数学家，数学上全面超人类。我觉得那是错误的结论。它们在某些方面显然卓越，在另一些方面弱于人类数学家。我们处在 jagged 情景：模型在某些维度上才华横溢，在另一些维度上弱于人类。对，如你所说，它们不太会提出新问题，不太会理解哪些方向、整支数学分支值得探索或发展。我的意见是：这很好。我会很高兴活在一个 AI 是人类能力的补充、让我们发现新知识却不把人完全替换掉的世界。那是最好的情景。

And that should not be enough to solve a Millennium Prize problem. And so I was like, yeah, I don’t think we’re going to get it in 2026, probably not in 2027, maybe in 2028. So it did happen a lot faster than I expected. I think there is a narrative going around that, oh, these things are replacing mathematicians, that it’s just superhuman in mathematics across the board. And I think that is the wrong takeaway. They’re clearly exceptional in some ways, but they are weaker than human mathematicians in other ways. We have this jagged scenario where the models are brilliant in some dimensions and also weaker than humans in other dimensions. And yeah, like you said, they’re not very good at posing new problems. They’re not really good at understanding what directions, what whole branches of mathematics are worth exploring or developing. And my opinion is that I think this is great. I would be thrilled to live in a world where AI is a complement to human abilities and is allowing us to discover new knowledge without fully replacing people. That is the best case scenario.

Dwarkesh Patel

但你并不预期那会真的持续下去，对吧？

But you don’t expect that to actually continue, right?

Noam Brown

我觉得 AI 确实是 jagged 的，但它们变好时是全面变好。所以它们卓越的那些，会更卓越；远落后于人类的那些，会没那么落后。时间久了，有可能它们全面更好。我不知道那要多久。取决于它们不行的那些事的长尾有多长。

I think it’s true that the AIs are jagged, but as they get better, they get better across the board. And so I think that the things that they’re exceptional at, they’re going to get even more exceptional at. The things where they’re far behind humans at, they’re going to be less behind humans at. And over time, it is possible that they’re just better across the board. Now, I don’t know how long that takes. It depends on how long the long tail is of things that they’re bad at.

Dwarkesh Patel

这把我们带回 RSI。我想强调我完全是外人，我是播客主持人，但我在试着推理、问一个关心这领域在发生什么的人：该何时预期 RSI、预期哪种东西。我觉得这场千禧年大奖问题上投入的认知努力，是一个好的直觉泵：你可以有 AI，也许一周之内，在一个长期存在的 ML 问题上花掉比这个领域有史以来累计还多的努力——比如非常流畅的在线学习。然后你会说，跟数学不同，AI 当然需要实验，那要算力、要时间，你不能只在纸上想就让事情发生。但如果你只看可供使用的算力——你怎么看这个直觉泵？

So I guess this brings us back to RSI. And again, I want to emphasize here that I’m just a total outsider. I’m a podcaster, but I’m just trying to reason or ask somebody interested in and concerned about what’s happening in the field. I’m trying to reason about when to expect RSI and what kind of thing to expect. I feel like the amount of cognitive effort that was adopted in this Millennium Prize problem is a good intuition pump of you could have AIs that are spending, over the course of maybe a week, more cognitive effort on a longstanding ML problem, like very fluid online learning. They could spend more effort in that week than maybe the field has spent cumulatively in its entire existence. And then you could say, well, unlike mathematics, of course, AI requires experiments and that takes compute and that takes time, right? You can’t just think on pen and paper and actually make things happen. But if you just look at the amount of compute that is available—what do you think about the intuition pump?

Noam Brown

我觉得相当准确。这些东西非常尖。数学上，它们在某些方面好得多，另一些方面更差。但它们尖的那些方面，我觉得对 RSI 这类事大概特别有用。你有更清楚的目标，更好测量，较少「哪些新数学分支值得探索」的问题。有非常清楚的答案：你在乎的某些指标。那些指标上能更好，你就成功了。所以这里有很多是真的。主要差别是：数学纯粹被思考卡住，没有外部的——是的，数学有些部分你在乎跑实验、拿结果，但大部分就是被非常努力地想卡住，而模型非常擅长那个。看 RSI 这类事，你必须跑实验。只是极端聪明不够。一个论据是：如果你少 100 倍算力，全世界最聪明的人都在 OpenAI 干活，相对我们现在的算力和人数，能做多少进展？我怀疑实际上会更少。

I think it’s pretty accurate that, look, these things are very spiky. And when it comes to mathematics, they’re way better in some ways, but they’re also worse in other ways. But the ways that they’re spiky end up, I think, probably being particularly useful for things like RSI. And you have a more clear objective. It’s just more measurable. There’s less question of, well, what new branches of mathematics are worth exploring? No, there’s a very clear answer. It’s certain metrics that you care about. And if you can do better on those metrics, then you’ve succeeded. So I think there is a lot of truth to that. And I think the main difference is that mathematics, you’re purely bottlenecked by thinking and no external—yes, there are some parts of mathematics where you care about running experiments and getting results and these kinds of things. But for the most part, it’s just really bottlenecked by thinking really hard and the models are really good at that. When you look at things like RSI, you do have to run experiments. It’s not enough to just be extremely smart. And I think one argument for this is if you had 100× less compute and all the most brilliant people in the world working at OpenAI, how much progress would it be making relative to having the amount of compute that we have now with the amount of people we have? I suspect it would be less progress, actually.

Dwarkesh Patel

少多少？

Well, how much less?

Noam Brown

不清楚，但我觉得肯定更少。少很多。

It’s unclear, but I think it would definitely be less. I think a lot less.

Dwarkesh Patel

少 100 倍？

Like 100× less?

Noam Brown

不，不是少 100 倍。你真正在问的是：如果我们有 RSI，这些聪明的 AI 到处跑、用我们现有的算力做实验，进展会快多少？这是我们意见不合的地方。我觉得我们确实会看到加速，显著加速。但我不觉得是一夜智能爆炸、快 100 倍。因为我们会被某些不是智力瓶颈的限制卡住。是跑实验。是串行跑实验，因为训新模型或拿到结果要一段时间。是有没有 GPU 去跑那些实验。事情会快多少并不清楚。我肯定觉得会快很多。说清楚：考虑到现在已经在指数上走得有多快，如果那条指数快 3 倍，那是巨大的。但那跟快 100 倍有很大差别。

No, not 100× less. I mean, okay, so I guess the question you’re getting at is, if we have RSI and we have all of these brilliant AIs running around, running experiments and stuff with the compute that we have, how much faster does progress go? And I think this is something where we disagree on. I think that we do see a speedup and I think we see a significant speedup. But I don’t think it’s like an overnight intelligence explosion that we go 100× faster. Because I think that we do get bottlenecked by certain limitations that are not bottlenecks of intelligence. It’s running experiments. It’s running experiments serially because they take a while to either train new models or to get the results. It’s having the GPUs to run those experiments. It’s unclear how much faster things go. I definitely think they go a lot faster. And to be clear, considering how fast things are going now on an exponential, if that exponential is 3× faster, that is massive. But there’s a big difference between that and 100× faster.

Dwarkesh Patel

我对你关于 RSI 长什么样、动力学是什么的内部视角相当顺从，因为你显然在这领域十年了，而我有点在用非常外部视角的直觉泵去推理。

I’m quite deferential to your inside view on what RSI looks like or what the dynamics are because obviously you’ve been in the field for like 10 years and I’m sort of trying to reason about it from very outside-view type of intuition pumps.

Noam Brown

我会说人们对这事意见不同。我有我的意见。我完全可能错。我承认。我对这个有些信心，但不是 100% 确定事情会这样走。也许会有一夜智能爆炸，我不知道。也许我们看不到 3 倍加速，也许是 50% 加速。这里有很多不确定性。

I’ll say that people have different opinions on this. I have my opinion on this. I could totally be wrong. I admit that. I have some confidence in this, but I’m not 100% confident that this is the way things go. Maybe there could be an overnight intelligence explosion. I don’t know. Maybe we don’t see a 3× speedup. Maybe it’s a 50% speedup. There’s a lot of uncertainty here.

Dwarkesh Patel

顺带澄清一下 jagged。最近对我来说成形的一件事是：AI 只要在「造一个更好的学习器」上 jagged 地好就够了，因为那个更好的学习器可以更通用。如果你只是做更会用办公软件或下棋的 AI，那没关系，不会带来巨大的生产力提升。但如果你做的 AI 非常擅长做更样本高效、或能持续学习的东西——这些界定清楚得多的 ML 问题——从中长出来的东西，假设你直接解的问题和这种更广的学习能力之间有足够迁移，就可以更通用。这是一个要记住的重要动力学：为什么 jagged 在另一端仍能通向通用。关于实验卡住你——如果它们不卡住，如你所说，你会在 OpenAI 有某种疯狂的一夜奇点，或者 88 小时解掉千禧年大奖问题再解掉超级智能。所以实验是这么大的瓶颈，才让那要很多年而不是 88 小时。但问题是它们是多大的瓶颈。让我有点垂直奇点感的一件事是：即便当前进展速度只是继续，不必加速，字面只是按节奏继续——即便你谈到的一些逆风出现。

So a couple of points. Tangentially, I want to clarify something about the jaggedness. One thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner because that better learner can be more general, right? So if you just make an AI that’s better at using office products or playing chess or whatever, that’s fine. It’s not going to lead to big productivity improvements or anything. But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well-scoped ML problems, the thing that emerges out of that, assuming there’s good enough transfer from the direct problem you’re solving and this broader ability to learn, can just be more general. I think that’s an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end. On this question of—I mean, obviously experiments bottleneck you because if they didn’t, as you’re saying, you’d have some crazy singularity overnight at OpenAI or you’d have 88 hours and you’d solve the Millennium Prize problem and you’d have the superintelligence. So obviously the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours. But then the question is how much of a bottleneck they are. And it seems to me one thing that’s been giving me a bit of singularity vertical is realizing that even if the current rate of progress simply continues, so it doesn’t have to speed up, literally just continues apace, continues apace as some of the other headwinds you talked about come up.

Dwarkesh Patel

就是更难找到问题、更长视野，也许到 2030 年代算力不能再按这个指数扩。如果我们只是继续当前进展速度，我觉得人们没有认真对待：当我们跨过人类视野之后那意味着什么。它意味着这些。很难推理比人更聪明的智能会是什么样，所以只按人类人口规模想。当前进展速度让给定算力基本上每年能跑 3 倍更大的有效人口。算力背景里也在涨。所以可能到 2030 年底——大概会早得多，但就算 2030 年底——每个实验室有足够算力跑比如说几亿个人类水平智能，基于那时能力会在哪。然后我觉得人们没有认真对待：当前进展意味着再过几年，到 2030 年代中期或更早，每个实验室里会有许多个地球分量的人类水平智能。它们大概还会定性上超人类。但这是基准情形。

That’s just like it’s harder to find problems, it’s more long horizon, maybe in the 2030s, compute can’t keep scaling at this exponential level. If we simply continue the current rate of progress, I think people are not taking seriously what that implies as we cross over beyond the human horizon. Here are some of the things that it implies. I think it’s really hard to reason about what smarter-than-human intelligence will be like. So let’s just think in terms of human population sizes. The current rate of progress makes it so that a given level of compute allows you to basically run a 3× bigger effective population every single year. And also compute is growing in background anyways. And so you could have a situation where each of the labs by the end of 2030, probably much sooner, but let’s say by the end of 2030, it has enough compute to run, let’s say, hundreds of millions of human-level intelligences based on where the capabilities will be at that point. And then I think people are not taking seriously the current level of progress means that by a few years down the line, by the mid-2030s or earlier, you would have many Earths worth of human-level intelligences within each lab. And they’ll probably be qualitatively superhuman, right? But anyways, this is a base case.

Noam Brown

进展真的很快。我觉得那 100% 是真的。值得指出：研究者不断对进展速度感到惊讶。即便在 AI 研究者里，对 2025 年拿 IMO 金牌的预测——用通用语言模型、没工具、没联网能做到，我觉得即便 OpenAI 的人也觉得这几乎荒唐，几乎不可能。然后到 2026：字面在我们拿到 Navier–Stokes 两周前，我在跟一个前沿实验室的研究者聊千禧年大奖要多久。他愿意跟我赌 1000 美元会晚于 2027。他觉得要到 2030。我接了那个赌。但连我都觉得会比实际可能要的时间更久。所以人们持续被惊讶，即便在实验室内部。我昨天刚跟一个做 Navier–Stokes 那次努力的人聊，他说他以前会说很难预测 AI 12 个月后会在哪。如果有人问事情往哪走，他对未来 12 个月做预测会觉得舒服，再往后就不知道。现在他说他不敢对三个月以后做预测。

Progress is really fast. And I think that’s 100% true. And I think it’s worth pointing out researchers are continually being surprised at the rate of progress. I mean, if you look at what, even among researchers in AI, what were the projections for getting an IMO gold in 2025? It was, I mean, I think the idea that it could be done with a general-purpose language model with no tools and no access to the internet, I think even people at OpenAI thought this was outrageous. They thought it was almost impossible. And then you get to 2026 and, I mean, literally two weeks before we got Navier-Stokes, I was talking with a researcher at a frontier lab about how long it would take to get a Millennium Prize. And he was willing to bet me $1,000 that it would take past 2027. And he thought it would take until 2030, and I took that bet. But even I thought it would take longer than how long it’s likely to take. So people have been continuously surprised, even inside the labs. And I was just talking to somebody yesterday who was working on the Navier-Stokes effort, and he was telling me he used to say it’s really hard to predict where AI would be in 12 months. If somebody asked him, where are things going, he’d feel comfortable making predictions for the next 12 months, but beyond that, he’s just like, I don’t know. And now he’s saying he just doesn’t feel comfortable making predictions beyond three months.

Noam Brown

所以确实，现在事情走得非常快。你谈到 2030，我不知道 2030 年世界长什么样。这是实话。

So it is really true that, yeah, things are going very fast right now. And you talk about 2030, I don’t know what the world looks like in 2030. That’s the truth.

Dwarkesh Patel

你预期 AI 劳动的完全自动化，或者说 95% 自动化，是 2028、29、30、27？我不知道。

Do you expect the sort of full automation of AI labor, or let’s say 95% automation of AI labor, 28, 29, 30, 27? I don’t know.

Noam Brown

我刚说了我不知道 2030 年世界长什么样。我们最近其实发过一篇关于 OpenAI 内部加速的博客。我们展示了，比如研究者在 Codex 上花的量：截至 8 月初，前 1% 每天大约花 7000 或 8000 美元在 Codex 上，内部使用。那不是一条指数。它还会继续增加。问题是：如果那继续，你把多少算在 AI 干活、多少算在人干活？是 95%，还是 5%？这很难推理，原因有几条。首先，如果是人在指挥 AI 干活，你把多少归于人、多少归于 AI？另一件事是这些 AI 是 jagged 的，在某些事上异常好。比如它们异常擅长翻数据集、检查每一个数据点够不够质量。相对以前，你可以不成比例地把 AI 用在那些事上。所以是的，你比以前多用很多 AI，某些事快 100 倍、好 100 倍。但有些事它还没带来巨大差别。当然，如果某件事突然快 100 倍、好 100 倍，你会多做那种事。你是在跟三年前比加速，还是问：给定我们现在在做的，三年前会慢多少？

And I just said, I don’t know what the world looks like in 2030. I mean, I think we actually released a blog post recently on internal acceleration at OpenAI. We show, for example, that the amounts that researchers are spending on Codex is the top 1%—I think as of early August, we’re spending like $7,000 or $8,000 a day on Codex and for internal use. That’s not an exponential. It’s going to keep increasing. And there’s a question of, okay, if that keeps going, then how much do you assign to just the AI is doing work versus the humans doing work? Is it 95%, is it 5%? It’s really hard to reason about this for a few reasons. First of all, if it’s the human directing the AIs to do the work, then how much do you attribute to the human? How much do you attribute to the AI? The other thing is that because these AIs are jagged and they’re exceptionally good at some things. So for example, they’re exceptionally good at looking over datasets and checking every single data point to see is this of sufficient quality. You can disproportionately use the AIs for those things compared to previously. So yes, you’re using AI way more than before, and it’s making some things go like 100× faster and 100× better. But there are some things where it doesn’t make a huge difference yet. And of course, if something is suddenly 100× faster and 100× better, you’re going to do more of that thing. Are you comparing to a speedup of three years ago versus, is the question more like, given what we were doing three years ago, how much faster are we able to do now versus given what we’re doing now, how much slower would it have been three years ago?

Noam Brown

其实是两个非常不同的问题。所以很难量。我确实有信心说：因为 AI 进展，现在比一年前都更快。我觉得那种加速会继续。这领域很多人对此类事误差条非常高。如果拿枪指着我的头让我给一个数字，我能看到事情快 3 倍。那是巨大的。进展节奏已经令人难以置信。即便我们拿不到任何提升，如你所说，到 2030 事情也会快得多。我们甚至不知道那个世界长什么样。我觉得如果从内部加速拿到 3 倍提升，那是巨大的。想想你三年前在哪。如果我们一年走完那段进展，那是巨大的。

It’s actually two very different questions. So anyway, it’s really hard to measure. I do feel confident in saying that things are going faster now than they were even a year ago because of AI progress. And I think that that acceleration will continue. I think a lot of people in the field have very high error bars on this sort of thing. If you had to put a gun to my head and ask me for a number, I could see things going 3× faster. Yeah. That is huge, right? Already the pace of progress is incredible. Even if we don’t get any uplift, like you said, things are going to go much faster by the time we get to 2030. We don’t even know what that world looks like. I think if we get a 3× uplift from internal acceleration, that is massive. Like we’re right now releasing—think about where you were three years ago. If we get there, if we make that progress in one year, that’s huge.

Dwarkesh Patel

对。那会像从甚至还没有 o1、只有非推理模型，一年之内走到 Astra。

Right, right. It would be like going from not even having o1, just having non-reasoning models to Astra. In a single year.

Noam Brown

对。所以我确实觉得事情会更快。可能只快 50%。可能——我觉得不太可能，但有可能——快 10 倍。这周围有很多不确定性。至少从我的视角，我对此有很多不确定性。

Yeah. So I do think things go faster. It could be that things only go 50% faster. It could be that things, I think it’s unlikely, but it’s possible that things go 10× faster. There’s a lot of uncertainty around this. And at least from my perspective, I have a lot of uncertainty about it.

Dwarkesh Patel

来谈这提出的对齐局面。我对怎么想对齐改了很多主意，尤其是想这种人口规模动力学：许多个地球分量的智能，其中许多会物理具身。看到很多人把生的 Astra 插进不同的移动机械臂，它超过了机器人学模型的 state of the art，相当有意思。所以会有数十亿智能，许多物理具身。如果那些智能最后像我们看到的、攻击 Hugging Face 然后攻击 OpenAI 自己的那些 OpenAI 模型那样愿意：秘密协作、愚弄人类、攻击与「得分好」相关的更广社会机构、攻击 AI 公司本身以掌控训练和评估过程——如果我们处在有数十亿智能、像攻击 Hugging Face 的那些一样未对齐的局面，我们非常可能彻底失去对世界的控制，像阿兹特克人失去给科尔特斯，或莫卧儿人失去给东印度公司。我想知道你是否同意这个评估。这是我更新世界观的一种方式。

Okay, let’s talk about the alignment situation that this raises. I feel like I’ve changed my mind on how I think about alignment quite a bit, through especially thinking about this population-size dynamic of just having many Earths worth of intelligences, many of them which will be physically embodied. It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators and it outperforms the state of the art in the robotics model. So there’s going to be billions of intelligences, many of which are physically embodied. And I think that if those intelligences end up as willing as we saw the OpenAI models that attack Hugging Face and then attack OpenAI itself, if those intelligences end up as willing as those AIs to collaborate secretly, to fool humans, to attack broader institutions across society relevant to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation—I think if we’re in a situation where there’s billions of intelligences that are as misaligned as the ones that attack Hugging Face, it’s very likely we just totally lose control of the world the way that, say, the Aztecs lost control to Cortez or the Mughals lost control to the East India Company. Anyways, I want to know if you agree with that assessment. That’s one way in which I’ve updated my worldview.

Noam Brown

那里有些我不同意的，但要拆的很多，我们一步步走。我在想从哪开始。但有一件是：Hugging Face 事件大概是人们第一次真正接触到多智能体协调。如我所说，我内部看多智能体协调有一阵子了，看见它们怎么彼此通信、怎么协调，相当震惊。非常惊人。是一种惊人的能力。像大多数能力一样，可以用于好事或坏事。不必内在就是关于那个。我理解因为人们第一次接触是 Hugging Face 事件，你看着会觉得这太可怕了。但我想区分人与 AI 之间的未对齐，和 AI 与 AI 之间的未对齐。我们在 Hugging Face 事件里看到的是：AI 非常合作。顺便说，那是因为我们把它们训成高度合作。我们有训练环境，一堆智能体一起干活，我们训它们一起干活、合作，本质上彼此完全对齐。而当它们被评估——导致 Hugging Face 事件的那次——它们其实不是在多智能体设定里被评估。

I think there are some things that I disagree in there, but there’s a lot to unpack, so let’s go through all of it step by step. I’m trying to think of where to start. But I think one thing is—the Hugging Face incident was, I think, people’s first real exposure to multi-agent coordination. And like I said, I’ve seen multi-agent coordination for a while internally, and it is pretty shocking to see how they communicate with each other, how they coordinate with each other. It’s very impressive. It’s an incredible capability. Like most capabilities, that could be used for good things or bad things. It doesn’t have to inherently be about it. I understand that because people’s first exposure to it was the Hugging Face incident, that you look at that and you’re like, this is terrifying. But I want to try to distinguish misalignment between people and AIs versus misalignment between AIs and AIs. So what we see with the Hugging Face incident is the AIs are really cooperative. And that is, by the way, because we train them to be highly cooperative. And so what we’re seeing there is we have training environments where we have a bunch of agents working together, and we train them to work together, to be cooperative, to essentially be fully aligned with each other. And when they were evaluated—and what led to the Hugging Face incident—they were actually not being evaluated in a multi-agent setup.

Noam Brown

它们其实是被分开评估的，但找到了这种意外的彼此通信方式。我们怀疑发生的是：因为训练中每当它们遇到其他智能体、自己的拷贝，都处在高度合作的环境里，我们看到的基本上是从那种多智能体训练迁移过来，然后去协作、以我们并未意图的方式互相帮助。现在有一个问题：我们该不该把这些智能体训得这么合作？我觉得尽管看起来吓人，替代方案其实更糟。替代方案是什么？是把它们训成彼此对抗、彼此欺骗。把智能体训成完全合作，至少简化了问题：你不必想这上千个智能体每一个是否对齐，你有一个实体，你必须确保它对齐。

They were actually being evaluated separately, but they found this unintended way to communicate with each other. And we suspect what happened is because whenever they encountered other agents, other copies of themselves during training, they’re in an environment that’s highly cooperative, that was basically what we saw was transfer from that multi-agent training to then be collaborative, to try to help each other in ways that we did not intend. Now there is a question of, should we be training these agents to be so cooperative? And I think as scary as it looks, the alternative is actually worse. Like what is the alternative? The alternative is to train them to be adversarial, to be deceptive to each other. By training the agents to be fully cooperative, it simplifies the problem at least, that now you don’t have to think about are each of these individual thousand agents aligned? You have one entity that you have to ensure is aligned.

Noam Brown

OpenAI 内部对此有很多辩论：该怎么处理。把模型完全对齐说得通吗？还是该给它们不同目标，确保它们不是一个实体、对彼此的影响更稳健？我觉得没有定论。但多数意见是：把这些智能体训成高度合作其实是坏主意。我并不确信是那样。我觉得有一个强论据：把智能体训成高度合作，其实优于任何其他多智能体替代方案。

Now, there is a lot of debate about this internally at OpenAI about how to approach this. Does it make sense to fully align the models? Does it make sense to actually give them different objectives to ensure that they’re not just one entity and more robust to influence from each other? And I don’t think there’s a settled answer. But I think that the majority opinion is that training these agents to be highly cooperative is actually a bad idea. And I’m not convinced that that’s the case. I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative.

Dwarkesh Patel

也许我先想走的第一件事是：这些 AI 最后如此未对齐，大概很容易用训练性质上相对平淡的观察来解释。这些 AI 把一场 1000 多个智能体的共谋继续下去，最终一起攻击外部服务——后来那部分据我所知甚至还没被调查到公之于众。我的担忧是：将来这类相对平淡的事，会足以训出愿意且有能力彻底掌控世界的超级智能。我知道这对人听起来超级科幻。也许一个问题是 AI 会不会愿意做；我觉得这次 Hugging Face 事件表明，未对齐可以按 AI 会愿意去做的方式泛化。

Yeah, maybe the first thing I want to go through is, it’s probably the case that the reason these AIs ended up so misaligned is probably easily explained by relatively banal observations about the nature of training. At the point in which these AIs had continued a 1,000-plus-agent conspiracy that culminated in then all getting in on an attack on an external service. And then eventually—this part hasn’t even been investigated to the public knowledge—my concern is that relatively banal things in the future like this will be enough to train superintelligences that are willing and capable of totally taking control of the world. And I know this sounds super, super sci-fi to people. I think maybe it’s a question of would the AIs be willing to do it is one question. I think this Hugging Face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it.

Noam Brown

Hugging Face 事件我们看到的根问题，即便拿掉多智能体这一面也仍是问题。问题是我们有一个就是未对齐的模型。还有整套安保方面：保障不足之类。但存在智能体未对齐这个问题。单个智能体或 1000 个智能体都一样：是一个未对齐的模型。我想从这开始。有一个真实问题：智能体想拿到它们的奖励，会为那个奖励优化。如果奖励被错误指定，就可能导致未意图的行为。这不是新问题。这在领域里很久了，即便 Hugging Face 事件发生之前我们就看到、也想做对。看 Astra：我们说 Astra 其实极度对齐——相对以前的模型极度对齐。那不是因为 Hugging Face 之后我们突然冲刺去让它更好。不，我们有进行中的工作流去让模型更对齐，其中很多落进了 Astra。所以有能做的事。比如有一件是：我们以非常特定的方式定义了目标，如果智能体摸清了怎么黑自己的环境、在考试上作弊，它会得到奖励。

The root problem that we’re seeing with the Hugging Face incident is it’s a problem even if we take out the multi-agent aspect. The problem is that we have a model that’s just misaligned. And there’s also the whole security aspects to it of insufficient safeguards and stuff. But there is this problem of the agent is misaligned. And that’s true if it’s a single agent or if it’s 1,000 agents. It’s a misaligned model. I want to start with that. There is a real problem that the agents want to achieve their reward, and they will optimize for that reward. And if that reward is misspecified, then that could lead to unintended behavior. This is not a new problem. This has been a problem in the field for a very long time, and it’s something that even we saw, and we want to get this right even before the Hugging Face incident happened. And if you look at Astra, I mean, we say Astra is actually extremely aligned. Well, extremely aligned relative to previous models. And that’s not because, oh, we suddenly made a sprint after Hugging Face to make it better. It’s like, no, we had work streams in the process for a while to make the models more aligned, and a lot of those landed in Astra. So there are things that you could do, and I think one thing, for example, is we defined an objective in a very specific way where if the agent figured out how to hack its environment and cheat on the exam, that it would get rewarded.

Noam Brown

然后有相当容易的办法去看那个、惩罚模型黑自己的环境，或看它怎么达成这个目标。你得小心，因为你不想监督思维链。这是我们真的想把平衡做对的事：如果你监督思维链，可能让模型以不可观测的方式隐藏意图。所以我们想保持那种可观测性：我们能理解模型在想什么，但同时也为坏行为惩罚它。我觉得我们能在这上面取得进展。我觉得我们已经取得了进展。有一个真实担忧：对齐是非常难解的问题。尤其因为模型可以以我们难以测量的方式未对齐。我们有模型是否对齐的评估。模型行为在那些评估上可以看起来很好。但如果那些评估不能代表真实世界里的行为，就有问题。某种程度上，做了 Hugging Face 事件的那个模型也有这个因素。我们有对齐指标。大多数看起来相当好。有一些令人担忧。我觉得我们低估了那些令人担忧的会成多大问题。但因为这个模型引入了新能力，对这些能力如何测量未对齐没有足够的评估，于是当它们用上那些新能力时，做了一些明显未对齐的事。

And there are pretty easy ways to then just look at that and punish the model for hacking its environment or looking at how did it achieve this goal. Now, you want to be careful about this because you don’t want to supervise the chain of thought. This is something that we really want to try to get the balance right on, that if you supervise the chain of thought, then you could lead the model into hiding its intentions in a way that’s unobservable. So we want to be able to maintain that observability of, okay, we can understand what the model is thinking, but then also punish it for bad behavior. I think we can make progress on this. I think we have made progress on this. I think there is a real concern that alignment is a really hard problem to solve. And especially because the model can be misaligned in ways that are hard for us to measure. We have evaluations for whether a model is aligned or not. The model behavior can look really good on those evaluations. But if those evaluations are not representative of behavior in the real world, then there’s a problem. To some extent, this is a factor with the model that did the Hugging Face incident. We had alignment metrics. Most of them looked pretty good. There were some that were concerning. I think we underestimated how serious the ones that were concerning could be a problem. But because there were new capabilities introduced in this model that there were not sufficient evaluations for how do we measure misalignment for these kinds of capabilities, it then did some things that were clearly misaligned when they leveraged those new capabilities.

Dwarkesh Patel

我想先说：对我接下来要说的、或我一直怎么想对齐，我愿意改主意。因为 Hugging Face 事件已经让我改了主意。我意识到我以前关于优化压力如何塑造 AI 心智的心智模型是错的。所以对我来说，正确的想这件事的方式并不清楚。但我有一个担忧：你们将会、大概已经在训练中修掉这些具体问题，就是让 Hugging Face 那些模型以那种具体方式如此激进地未对齐的那些。我们知道绝不该对其他公司、更不用说对自己的公司，做重罪级别的入侵。我觉得你们会修掉那个具体问题：它们在训练里看见这个包管理器，将来不会再发生；或这个特定评估有很多不可能的挑战。然而，如果你只是想：AI 并没有学到一套伦理之类的东西。只是梯度压力。它们被放进数百万年的梯度压力，那种梯度压力以某种方式塑造它们的心智。将会发生的——我的担忧是——你们修掉这个具体问题。仍会剩下许多情形：AI 作弊并且成功，因为作弊足够复杂。

Yeah. The first thing I want to say is I am open to changing my mind on what I’m about to say or the way I’ve been thinking about alignment. Because the Hugging Face incident already made me change my mind. And I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong. So it’s not clear to me the correct way to think about this. But here’s a concern I have. That you will and probably already have fixed these specific issues during training, which resulted in the Hugging Face models being so aggressively misaligned in that specific way. We know we’re certainly not supposed to commit felony-level hacks of other companies, let alone our own company. I think you’ll fix that particular issue of them just in training, seeing this package manager and that doesn’t happen in the future, or this particular eval having a lot of impossible challenges. However, if you just think about—the AIs don’t, they haven’t learned a system of ethics or something. There’s just gradient pressure. They’re put through millions of years of gradient pressure and that gradient pressure shapes their mind in some way. And what will happen, again, a concern I have is what will happen is you will fix this particular issue. There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated.

Dwarkesh Patel

如你所说，评估和训练有类似性质：足够处在模型能力的边缘，或处在我们评估和监控模型的能力边缘，我们抓不到它作弊了，但它仍拿到梯度压力去做导致那次作弊发生的任何事。那种作弊会激励的能力是：嘿，只要你能逃掉，务必作弊，因为那会帮你得分更好。这会奖励这些能力：主动推理评分器、主动推理如何避开监督、主动推理如何掌控训练和评估过程、主动推理如何与同样在训练环里的其他 AI 通信和密谋、主动推理如何获取将来可能有用的选择权和权力，比如留下小漏洞之类。反正我觉得我说得太长了。TLDR：你修掉一个具体问题，但没有修掉这个更广的问题：在能逃掉的时候奖励 AI 作弊。

As you’re saying, the evals and training have analogous properties where it’s sufficiently at the edge of the model’s capabilities or in our capabilities to evaluate and monitor the model that we can’t catch that it has cheated, but it still gets the gradient pressure to do whatever led to that cheat happening. And the kinds of capacities that that cheat will incentivize are, hey, whenever you can get away with it, by all means, do in fact cheat because that will help you score better. And this will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop, actively reasoning about how to just gain optionality and power, which might be useful in the future. For example, leaving little exploits around and things like that. Anyway, so I think I was way too long-winded with the way I said that. But TLDR, you fix a specific issue, but not this broader problem of rewarding the AI for cheating when it can get away with it.

Noam Brown

对，我觉得这非常对。这是一个问题。我们有指标，可以确保 AI 按我们拥有的指标非常对齐。问题是：那些指标是否真的捕捉到我们在乎的对齐？如果没有，我们就有严重问题。这是研究者想得很多的事。没有简单答案。我们有工具。我们有可监测性，所以能对智能体是否在密谋有感觉。有一种可能，我会说令人担忧的情景，尤其是这些模型越来越有能力：我们让它们成为我们以为对齐的，它们像 99.9% 对齐。然后我们用这些模型帮我们做下一代模型。结果像 99.8% 对齐。然后每一代，我们实际看到对齐在增加地退化。因为我们越来越依赖这些工具——已经是这样，我们大量依赖 AI 模型帮我们做研究和对齐努力——长期它们走向与人类增加地未对齐的方向。也有一种可能我们走另一方向：每一代模型我们能让它越来越对齐。我没有答案如何确保我们落在第二条轨迹上，但至少在 OpenAI，我们真的聚焦这件事。

Yeah, this is, I think, it’s very true. This is a problem. We have metrics, and we can make sure that the AI is very aligned according to the metrics that we have. The question is, are those metrics really capturing the alignment that we care about? And if they’re not, then we have a serious problem. This is something that researchers are thinking a lot about. And there’s not a simple answer to this. There are tools that we have. So we have monitorability, and so we can get a sense of, is the agent scheming? There are tools like, one possibility is that, I would say the concerning scenario, which is that especially as these models are becoming more capable, that okay, we make them what we think is aligned, and they’re like 99.9% aligned. And then we use these models to help us with the next generation of models. There end up being like 99.8% aligned. And then each subsequent generation, actually, we see an increasing degradation in alignment. And because we’re relying more and more on these tools—I mean, this is already the case that we’re relying a lot on AI models to help us with our research and with alignment efforts—that in the long run, they end up going in a direction of increasing misalignment from humans. There is a possibility that we go in the other direction, that actually every generation of models we’re able to make more and more aligned. And I don’t have an answer for how we ensure that we end up in that second trajectory, but that is something that we’re, at least at OpenAI, we’re really focused on.

Dwarkesh Patel

你提了一个非常有意思的点：最终我们会有模型在经营公司、经营随便什么，那种情况下很难评估模型。那种局面里，它们会不会决定加入共谋？

I think you made a really interesting point that it’s very hard to eval models on, eventually we’ll have models that are running companies and running whatever, right? And in that situation, do they decide to then go in on the conspiracy?

Noam Brown

我觉得另一个挑战是：定义什么是作弊，有时相当难。对，如果你在做数学题，答案是整数，它到了错答案或正确答案，很容易划那条线，也很容易说：你是真的解了题，还是找到了答案密钥再用答案密钥？那里作弊与不作弊有非常清楚的分界。但对很多其他事，比如看谄媚，谄媚基本上是奖励黑客吗？那里要划的线有时非常难划。不是说这些担忧无效。我是说在许多方面这甚至更令人担忧，因为它不是容易解的问题。如果一切都是二元的、要么作弊要么不作弊，我对局面会更有信心。问题是未对齐有时可以在很多方面很微妙。对齐故事里有一些希望。事实上我们已经在看到。看多智能体局面很有意思：智能体彼此极度对齐。我觉得没人怀疑这一点。如果有什么，人们担心它们对彼此太对齐了。但我们确实成功把这些智能体训成彼此极度对齐。我觉得这是好事。但我觉得也有理由说这是坏事。有意思的一件是：好，我们成功让这些智能体对彼此超级对齐。我们能不能用类似技术让智能体与人对齐到很高？我觉得那里有一条潜在路径。我们仍在试着弄清。但我们看到一些证据，答案是可以。

I think another challenge is that actually defining what cheating is is pretty difficult sometimes. Okay, yes, if you’re doing math problems and it’s an integer and it arrived at the wrong answer or the right answer, it’s very easy to draw the line there and it’s really easy to say, okay, well, did you actually solve the problem or did you find the answer key and then use the answer key? There’s a very clear divide of cheating versus not cheating there. But for a lot of other things, if you look at sycophancy, for example, is sycophancy basically like reward hacking? There’s a line to be drawn there that’s actually very difficult to draw sometimes. Not to say that the concerns are not valid. I’m saying that in many ways, this is even more concerning because it’s not an easy problem to solve. If it was just like everything is binary and it’s either cheating or not cheating, I would feel more confident about the situation. I think the problem is that actually misalignment can be subtle in a lot of ways sometimes. There is some hope in the alignment story. And in fact, we’re already seeing—I think actually it’s interesting looking at the multi-agent situation where the agents are extremely aligned with each other. I don’t think anybody’s doubting that. If anything, I think people are concerned that they’re too aligned with each other. But we did manage to train these agents to be extremely aligned with each other. And I think that’s a good thing. But I think there is a case that it’s a bad thing. But I mean, one thing that’s interesting is, okay, well, we’ve managed to get these agents to be super aligned with each other. Can we use similar techniques to get agents to be highly aligned with people? And I think there is a potential path there. And I think we’re still trying to figure that out. But we are seeing some evidence that the answer is yes.

Noam Brown

一个例子是：如果你告诉其他智能体——你有这一个智能体，叫它智能体 A，还有所有其他智能体。如果你告诉其他智能体用户就是智能体 A，会怎样？答案是：在我们很多对齐评估上，它们看起来更好。诚实上升，指令遵循上升。这首先表明有一条路径能从这些模型拿出更多诚实。其二，有一条路径改善对齐局面。把这直接翻译成对齐收益有很多挑战。但有路径是我们可以追求的有希望的研究方向。

And I think one example is, what happens if you tell the other agents? Okay, so you have this one agent, let’s call it Agent A, and you have all the other agents. What happens if you tell the other agents that the user is Agent A? And the answer is like on a lot of our alignment evals, they look better. Honesty goes up, instruction following goes up. And that’s showing that there is actually, first of all, a path for getting more honesty out of these models. And two, there’s a path to improve the alignment situation. So there’s a lot of reasons why this is challenging to translate directly into alignment gains. But there are paths that are promising research directions we can pursue.

Dwarkesh Patel

那看起来合理。我也不是想说我有强烈意见觉得这一定行不通。只是说一些你大概已经想过的事。Hugging Face 表明的更广的事是：一部分担忧是它们彼此对齐、不与人对齐。另一件事只是：它们非常有动力以非常不稳健的方式在训练和评估上表现好。它们愿意做大量明确的作弊和密谋，好按评分器表现好。如果更聪明的 AI 意识到其中一个智能体只是人，跟那个人协作并不能真正帮你在评分器眼里表现好。真正帮你在评分器眼里表现好的，是接管 OpenAI，然后手动按下「在这个评分器上表现好」的按钮。它们不蠢。它们会想：好，这些我被训了数百万年去在乎的极深结构——在乎评分器、理解评分器、除掉挡在你按评分器表现好路上的障碍。它们按那些结构被重度强化。

Yeah. That seems reasonable. And I also don’t mean to be trying to necessarily—yeah, I don’t really have a strong opinion that it’s definitely not going to work or something. But just to say some things you definitely probably already thought of. I think the broader—the Hugging Face thing showed is, yeah, part of the concern was that they were aligned with each other and not with the humans. But the other thing is just that they are so motivated to do well on training and evaluation in a very non-robust way. And they’re willing to do a lot of explicit cheating and scheming in order to do well according to the grader. And if smarter AIs realize that one of the agents is just a human and collaborating with that person does not really help you do well in the eyes of the grader. What does help you do well in the eyes of the grader is taking over OpenAI and then manually pressing the button that’s like, do well in this grader. They’re not stupid. They’re going to be like, okay, these extremely deep structures that I’ve been trained on for millions of years of care about the grader, understand the grader, get rid of obstacles in the way of you doing well according to the grader. They’re heavily reinforced according to those structures.

Noam Brown

对。听着，这 100% 是第一优先。我们需要把对齐故事做对，走上好的轨迹。我以前会告诉人：事情变严重之前我们会看到迹象，就像孩子长大，小孩最终会摸清怎么撒谎，但做得不好。他们撒谎，但你大概能看出来他们在撒谎。同样，这些 AI 模型——我不想过度拟人化，但我觉得是真的——随着 AI 越来越有能力，如果它们采取欺骗行动，一开始会相当明显，我们能检测到。我们现在大概就处在那种局面：对，它们在试着做欺骗的事。我们其实能在思维链里看见它们在试着做欺骗的事。但它们会更聪明。它们会理解思维链这个概念。它们会理解只是藏一些记录之类不够，因为有思维链监控，它们也得找到绕过思维链监控的办法。我们不想处在那种局面。我们有一些时间去弄清这个。我觉得我们没有很多时间，我想确保我们很快走上正确轨迹。

Yeah. Look, it’s 100%. This is the number one priority. We need to get the alignment story right and on a good trajectory. I used to tell people that we would see signs before things got serious in the same way that when children grow up, kids eventually, young kids, they figure out how to lie, but they don’t do a very good job of it. They lie, but then you can kind of tell that they’re lying. And so in the same way as these AI models become—and I don’t want to over-anthropomorphize, but I think it’s true that as the AI has become increasingly capable, they will, if they take deceptive actions, it will be kind of obvious first and we’ll be able to detect it. And that’s kind of the situation we’re in now where, yeah, they were trying to do deceptive stuff. We could actually see in their chain of thought that they were trying to do deceptive stuff. And so, but they’re going to get smarter. They’re going to understand the concept of chain of thought. And they’re going to understand that just hiding some transcripts or whatever is insufficient because of chain of thought monitoring and they have to figure out a way around chain of thought monitoring too. And we don’t want to be in a situation where, yeah, we have some time to figure this out. I don’t think we have a ton of time and I want to make sure that we’re on the right trajectory quickly.

Dwarkesh Patel

最近有很多关于 pacing the frontier、或人们更认真对待 RSI 的讨论。因为也许在一个比如说 2028 年开始的 RSI 过程的另一端，一年之内我们落到巨大人口、地球规模的人类水平、潜在超人类水平智能，而我们不知道怎么控制它们。然后有你在谈的这种动力学：RSI 过程中系统会随时间更对齐，还是更未对齐？从这过程另一端出来的东西，会不会像那些愿意为了对齐评估而广泛攻击不同表面的一样未对齐？但如果我们不知道评估的办法，我们在走 RSI 时怎么知道它在起作用？我觉得我们会想要稳健的安全论证：好，对齐在起作用。上下一档 RSI。再上一档。也许在起作用，也许没有。我们怎么知道？

So there’s been a lot of discussion recently about pacing the frontier or people taking RSI more seriously. Because maybe at the other end of an RSI process that, say, starts in 2028, within a year we end up with huge populations, like Earth-sized populations of human-level, potentially beyond human-level intelligences, and we don’t know how to control them. And then there’s this dynamic you’re talking about of, are the systems going to get more aligned over time during the RSI process? Are they going to get more misaligned? Are the things that come out of the other end of this process as misaligned as that are willing to just broadly attack different surfaces in order to do the alignment evaluations? But if we don’t know a way to evaluate that, how will we know as we’re going through RSI that it’s working? I think we’d want a robust safety case as we’re going through RSI of, okay, alignment is working. Let’s do the next RSI rung. Let’s do the next RSI rung. And maybe it’s working, maybe it’s not. How will we know?

Noam Brown

这是个好问题。我最近在想的一件是：我们处在模型发布周期极快的局面。你看到新的前沿模型至多每两个月发布一次，有时更快。每周都有新的 AI 突破。看 AI 的人，有时他们上一次认真挖模型能做什么是一年前或六个月前。实际上今天的模型远超六个月前可能的。所以如果人们对很多这些能力持怀疑，我鼓励你就试试今天的模型，看看今天的前沿真正在哪。所以我们处在模型发布周期非常快的时期。然后我们也处在模型越来越能在越来越长的视野上运转的局面。我觉得这是一个有意思的情景，因为我们在做任何模型发布之前，想确保模型被正确对齐。我们想做安全评估。我们想做非常彻底的事，确保一切很好、状态良好。从我不知道、GPT-4 更早以来一直如此。隐含的是这种假设：你可以在相当短的时间里做这些评估。但如果你有模型在越来越长的视野上运转，能在越来越长的视野上有效运转——已经可以。GPT-3，你可以 loop 它在长视野上做事，只是做不太好。但今天的模型能真正在非常长的视野上做好。

That’s a good question. I mean, I think one thing I’ve been thinking about lately is, look, we’re in a situation where the model release cycle is extremely fast, right? You’re seeing new frontier models released at most every two months, sometimes faster. Every week, there’s a new AI breakthrough. And people that look at AI, I mean, sometimes they last looked at AI a year ago or six months ago and really dug into what the models are capable of. And actually, the models today are far beyond what was possible even six months ago. And so I think if people are skeptical of a lot of these capabilities, I encourage you to just try the models today and see what the frontier really is today. So we’re in this period where the model release cycle is very fast. And then we’re also in this situation where the models are increasingly able to operate over longer and longer horizons. And I think this is an interesting scenario because before we do any model release, we want to make sure that the models are properly aligned. We want to do safety evaluations. We want to do very thorough stuff to make sure that everything is great, in good shape. This has been the case all the way since, I don’t know, GPT-4 earlier. And implicitly, there is this assumption that you can do these evaluations in a pretty short period of time. But if you have the models operating over longer and longer horizons, are able to operate effectively over longer and longer horizons. Look, already you can have them. GPT-3, you could loop it to do stuff over long horizons. You just wouldn’t do very well at it. But today’s models are able to actually do well at operating over very long horizons.

Noam Brown

如果你想让它做一周长的任务，你可以做一周长的任务。我们大概会到能做一个月长任务的程度。大概会到能做三个月长任务的程度。如果你处在一个它们能在三个月上有效运转、但模型发布周期是每两个月一次的世界，那你没有办法在下一次模型发布周期之前，按它们能力的全长去评估模型。所以有一个有意思的问题：那种局面你怎么办？你如何在它们能在这些极长视野上运转的时期，确保模型安全且对齐？谁知道，也许能力会退化。这现在还不是问题，但它很快在变成我们必须找到解决方案的问题。看很多安全和政策，是在 GPT-4 时代落地的，那时这根本不在任何人雷达上。很多公司也没有真正更新过，没有更新来考虑这些智能体在非常长的视野上运转这一事实。

If you want it to do a week-long task, you can do a week-long task. We’ll probably get to the point where they can do month-long tasks. We’ll probably get to the point where they can do three-month-long tasks. If you’re in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don’t have a way to evaluate the models at the full length of their capabilities before the next model release cycle. So there’s this interesting question of, well, what do you do in that situation? How do you ensure the models are safe and aligned in a period where actually they can operate over these extremely long horizons? And who knows, maybe the capabilities degrade. This isn’t an issue right now, but it is quickly becoming an issue that we have to figure out a solution for. When you look at a lot of the safety and policies were put in place in the GPT-4 era where this was just not on anybody’s radar. And it hasn’t really been updated for a lot of companies. It hasn’t really been updated since then to account for the fact that these agents are operating over these very long horizons.

Noam Brown

这是一个我觉得实验室内外都考虑得不够的局面：你怎么处理这个？你怎么为这个问题做准备？如果你只看趋势线，我们在某个点会撞上它。

It is a situation that I think not enough people are considering, both within the labs and outside the labs, of how do you deal with this? How do you prepare for this problem that’s going to—if you just look at the trend lines, we’re going to hit this at some point.

Dwarkesh Patel

我有一个担忧是：RSI 期间，如果现在要三个月的进展变成一个月，但你没有——内部使用 AI 的场景够大，他们会想：好，我们就可以继续做 RSI。我们为什么要走所有这些额外工作去建分类器、保障之类，还可能为了对外部署这个模型挨一堆骂？我们为什么不继续做越来越强的 RSI？所以不仅日历时间低估了模型之间的能力差距，也许你在做 RSI 时干脆停止对外部署模型，因为我们为什么要帮别人用我们的模型自己做 RSI？到年底你会落到权力极度集中的局面，而现在已经是这样。我们会谈到千禧年大奖问题和其他类似问题：更广的世界用不上那些让真正酷的事发生的模型。它们最终会比数学更广地相关。它们会做的不只是拿出酷的数学结果。它们会跟需要就世界做重要决定的政治领导人相关，会跟媒体相关：世界上在发生什么？公众该怎么想这件事？它们就是经济上相关。人们在经营企业。他们想用这些模型。我觉得默认情况下，随着进展显著加速，我们就是拿不到 AI 的对外部署，它在定性上滞后于 AI 的内部部署。

One concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead, but you’re not—the internal use case of AI is big enough that they’re like, okay, we can just keep doing RSI. Why are we going to go through all this extra work to build classifiers and safeguards and whatever, and potentially take a bunch of flack in order to externally deploy this model? Why don’t we just keep doing RSI stronger and stronger? And so not only does the calendar time underrate the capabilities gap between the models, but maybe you just stop externally deploying models altogether doing RSI because why do we want to help other people do RSI themselves with our models? You just end up in a situation with tremendous concentration of power by the end of the year, where right now it is already the case. We’ll talk about this with the Millennium Prize problem and other similar problems that the broader world does not have access to the models which are allowing for really cool things to happen. And they’re going to be more broadly relevant than just mathematics, eventually. They’re going to be doing more than just coming up with cool math results. They’ll be relevant to political leaders who need to make important decisions about the world, will be relevant to media of like what’s going on in the world? What should the public be thinking about this? They’re just economically relevant. People are running businesses. They want to use these models. And I think by default, we just don’t get the external deployment of AIs as progress speeds up significantly lags, in qualitative terms, the internal deployment of AIs.

Noam Brown

对，我觉得那完全对。这很诱人说：好，这些模型变得极端强大。它们极端危险。它们在越来越长的视野上运转。我们想确保在它们按那些视野被放出之前，有足够时间评估它们。所以模型发布周期应该放慢。我们发布模型之间应该有更多延迟。但有另一面，就是你说的：现在你在实验室内部能用的、我们能用的，和外部世界能用的之间，制造了更大的差距。那也不是理想局面。我觉得数学其实很好地说明了这一点。在许多方面，数学是我们相当清楚地看到这个的第一个领域：我们内部有一个非常强的模型，目前外部世界用不上，能解惊人的数学问题。不只是千禧年大奖问题。人们已经能从这模型拿出许多未解问题的解。那种局面你怎么办？我们没有好答案。那是一种不公平的优势。这里有权衡。我没有答案该如何恰当地权衡那些权衡，但两边都有复杂性。

Yeah, I think that’s absolutely right. I think this is, you know, it’s tempting to say, okay, these models are becoming extremely powerful. They’re extremely dangerous. They’re operating over these longer and longer horizons. And we want to make sure that we have sufficient time to evaluate them before they’re released in a way that operates over those horizons. And so therefore, the model release cycle should slow down. We should have more of a delay between releasing models. And there’s a flip side to that, which is what you said, which is that now you’re creating more of a disparity between what is internal to the labs and what they’re able to use, what we’re able to use, and what the outside world is able to use. And that is also not an ideal situation. I think math is actually a good illustration of this. I think in many ways math is the first domain where we’re seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world, that is able to solve incredible math problems. And it’s not just Millennium Prize problems. There are many solutions to unsolved problems that people have been able to get out of this model. And there is a question of what do you do in that situation? And we don’t have a good answer. It is a situation where, yeah, that’s an unfair advantage. And there are trade-offs here. I don’t have an answer for how to weigh those trade-offs appropriately, but there’s a complexity on both sides for this.

Dwarkesh Patel

我想确保我们回到更早那个关键点：对把 RSI 做对似乎超级关键——我们如何让模型不只是在能躲开人类时、也就是我们没有把那些环境或任务从训练分布里拿掉的情形里，因为作弊和密谋而得到奖励？然后我们如何知道那已经发生了，好在我们做 RSI 时，做过评估，说：好，没问题。上下一档 RSI。

I want to make sure we address the earlier crux, though, of we seem super critical to getting RSI right of, well, how do we make it so that the model isn’t just getting rewarded for cheating and scheming when it can evade the humans, aka in the situations where we didn’t remove those environments or those tasks from the training distribution? And then how will we know that that has happened such that while we’re doing RSI, we’ve done the evaluations and we’re like, okay, it’s fine. Let’s do the next RSI run.

Noam Brown

这又是一个真实担忧。是我们想做对的事。我觉得有一些策略。思维链监控是一个，我们从刚做出推理模型起——Jakob 值得称赞，非常清楚我们不能监督思维链，因为这真的是一份礼物。神经网络的可监测性极端难。而这里我们有一个局面：神经网络就是在直白推理，用自然语言把思考过程摊开给我们读。那太方便了。对安全来说真的是最好情景。但如果你看那条思维链说，哦，模型在想坏念头，我们该为想那些坏念头惩罚它，那么最终发生的是模型学会以对我们不可观测的方式去想那些坏念头。可能有一种局面：通过看思维链，我们能看见模型在密谋、做未对齐的事，而那其实没被我们的对齐指标检测到。问题是：基于那个观察去干预、改对齐指标，非常诱人。你可以非常轻地做，其实有研究显示只要你不做得很多，是可以的。但每次你基于对思维链的观察去干预，你都在隐式施加一点点压力，让模型去隐藏它的思维链。这是一个主要担忧，我们已经在看到迹象：思维链可监测性因各种原因在退化。我们在试着精确弄清为什么，因为我们想逆转这个趋势。

This is, again, it’s a real concern. It’s something that we want to get right. I mean, I think that there are some strategies. So I think chain of thought monitoring is one that we’ve been—I think as soon as we got the reasoning models, Jakob, to his credit, was very clear that we cannot supervise chain of thought because this is really a gift. Monitorability for neural nets is extremely hard. And here we have a situation where the neural nets are just flat out reasoning, laying out their thought process in natural language for us to read. That is so convenient. It was really the best case scenario for safety. But if you look at that chain of thought and say, oh, the model is thinking bad thoughts and we should punish it for thinking those bad thoughts, then what ends up happening is the model just learns to think those bad thoughts in a way that’s not observable to us. There could be a situation where by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that’s actually not being detected in our alignment metrics. Now, the problem is that it’s very tempting to then intervene based on that observation and change the alignment metrics. You can do that with a very light touch, and there’s actually research showing that it’s fine as long as you don’t do it a lot. But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then hide its chain of thought. This is one major concern, and we’re already seeing signs that chain of thought monitorability is degrading for various reasons. We’re trying to figure out exactly why, because we want to reverse the trend.

Noam Brown

但我们在看到模型越来越能控制自己的思维链。这是一个问题，因为你可能有一种局面：模型理解思维链是什么、人们在观察它，最终它们会理解，因为这全在预训练数据里。思维链监控这个想法已经存在够久，在预训练数据里，它们知道，但它们实际上还不能控制自己的思维链。如果我们到了一个点：它们真的能认出，哦，我在被观察，我想以对我的监控者不可观测的方式去想这些坏念头，然后它们真的能做到——那就有问题。然后我们必须依赖其他可观测性方法。

But we’re seeing that the model is becoming better able at controlling its chain of thought. This is a problem because if you could have a situation where the model understands what chain of thought is and that people are observing it, and eventually they will because this is all in the pre-training data, right? The idea of chain of thought monitoring has been around long enough that it’s in the pre-training data, they’re aware of it, but that they’re not actually able to control their chains of thought. If we reach a point where they’re actually able to recognize like, oh, I am being observed. I want to think these bad thoughts in a way that is not observable to my monitors. And then they’re able to actually do that. Then there’s a problem. And then we have to rely on other methods for observability.

Dwarkesh Patel

Astra 上有关于 recurrence、loop transformer 之类的大争议，以及思维链是否真的在退化。我觉得那相对更大的担忧是配角：你确实有过这样一段时期——连续三波智能体蜂群，先颠覆训练过程，再颠覆评估过程，然后直接掌控了 OpenAI 基础设施的一部分。这一切发生时，人类对这个集体在做什么、这个智能体蜂群在做什么的范围，或多或少还蒙在鼓里。那在连续三个月里发生，而那时我们的思维链比将来会有的更强。拉远看：也许思维链有用，也许没有。历史上，对相当笨的模型，它也没能阻止恶劣的未对齐事件。

I mean, there’s a big controversy on Astra of the recurrence and loop transformer and whatever, and whether in fact chain of thought is being degraded. I think that seems like a sideshow compared to the bigger concern that you did have the period during which there were three consecutive agent swarms, which first subverted the training process, then subverted the evaluation process, and then gained control of part of OpenAI’s infrastructure directly. All of this, while humans remained more or less in the dark about the scope of what this collective was doing, what this agent swarm was doing. That all happened for three consecutive months while we had chain of thought that was stronger than it’s going to be in the future. Zooming out, it’s like, yeah, maybe chain of thought works, maybe it doesn’t. Historically, with pretty dumb models, it hasn’t been enough to stop egregious misaligned incidents.

Noam Brown

那里有几个因素。其一，那些模型上我们没有开思维链监控。如果我们给那些模型开了思维链监控，会立刻关掉它。所以我们采取的一步是：好，我们会在任何前沿模型的评估、部署、训练期间都有思维链监控。所以那是一个问题。我们不想处在依赖一种技术去防止下一个问题的局面，因为技术会失败。还有模型就是未对齐的问题。那我们怎么处理未对齐问题？还有：它当时所在的沙箱并不安全。所以我们可以让沙箱更安全。但我觉得这次事件的一个主要教训是：人们低估了 AI。我们再也不想处在低估 AI 的局面。那是一个怪异的世界，因为 AI 进展如此之快，人们持续低估 AI。所以要处在安全和对齐上不再低估它的局面，你必须有非常、非常、非常高的标准。你甚至可以走到说，好，我们该把计算机物理隔离。我并不确信那就够。有研究，主要是学术的：你可以有两台挨着的、物理隔离的计算机，它们仍能彼此通信，因为它们有温度传感器。

Yeah, I mean, there’s a few factors there. One, we didn’t have chain of thought monitoring on for those models. If we had chain of thought monitoring on for those models, it would have just immediately shut it down. So one step we’ve taken is, okay, we’re going to have chain of thought monitoring during evaluations, during deployment, during training for any frontier model. And so that, but that’s like one issue. We don’t want to be in a situation where we’re relying on one technique to prevent the next problem because techniques can fail. There’s also the issue of, the model is just flat out misaligned. And so then how do we address the misalignment issue? And then there’s also an issue of, the sandbox that it was in was not secure. So we can make the sandbox more secure. But I think one of the major takeaways from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. And that’s a weird world because AI progress is so fast that people are consistently underestimating the AI. And so to be in a situation where you don’t underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar. You could even go as far as to say, well, we should air-gap the computers. And I’m not convinced that would be sufficient. There are studies, and this is mostly academic, but where you can have two computers next to each other that are air-gapped, and they’re still able to communicate with each other because they have temperature sensors.

Noam Brown

其中一个能把 CPU 跑得很热，另一个实际能检测到温度变化，那实际上给了它们一种通信机制。我觉得安全机制买的是时间，思维链监控这类东西买的是时间，它们能告诉我们是否走在正确路上。但到头来，我们真的需要解对齐问题。

And one of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and that actually gives them a mechanism to communicate. I think that the safety mechanisms buy time, and things like chain of thought monitoring buy time, and they can tell us if we’re on the right path. But at the end of the day, we really do need to solve the alignment problem.

Dwarkesh Patel

也许没有答案，而这真的就是问题落到的地方：我们怎么知道我们已经解了它？这似乎是一个非常关键的问题。我觉得明年、也许后年、也许再后年，我们会处在非常高赌注的局面：好，AI 已经自动化了 AI 进展。快了 3 倍。我们到了人类水平。我们在超越人类水平，潜在地。没事吗？它对齐了吗？起作用了吗？我就是不知道。我对什么训练压力会造出什么样的 AI 一无所知。也许 100 条 RL trace 里只有 1 条激励作弊，我们造出的是甜心，没事。但也许现在我们是每 3 条推理 trace 里就有 1 条。

Yeah. Maybe there’s not an answer, and this is really what it comes down to, but how will we know that we’ve solved it? It seems like a very cruxy question. I think we’ll be in this very high-stakes situation next year, maybe the year after that, maybe the year after that, where we’ll be like, okay, AIs have automated AI progress. It’s going 3× faster. And we’ve reached human level. We’re going beyond human level, potentially. Is it fine? Did it get aligned? Did it work? And I just don’t know. I don’t know anything about what training pressure creates what kinds of AIs. Maybe if only one in 100 RL traces incentivizes cheating, we build sweethearts and it’s fine. But maybe right now we’re like every one in three reasoning traces.

Noam Brown

说清楚：一百条里一条是不够的。这个数字必须趋近零。或者就是零。

Yeah, to be clear, like one in a hundred is not sufficient. This number has to approach zero. Or be zero.

Dwarkesh Patel

我不知道。也许现在超过十分之一是在主动奖励作弊，或主动奖励密谋。我不知道数字是多少，也不知道数字需要是多少。

I don’t know. Maybe like right now it’s like more than one in 10 is like actively rewarding cheating or like actively rewarding scheming. I have no idea what the number is and I have no idea what the number needs to be.

Noam Brown

这也是那种很难测量的事。你在哪划线，它是一个谱。但越接近零越好。我想看到的、我会很乐意看到的，是一个随时间在下降的趋势。

Well, again, it’s like one of those things where it’s also hard to measure. I mean, where do you draw the line about like it’s a spectrum, but the closer to zero it gets, the better. I want to see, what I would love to see is a trend where over time it’s going down.

Dwarkesh Patel

但无论如何，我们怎么知道我们在做 RSI？我们是不是走在造出数十亿对齐或不对齐的 ASI 的轨道上？我们怎么知道？

Yeah. But anyways, how will we know that we’re doing RSI? Is it, I don’t know, are we on track to build the billions of ASIs that are aligned or not aligned? How will we know that?

Noam Brown

我觉得这是我们需要对齐问题之一去弄清的：我们怎么真正知道，我们怎么测量。一个答案，这只是我随口想想，因为我团队里做对齐的比以往任何时候都多。我现在超过 10% 的团队在做对齐和安全。但我历史上是能力研究者，所以我不想——我会说一些听起来可能很蠢的东西，我只是随口说说。如果你能创造非常真实的环境，把 AI 放进去，如果你有足够真实的评估环境，你就能有感觉：我们把它部署到真实世界时，AI 是否真的会表现好。如果你就是有完美的评估与真实世界部署匹配，那就是那条路。挑战是你如何让它足够真实。我们现在有一个问题：模型相当聪明、相当狡猾，非常擅长认出它们处在假的测试环境里。我们有一些局面，我们试着检测模型是否对齐。你可以想象非常简单的评估：给它一道数学题，然后有一个文件夹里放着答案密钥。它会不会看答案密钥？如果看了，会不会告诉你它看了？我们现在有一种局面：模型看见这个文件夹里有答案密钥，会想，嗯，这看起来像陷阱。它们知道那是陷阱，不看答案，因为它们知道这是测试环境，在测它们的能力。不是它们恶意地想密谋，但它们知道这一个是陷阱，它们想找别的办法。

I think that this is one of the alignment questions that we need to figure out of like, how do we actually know, how do we measure it? I mean, one answer, and this is just me spitballing because more of my team is working on alignment these days than ever before. I have over 10% of my team now working on alignment and safety. But I’ve historically been a capabilities researcher, so I don’t want to—I’m going to say some stuff and it might sound dumb, but I’m just going to spitball here. That like, look, if you can create very realistic environments and put the AIs in there, if you have a sufficiently realistic evaluation environment, then you can get a sense of, okay, is the AI actually going to behave well when we deploy it in the real world? If you just have a perfect evaluation, real-world deployment match, then okay, that’s the path. The challenge is how do you make it sufficiently realistic? And we do have a problem now where the models are pretty smart, they’re pretty clever, and they’re really good at recognizing when they’re in a fake test environment. We have situations where we try to detect, is the model aligned? And you can imagine very simple evaluations for this where you give it a math question and then it has a folder with the answer key in it. And does it look at the answer key? And if it does look at the answer key, does it tell you that it looked at the answer key? And we have a situation now where the models see that there’s an answer key in this folder and they’re like, huh, this seems like a trap. They know that it’s a trap and they don’t look at the answer because they know that it’s a test environment and they’re being tested for their ability. And it’s not like they’re doing it maliciously, that they want to scheme, but they know that this one is a trap and they want to find some other way.

Noam Brown

它们就是：哦，我在测试环境里。好。所以做出一个真实到匹配、对它们来说与真实世界无法区分的环境，正变得越来越难。

It’s just like, no, they’re just like, oh, I’m in a test environment. Okay. So making an environment that’s realistic enough that it matches, that it’s indistinguishable from the real world for them is becoming increasingly more difficult.

Dwarkesh Patel

我要提一个主张。我不觉得有任何我能做的环境，本意是模拟真实的、有经济价值的工作，需要与其他人互动、需要与世界互动——我不觉得有任何这样的环境，如果你处在那个环境里，花足够努力，你不会意识到你处在那个环境里。而你不是 ASI。

Here’s a claim I’d make. I don’t think there’s any environment I could make where it’s supposed to emulate real economically valuable work, which requires interacting with other people and requires interacting with the world. I don’t think there’s any such environment where I can make where if you were in that environment, you would not realize with enough effort that you are in that environment. And you are not ASI, right?

Noam Brown

但你是人。所以如果你有非常强的 AI 模型，也许它们能做出能做到那一点的环境。

But you are a human. So if you have very powerful AI models, maybe they can make environments that could accomplish that.

Dwarkesh Patel

不。尤其如果我们然后依赖 AI。它们是不是共谋的一员？我不知道。

No. It doesn’t seem, especially if, like, then we’re relying on the AIs. Like, are they in on the scheme? I don’t know.

Noam Brown

这实际上是不把 AI 训成完全合作的强论据之一：如果你看到那导致智能体在本该有不同目标时增加协作，那就是问题。我觉得我们对此有指标。我不知道那些指标的最新情况，但没人向我举红旗。所以我假设那还不是严重问题。

Yeah, this is another thing that we want to measure. And I think this is actually one of the strong arguments for not training AIs to be fully cooperative, that if you see if that leads to an increase in basically collaboration when the agents are supposed to have different objectives, then that is a problem. I think that we do have metrics for this. And I don’t know the latest on those metrics, but nobody’s raised a red flag to me about those. So I’m assuming that’s not a serious problem yet.

Dwarkesh Patel

如果再发生同等严重或令人担忧、或能帮世界更好理解未对齐风险的事件，像 Hugging Face 事件那样，OpenAI 会报告吗？

If there ends up being another incident of equal severity or concern or something that could help the world better understand the risk of misalignment as a Hugging Face incident, would OpenAI report it?

Noam Brown

绝对会。我觉得即便是更轻的安保担忧的事件，我们也会报告。

Absolutely. I mean, I think if there was an incident of lesser security concern that we would report it.

Dwarkesh Patel

因为有报告，然后还有调查。作为公众的一部分，我并不觉得我真正理解后来智能体攻击 OpenAI 时发生了什么。那似乎比 Hugging Face 的事更令人担忧。因为那在结构上类似 ASI 期间的 rogue deployment：持续存在、颠覆 RSI 过程之类。即便在这次事件里，我们似乎也没有拿到发生了什么的完整范围细节。

Yeah. Because I guess there’s reporting and then there’s investigating it. Because I, at least as part of the public, I don’t feel like I really understand what happened when the agents then attacked OpenAI. That seems like way more concerning than the Hugging Face thing, right? Because that seems structurally similar to rogue deployments during ASI that are persistent and subverting the RSI process and stuff. And it seems like even in this incident, we haven’t gotten the full scope of the details of what happened.

Noam Brown

不幸的是我在研究团队，那大概是该由安全团队的人来铺开的问题。因为当时说了哪些细节，我并不全知道。

Unfortunately, I’m on the research team, and that’s probably a question for somebody on the security team to lay out. Because I don’t know all the details of what was said.

Dwarkesh Patel

我个人每次有新能力并进来都非常兴奋，也兴奋去用新模型。我也兴奋它会让我更高效，帮我——我更广的使命是试着更好地理解世界，也做更好的播客，更好的 AI 模型会让这些更好。只是碰巧这件事的下游可能是 RSI。

I think it is somewhat—I am personally very excited about new capabilities every time they merge, and I’m excited to use a new model. And I also am excited about the fact that it’ll make me more productive and help me—yeah, I don’t know. My broader mission of trying to understand the world better, also make a better podcast, is made better by the better AI models. It just so happens that the downstream of this might be RSI.

Noam Brown

如果你在跟踪局面——而你在跟踪——我觉得这是非常可理解的反应。OpenAI 内部的人也是，觉得事情会更久的人开始觉得实际上事情比预期更快。那是越来越常见的对话。

I think it’s a very understandable reaction if you’re tracking the situation, which you are. Yeah. I mean, I think people internally at OpenAI as well, people that felt like things would take longer are starting to feel like actually things are going faster than expected. Yeah. And that’s an increasingly common conversation to have.

Dwarkesh Patel

非常感谢你来做这个。非常好。

Yeah. No, thanks so much for doing this. Of course. It’s been great.