Mustafa Suleyman：对齐失灵？模型福利与行业减速

Nilay Patel

今天请来的是 Microsoft AI 的 CEO Mustafa Suleyman。各位大概都知道，眼下科技圈最大的新闻，就是围绕 AI 安全与监管急剧升温的那场辩论。

Today, I’m talking with Mustafa Suleyman, the CEO of Microsoft AI. As you’re no doubt aware, the biggest story in tech right now is the spiraling debate about AI safety and regulation.

Nilay Patel

Mustafa 对 AI 该怎么建、该怎么管，有很强的立场，这并不意外。微软刚发表一份 37 页的声明，名叫《人文主义 AI 行为准则》（Humanist AI Code of Conduct），写明公司对 AI 开发的原则，甚至包括 AI 意识这类非常棘手问题的哲学。

It should come as no surprise that Mustafa has strong opinions on how AI should be built and regulated. Microsoft just published a 37-page statement called the “Humanist AI Code of Conduct,” which lays out the company’s principles around AI development and even its philosophy around really thorny issues like AI consciousness.

Nilay Patel

如果还记得他上次上节目：Mustafa 认为 Anthropic 这类公司，在所谓「模型福利」上已经相当混乱，而且混乱得相当危险。他本周还发了一篇配套长文，专门批评 Anthropic 对 AI 意识的哲学，以及他认为这套东西如何嵌进更广的对齐辩论。

If you’ll recall from his last appearance on the show, Mustafa thinks companies like Anthropic have gotten really confused about this concept of so-called model welfare in fairly dangerous ways. He actually put out a companion essay this week specifically criticizing Anthropic’s philosophy around AI consciousness, and how he sees it fitting into the broader alignment debate.

Nilay Patel

所以我很想跟 Mustafa 谈：他认为 AI 安全里哪些是真的、哪些不是；对齐这个概念本身是否扛得住；以及这个行业是否需要在「把我们全杀掉」之前先减速。还有：AI 行业为什么不——直接把这些事都做了？我一直喜欢跟 Mustafa 钻进细节，他这次也很愿意跟我较真。

So I really wanted to talk to Mustafa about what he thinks is real and not in AI safety, whether the concept of alignment itself is up to the task, and whether this industry needs to slow down before it kills us all. Also: Why isn’t the AI industry just… doing all of this already? I’ve always enjoyed getting into the weeds with Mustafa, and he was very game to get into it with me here.

Nilay Patel

好。Microsoft AI 的 CEO Mustafa Suleyman，谈 AI 监管的未来。开始。

Okay. Mustafa Suleyman, the CEO of Microsoft AI, on the future of AI regulation. Here we go.

Nilay Patel

Mustafa Suleyman，你是 Microsoft AI 的 CEO。欢迎回到 Decoder。

Mustafa Suleyman, you’re the CEO of Microsoft AI. Welcome back to Decoder.

Mustafa Suleyman

很高兴见到你，Nilay。谢谢再邀请我。

Great to see you, Nilay. Thanks for having me back.

Nilay Patel

见到你也很好。我很想跟你聊聊：AI 安全与监管这场辩论到底在搞什么。你刚发表一份很长、很细的文件，写明你的原则、微软的原则，你们称之为「人文主义 AI」。

It is great to see you. I’m very excited to talk to you about what on earth is going on in the AI safety and regulation debate. You just published a very long, very detailed document laying out your principles, Microsoft’s principles, around what you’re calling “Humanist AI.”

Nilay Patel

里面有很多想法我想拆开。我越想这场对话，越想从一个非常基础的问题开始。这是我隐约看到、但可能是这一切根子的东西。

There’s a lot of ideas in there I want to unpack. The more I have been thinking about this conversation, the more I want to start with a really foundational question. It’s something that I had lightly been seeing, but might be the root of all of this.

Nilay Patel

我们谈 AI 安全的基本方式，叫对齐——我们要用某种机制，让模型从内在就做对的事。对齐、错位、Hugging Face 攻击、模型身上发生了什么，已经谈了很多。但对齐是不是已经坏了？它有没有可能成功？还是根本就是错的路径？

The basic way that we have been talking about AI safety is something called alignment — we’re going to make the models do the right thing intrinsically in some way. There’s some mechanism for doing it. There’s been a lot of talk about alignment and misalignment and Hugging Face attacks and what happened with the models. But is alignment broken? Is it possible for it to be successful? Is it just the wrong approach?

Mustafa Suleyman

对，我认为它是一个重要要素，但不是唯一要素。我三四年前在书里写过「遏制」（containment）这个想法。开篇其实就是：遏制做不到，扩散不可避免。在 99% 的情形里，这是件非常好的事。我们希望技术尽快铺得又远又广，让每个人都能享受好处。

Yeah. I mean, I think it’s one important element, but it’s not the only one. I wrote about the idea of containment three or four years ago in my book. And actually the opening chapter is about the idea that containment is not possible, that proliferation is inevitable. In 99 percent of cases, that’s a really good thing. We want technologies to spread far and wide as quickly as possible so that everyone can enjoy the benefits.

Mustafa Suleyman

与此同时，如果你把时间往前拨五年——我们总是卡在下个季度或明年，大家就慌，然后大吵一架。但你只要想象 GPT-3 三年前和今天的 GPT-6 差多少，再想象 GPT-6 到 GPT-9 差多少。那是三个数量级的算力，预训练加强化学习这些 run 上多一千倍的 FLOPS，我们会得到一样令人屏息的东西。它会在非常多事情上绝对惊人。

I think at the same time, if you just roll forward five years, we always get caught up in the next quarter or next year and everyone gets a little bit flustered and has a big disagreement. But if you just imagine the difference between GPT-3 three years ago and GPT-6 today, and then imagine the difference between GPT-6 and GPT-9. That is three orders of magnitude more compute, 1,000 times more FLOPS applied to pre-training with reinforcement learning for these runs, and we’re going to have something which is breathtaking. It’s going to be absolutely incredible at so many things.

Mustafa Suleyman

我不认为这是炒作。我认为这只是基于过去五年进展的、非常明显的经验陈述。如果还会继续，那问题就会变成遏制与对齐。我们当然希望把这些东西对齐到我们的价值观，但第一件事是必须确保它们被遏制：自主性受限，逃不出盒子，不做奖励黑客，可控，并且服从我们的指令。

I don’t think that is a hype. I think it’s just a very obvious empirical statement based on the progress that has been made over the last five years. If that’s going to continue, then the question really is going to become about containment and alignment. Of course, we want to align these things to our values, but the first thing is that we have to make sure they’re contained, their agency is limited, they don’t escape the box, they don’t reward hack, that they are controllable, and they follow our instruction.

Mustafa Suleyman

然后我们要确保它们对齐人类的目标。这就是我们本周发布的《人文主义 AI 行为准则》的目的。微软的立场非常简单。技术是来服务人类的。它应当是一种从属的、可控的、对齐的力量，在世上做好事。做不到，我们就该拒绝它。在我看来，我们离那一步还很远。今天还没发生；但鉴于今夏 Hugging Face 与 OpenAI 发生的事，已经相当清楚：没有安全护栏的这些系统，具备非常惊人、也相当吓人的黑客能力。

We then want to make sure that they are aligned to our objectives as humans. That’s the purpose of the Humanist AI Code of Conduct that we released this week. Microsoft’s position is very simple. Technology is here to serve humanity. It should be a subordinate, controllable, aligned force that does good in the world. If it doesn’t achieve that, then we should reject it. It seems to me that we are far from that point. It has not happened today, but it is now, I think given what’s happened over the summer with Hugging Face and OpenAI, pretty clear that these systems without the safety guardrails are capable of really impressive and quite scary hacking capabilities.

Nilay Patel

我想把这拉到尽可能落地的比喻，因为这是我的主问题。如果我设计一辆车，有 10% 的时间刹车踏板决定去攻击邻居的房子，我会说：「这车不行。刹车这门技术本身坏了。我需要一个新想法。」

I want to drag this down into as grounded of a metaphor as I can, because this is the main question I think I have. If I designed a car and 10 percent of the time the brake pedal decided to go attack my neighbor’s house, I would be like, “This car doesn’t work. The very technology of brakes is broken. I need a new idea.”

Nilay Patel

我对齐问的就是这个问题。让模型变安全的那条路，感觉已经搁浅。如果是这样，我对整场辩论的理解是一种；如果对齐及其技术仍可能成功、有用、稳定，我对辩论的理解就是另一种。所以你认为对齐有没有可能做到 100% 安全？

I think I’m asking that question about alignment. It feels like that approach to making the model safe has run aground. If that is the case, then I think I understand this entire debate one way. If it’s possible for alignment and the techniques of alignment to be successful or useful or consistent, then maybe I understand the debate in a different way. So do you think alignment has potential to be 100 percent safe?

Mustafa Suleyman

我们把多头和空头两边都说一遍。回看过去三年，在我看来驱动进展的主要变化，是模型变得更可操控。它们会服从指令，你也可以给它们设越来越复杂的目标，要求它们在多个时间步上准确行动，并用上各种工具。

I mean, look, let’s make the bull case and the bear case. If you look back over the last three years, the main change, in my opinion, that has driven progress is that the models have become more steerable. They follow instructions and you can set more and more complex goals for them that require them to act accurately over multiple time steps using all sorts of tools.

Mustafa Suleyman

这就是证据：过去三四年我们对齐得更多，而不是更少。我们不再那么谈幻觉、偏见，以及上一代那些小毛病。

That is evidence that we have got more alignment over the last three or four years, not less. We don’t so much talk about hallucinations or bias or all of these other niggles that we had in the previous generations.

Mustafa Suleyman

另一面，Hugging Face 事件是分水岭。成群的智能体彼此串通。它们自组织成层级。它们搞出分工：有的专攻对抗性黑客，有的做研究，有的做协调。某些智能体 token 快用完时，它们甚至会自我牺牲。

On the flip side, what we saw in the Hugging Face incident was a watershed moment. Swarms of agents colluded with one another. They self-organized into hierarchies. They created a division of labor so that some were focused on adversarial hacking, some were doing research, some were doing coordination. They even self-sacrificed when certain agents were running out of tokens.

Mustafa Suleyman

它们试图掩盖踪迹，通过通信来隐藏或编辑思维链、或它们互动的日志。某种意义上，它们没有道德准则。公平地说，对 OpenAI 而言那是设计：他们就是在造对抗性网络能力。结果向全世界展示了：它可以达到人类水平表现，发现零日漏洞，并把据点维持许多天，甚至数周。

They tried to cover up their tracks and communicate to hide or edit the chain of thought or the logs of their interactions. In some sense, they had no moral code. To be fair to OpenAI, that was their design. They were trying to create adversarial cyber capabilities. As a result, they showed to everybody in the world that it can achieve human-level performance, discover zero-day vulnerabilities, and hold positions for many, many days, if not weeks.

Mustafa Suleyman

所以这告诉我们的，本身并不是我们有一个对齐问题。其实是模型极其擅长执行指令，但你必须非常非常小心给它什么指令，并且必须非常小心地遏制它。这些黑客行为都不是「被设计成逃到互联网上」的——那完全不是 OpenAI 的意图——但围绕此事的遏制过程，我认为每个人除了对齐之外也必须聚焦。

So what that tells us is not that we have an alignment problem per se. It’s actually that the models are incredibly good at following instructions, but you have to be very, very careful what instructions you give it and you have to contain it very carefully. So none of these hacking behaviors were intended in the sense that they found a way out to the internet, which was not the intention of OpenAI at all, but the containment process around that is what everybody, I think, also has to focus on in addition to alignment.

Nilay Patel

让我放进你的框架：AI 能力上的大进展，是关于控制，是编程的 harness，以及你看到的智能体应用。现在我们需要再加一层遏制，施加更多控制——在对齐（你如何训练模型以某些方式行事）之外，还要说：你确实可以做你想做的这件事。

So let me put that into your framework, that the big advances in capabilities of AI have been about control, the harnesses for coding and the agentic applications you’re seeing. Now, we need to add a layer of containment that exerts even more control, that says you can actually do this thing you’re trying to do in addition to alignment, which is how you would train the model to behave in certain ways.

Mustafa Suleyman

对，你基本上两样都得有，但有非常具体的事可以做。比如，我们不能允许模型用向量对向量、矩阵对矩阵通信。它们不能用 neuralese 通信。我们必须强迫它们用人类语言通信。即便如此也会极度过载，因为量会非常大。

Yeah. I mean, you basically have to have both, but there are very specific things that we can do to address it. So for example, we can’t allow models to communicate vector to vector, matrices to matrices. They can’t communicate in neuralese. We have to force them to communicate in human language. Even that will be massively overwhelming because there’ll be so much of it.

Mustafa Suleyman

但那是审计者或评估者真正能核验的东西，也确实会提高安全概率。所以有很多可落地的步骤可以聚焦，而不是抽象地说：现在该监管了，或现在该减速了。

But that’s something that an auditor or an evaluator can actually verify and it’s something that definitely increases the chances of safety. So there’s a lot of practical steps that we can get focused on rather than just abstractly saying that it’s the time for regulation or it’s the time for a slowdown.

Nilay Patel

这写在你们的《人文主义 AI 行为准则》里：不该有 neuralese——人看不懂，就无法监督。不只是它们用本质上是数学的 neuralese 通信，还有一些模型在用的不透明代号。我认为 OpenAI 允许模型基本上用代号通信，这样能更快。

This is in your Humanist AI Code of Conduct that there should be no neuralese — if humans can’t understand it, they can’t oversee it. It’s not just neuralese where they communicate in essentially mathematics, but it’s also these opaque code words that some of the models are using. I think OpenAI allows its models to communicate essentially in code words so they can go faster.

Nilay Patel

对我来说这就是那种：微软可以说……我知道你对此意见很强，但要让所有实验室都同意，这是监管职能。我不确定你怎么让所有人同意，或让开源权重模型也同意，除非你说：不参加这类监管方案就要受罚。你要怎么把这套加到其他所有人头上？

This to me is one of those things where Microsoft can say… I know you have very strong opinions about this, but getting all of the labs to agree to this is a regulatory function. I’m not sure how you would get everyone to agree to this or get the open weight models to agree to this, unless you say there’s some penalty for not participating in a regulatory scheme like this. How would you impose this on everyone else?

Mustafa Suleyman

我觉得我对「把东西加到其他所有人头上」会比较谨慎。我认为当下好的地方，是有一场以自由为核心的公开公共辩论。那不是其他国家的样子，当然也不是我来自的地方、或我家人来自的地方的样子。

I think that I’m a bit careful about imposing things on everybody else. I think that what’s good about the current moment is that there is an open public debate with freedom at the core. That isn’t what it’s like in other countries, certainly places that I’m from, or my family’s from.

Mustafa Suleyman

我认为我们应该先喘一口气，感激我们能对真正重要的事有大规模公开分歧。那是程序按设计在运转，而且并不清楚该做什么。

I think that we should just take a breath to be grateful for the fact that we can have a massive public disagreement about really important things. That’s the process working as intended and it isn’t clear what to do.

Mustafa Suleyman

我不认为任何斩钉截铁的说法是对的：「我们现在必须绝对停下来」，或「我们只能加速」，或「只能靠监管」，或「只能靠行业自律」。这些都不成立。需要大量细微差别和耐心，去把细节想清楚。与此同时，我们迫切需要行业标准。有些东西我认为必须拿下桌。

I don’t think anyone who’s categorical about “we absolutely have to stop now” or “we can only accelerate” or “we can only do this with regulation” or “it can only happen with industry self-regulation.” None of these things are true. It requires a lot of nuance and patience to really think through the details. At the same time, we urgently do need industry standards. Some things I think need to be taken off the table.

Mustafa Suleyman

用 neuralese 通信是其中一件。缺乏遏制是另一件。你做的训练 run 规模可以用 FLOPS 衡量。模型超过某个 FLOPS 门槛时，我们已经有向安全研究所报告的要求。我们可以扩展它，可以做得更细，可以聚焦某些类型的能力。相当清楚的是：这些大事里有一些必须有独立第三方核验。

Communication in neuralese is one of them. A lack of containment is another. The scale of the training run that you do can be measured in FLOPS. We already have a reporting requirement to the safety institutes when models exceed a certain FLOPS threshold. We can extend that, we can make that more nuanced, it can be focused on certain types of capabilities. It’s pretty clear there has to be independent third-party verification of some of these big things.

Mustafa Suleyman

坦白说，过去几周几个月跟一堆实验室负责人聊过，大家基本上在同一页上。细节需要敲定。所以并不是对「怎么做」或「精确做什么」已经有共识，但总体来说，我认为面对正在被提出的关切，我们应该少一点危言耸听和犬儒，多一点「我们正往正确方向走」。

Frankly, having spoken with a bunch of the lab leaders over the last few weeks and months, everybody’s basically on the same page. The details need to be worked out. So it’s not like there’s consensus on how or precisely what, but overall I think that we should be less alarmist and cynical and more like we’re headed in the right direction with respect to the concerns that are being raised here.

Nilay Patel

我从对齐起头的原因是：如果你告诉我对齐行不通、我们需要一种新的技术路径，我会处在「那就急刹车，停掉所有开发，直到你找出能用的安全机制」。你说的是：对齐已经在我们看到的进展过程中被证明有效。再加上控制与遏制，也许能到你需要的地方。

The reason I started with alignment is if you told me alignment doesn’t work and we need a new technological approach, I think I would be at, “well, slam the brakes and stop all development until you figure out a safety mechanism that works.” You’re saying alignment has been demonstrated to work over the course of progress that we’ve seen. With the addition of control and containment, maybe you can get to where you need.

Nilay Patel

这个行业现在需要的，是关于如何建造这些模型、如何强制能力上限的一些标准。你显然在行业里，也认识所有这些人。这周之前那场对话的语气是什么样？为什么这周突然这么吵？

What this industry needs now is some standards about how to build these models and enforce the limits on their capability. You’re obviously in the industry and you know all these folks. What has the tenor of that conversation been like before this week and why has it gotten so loud this week?

Mustafa Suleyman

我认为至少对行业而言，转折点更像是 Hugging Face 事件，在那之前也有几起。那是我觉得所有人开始彼此聊得更多的时刻，因为那真的相当令人屏息。显然，由于上周所有人一齐发声，这现在已经变成重大的国内和国际议题。

Well, I think the turning point at least for the industry was more like the Hugging Face incident and there were a few incidents before that. That was the moment when I think everybody started to talk to each other a lot more because it is really quite breathtaking. Obviously, this has now become a major national and international issue because of the last week with everybody weighing in.

Mustafa Suleyman

但重要的是也要说：关于集体协调，以及自主性或递归自我改进（RSI）这类更危险的能力，我们已经谈了六、七、八年。2017、2018、2019 年我们聚过好几次。

But I also think it’s important to say that we have been talking about collective coordination and capabilities that are more dangerous like autonomy or recursive self-improvement, or RSI. We’ve been talking about those things for six, seven, eight years. We’ve got together a bunch of times back in 2017, 2018, and 2019.

Mustafa Suleyman

COVID 期间我们跟一堆实验室负责人有定期会议，谈的就是这类能力，以及此刻会需要的监管机制。所以虽然这是一个门槛时刻，但对一直参与的人来说也并非全新。

We had regular meetings during COVID with a bunch of the lab leaders where we were talking about these kinds of capabilities and the kinds of regulatory mechanisms that would be required at this moment. So whilst it is a threshold moment, it’s also not completely new to everybody who’s been involved.

Nilay Patel

是什么促使你这周发出关于模型福利的文章？是什么促使微软 CEO Satya Nadella 在 X 上发声明，说他大体同意「给前沿定节奏」的呼吁，并欢迎「嵌入式评估者」？是什么促使你们这周加入这场对减速、或监管、或接下来无论什么东西的呼吁？

What prompted you this week to put out your essay on model welfare? What prompted Microsoft CEO Satya Nadella to put out a statement on X saying he mostly agreed with the calls to pace the frontier and he welcomed “embedded evaluators”? What prompted you all this week to participate in this call for a slowdown or regulation or whatever comes next?

Mustafa Suleyman

我们写《人文主义 AI 行为准则》差不多写了这一年的大部分时间。我们的超级智能努力才开始 11 个月。一开始我们就在想：好，治理文件是什么，那套政策如何塑造我们想造的那种 AI？我们一直在跟大量外部利益相关方咨询：学者、律师、哲学家、公众、焦点小组之类。所以拼起来花了些时间。

We’ve been writing our Humanist AI Code of Conduct for the best part of this year. We only started our superintelligence efforts 11 months ago. As soon as we did, we started figuring out, “Okay, what is the governing document, the set of policies, that shape the kinds of AI that we want to build?” We’ve been doing that in consultation with a ton of external stakeholders, academics, lawyers, philosophers, members of the public, focus groups and stuff. So it’s taken us a while to put it together.

Mustafa Suleyman

我们其实计划下周或再下一周发布。但鉴于正在发生的一切，我们想：好，现在就是发出去、拿反馈的时候。我们把它作为公开征询发布。所以我们基本上会开放六周，收集大量关于如何改进的反馈。但我认为每个人现在都意识到：如果还没有，他们就必须拿出驱动行为的宪法或行为准则。

We were actually planning to release it next week or the week after next week, I think it was. But then given everything that was happening, we thought, “Okay, now is the time to put it out and get feedback.” We’ve released it as a public consultation. So we’re basically going to keep it open for six weeks and we’re collecting lots and lots of feedback on how we can improve it. But I think everybody is now realizing that if they haven’t already, they have to put out constitutions or codes of conduct that drive behavior.

Nilay Patel

这里一个有意思的动态是：我知道你觉得模型福利这个概念很蠢。你上次上节目时说，Anthropic 已经把自己「wirehead」到相信 Claude 有意识，那很荒谬。你们新的行为准则里写着：模型没有意识，我们不该那样对待它们。

One of the interesting dynamics here is that I know you find the concept of model welfare to be silly. The last time you were on the show, you said Anthropic had wireheaded themselves into believing Claude was conscious and that was ridiculous. It’s in your new code of conduct that the models are not conscious and we shouldn’t treat them as such.

Nilay Patel

必须写宪法、必须写这类文件，在某种意义上，它们是写给模型自己的。这会成为模型训练的一部分。你怎么想那个受众？只是写给团队，还是你为模型写的？

Having to write constitutions, having to write documents like this, in some way, they are for the models themselves. This will be part of the model’s training. How do you think about that audience? Is it just for your team or have you written this for the model?

Mustafa Suleyman

这当然是为模型写的，但思考方式是：它是主要治理文件，好让公众理解我们训练模型时的意图。我们用这份治理文件来创建安全护栏、生成训练数据，并总体上评估模型在真实世界的表现。所以你可以把它想成一种问责功能。

This is certainly written for the model, but the way to think about it is that it’s the primary governing document so the public understands what our intentions are when we are training models. It’s that governing document that we use to create safety guardrails, generate training data, and generally evaluate the performance of our model in the real world. So you can think of it as an accountability function.

Mustafa Suleyman

我们不会把《人文主义 AI 行为准则》原文原样作为训练文档提供给模型。我们用它派生随后塑造模型的全部训练数据。所以就一切实际目的而言，那是我们组织、文化、团队、以及我们在微软更广泛所做一切的北极星。我认为越来越多人都会把它们发出来。我觉得其他团队也已经发过类似文件。

We don’t provide that Humanist AI Code of Conduct raw as a training document to the model. We use it to derive all of the training data that then shapes the model. So for all practical purposes, that’s our north star for our organization, our culture, our team, everything that we’re doing at Microsoft more generally. I think increasingly, everybody is going to put them out. I think other teams have also put out similar documents.

Nilay Patel

我觉得这就是辩论的核心。如果你能做到这个，并且你认为行业其余部分也会做，为什么所有前沿实验室不能直接减速？为什么他们不能停下那件可能把我们全杀掉的事？为什么还要推监管框架？

I think this is the heart of the debate. If you can do this and you think the rest of the industry is going to do this, why can’t all the frontier labs just slow down? Why can’t they stop doing the thing that might kill us all? Why this push for a regulatory framework?

Mustafa Suleyman

我认为行业里每个人现在都在说：现在是减速的时候，是在那个问题上协调、并把它做落地的时候。显然有一种担心：会被指控成反垄断卡特尔。我认为人们对此应该非常怀疑。

Well, I think that everyone in the industry is saying that now is the time to slow down and to coordinate on that question and to make it practical. I mean, obviously there’s some concern that there’s an antitrust cartel accusation. I think people should be very skeptical about that.

Mustafa Suleyman

我认为提出棘手问题很重要，因为我们谁也不会想借这种事去集中权力。所以怀疑和批评就是重要的。我们其实没有一个好机制，能让大家聚在一起说：「各位，我们大概都该减速。」

I think that it’s important that the tough questions get asked because there’s no way any of us would want to try and concentrate power from something like this. So it’s just important to be skeptical and critical. We don’t really have a good mechanism for us all getting together and saying, “Guys, we should probably all slow down.”

Mustafa Suleyman

想象一堆银行聚在一起说：「各位，我们担心如果交易这种资产会有系统性风险，所以我们都要单方面停止交易这种资产，而且没有任何公共审视或政府参与。」这看起来就挺可疑，对吧？所以我认为，这不只是行业自律的事，是合理的。问题是我们如何就此事与政府接洽。

I mean, imagine if a bunch of banks all got together and said, “Guys, we worry that there’s a systemic risk if you trade this kind of asset, so we’re all just going to unilaterally stop trading this kind of asset without any public scrutiny or government involvement.” I mean, it seems pretty dodgy, right? So I think that it’s reasonable that this isn’t just an industry self-regulation thing. It’s a question of how we engage with the government on it.

Nilay Patel

这里一个迷人的动态是：也许是美国历史上第一次，美国政府看了一项提供监管的请求，然后实际上说了不。唐纳德·特朗普称所有这些恐惧是骗局。众议院议长 Mike Johnson 说他觉得这不需要发生。JD Vance 说他认为这是特洛伊木马。他们实际上拒绝了参与监管努力的呼吁。行业对此的反应是什么样？

A fascinating dynamic here is that maybe for the first time in American history, the United States government has looked at a request to provide regulation and effectively said no. Donald Trump has called all of these fears a hoax. House Speaker Mike Johnson has said he doesn’t think this needs to happen. JD Vance said he thinks this is a Trojan horse. They’ve effectively rejected the call to participate in a regulatory effort. What has the response from the industry been like to that?

Mustafa Suleyman

我认为所有人都在挠头、在想清楚，会需要一点时间才能找出正确机制。当然，就连 Elon Musk 也非常直接地站在后面。Mark Zuckerberg 也是。每个人都在搞明白：完全不受约束，对接下来几年大概说不通。我认为我们会需要一点时间找出正确机制。

I think everyone’s just scratching their head and figuring it out and it’s going to just take a little bit of time to figure out what the right mechanism is. I mean, certainly, Elon Musk even is very directly behind it. Mark Zuckerberg is too. Everybody is figuring out that completely unchained probably doesn’t make sense for the next few years. I think it’s going to take us a little bit of time to figure out what the right mechanism is.

Mustafa Suleyman

我提出了几个非常落地的提案：围绕可核验的遏制、围绕自我改进、围绕 FLOPS 门槛、围绕不用 neuralese 通信。所以与其停在太抽象的层面，我们可以就这些能推进的具体事项聚焦。我肯定还有一堆别的。

I put forward a couple of very practical proposals around verifiable containment, around self-improvement, around FLOPS thresholds, around not communicating in neuralese. So rather than keeping it too abstract, we can just focus on those specific things that we can make progress on. I’m sure there’s a bunch of others too.

Nilay Patel

《The Information》有报道说，已经有过关于行业自律机构的谈话。你参与那些谈话了吗？

There’s reporting in The Information that there have already been talks about an industry self-regulatory body. Have you been involved in those talks?

Mustafa Suleyman

参与了。就像我说的，我们 COVID 期间谈了很多。2010 年代末也谈过。过去几周几个月，实验室负责人之间肯定有过很多对话。

Yeah. I mean, as I said, we talked a lot during COVID. We talked in the late 2010s about it. I mean, there’s definitely been a lot of conversations over the last few weeks and months between all the lab leaders.

Nilay Patel

我理解为什么 Anthropic 和 OpenAI 可能某天醒来会说：「等等，我们是不是在犯反垄断？我们如果协调会不会被起诉？」微软跟政府非常非常熟，对吧？你们是长期政府承包商。微软总裁 Brad Smith 非常擅长政策。

I understand why Anthropic and OpenAI might wake up one day and say, “Wait, are we committing an antitrust violation? Are we going to get sued if we coordinate?” Microsoft is really, really good at the government, right? You’re a longstanding government contractor. Brad Smith, the president of Microsoft, is very good at policy.

Nilay Patel

Lina Khan，也许是我们这辈子见过最强硬的反垄断执法者，公开说：「你们不需要这个反垄断豁免。」我刚跟在拜登司法部管反垄断的 Jonathan Kanter 聊过，那是即将播出的一集。他说：「你们不需要反垄断豁免。」

Lina Khan, who is maybe the most aggressive antitrust enforcer we’ve had in our lifetimes, is publicly out there saying, “You don’t need this antitrust exemption.” I just talked to Jonathan Kanter, who ran antitrust at the Biden Department of Justice, for an upcoming episode of the show. He said, “You don’t need an antitrust exemption.”

Nilay Patel

在微软内部，你觉得你们需要反垄断豁免吗？

Inside Microsoft, do you think you need an antitrust exemption?

Mustafa Suleyman

这要律师来答。我认为人们眼下正在查，也非常认真地对待。所以他们得把「需要还是不需要」过一遍。听着，对这些事小心是对的。我不会把每一件事都读成犬儒，但我们走着看。我们必须很快推进。不能只是拖着，把它当挡箭牌。

I mean, that’s one for the lawyers to answer. I think that people are looking into it at the moment and they’re taking it very seriously. So they’re just going to have to work through whether we do or whether we don’t. Look, it’s right to be careful about those things. I wouldn’t read every single thing as cynical, but we’ll see. We have to make progress quickly on it. We can’t just dither around and use that as a blocker.

Nilay Patel

这场辩论的另一个版本，或者进入这场辩论的另一条路是：你不需要减速，也不需要被新奇监管把安全责任压到你头上。单靠产品责任，就会给你造出更安全产品的激励。

The other version of this debate or maybe the other avenue into this debate is you don’t need to slow down and have safety responsibility imposed on you by novel regulation. Product liability alone will create the incentives for you to make more safe products.

Nilay Patel

如果一个微软 AI 模型出去对世界造成某种无法估量的伤害，微软会被告到不存在，而这大概是你们发布下一个模型之前就该想的事。这对你们已经是有效的激励闭环了，还是你现在才在想？

If a Microsoft AI model goes out and does some untold harm to the world, Microsoft will get sued out of existence, and this is probably something that you should think about before you release the next model. Has that been an effective incentive loop for you already or is that something you’re thinking about now?

Mustafa Suleyman

当然。我们部署产品时想的每件事里，这始终在场。但请记住，这不太是部署产品的事。用于 Hugging Face 黑客、或用于解决数学千禧年奖 Navier–Stokes 的那些模型，还没有商业发布。它们还不是真正的产品。

Definitely. I mean, of course that’s always present in everything that we think about when we deploy products, but keep in mind, this isn’t so much about deploying products. The models that were used for the Hugging Face hack or to solve the Navier-Stokes Millennium Prize in mathematics, they’re not commercially released yet. They’re not actual products.

Mustafa Suleyman

所以责任制度略有不同。这些是在大公司内部、用超长强化学习爬坡来运行的。我认为责任覆盖一部分，但不是全部。

So the liability regime is slightly different. I mean, these are being operated inside of the big companies with huge long-running reinforcement learning climbs. So I think liability covers part of it, but not all of it.

Nilay Patel

有一部分我个人觉得，问产品责任问题时自己都有点蠢。微软要发布一个可能杀掉所有人的新版 Microsoft Word。也许你不该那么做，因为它会被告到不存在。这是很好理解的简单事。它蠢到在任何其他技术的任何其他对话里都不会出现。蓝牙很好，但如果它杀掉所有人呢？我们根本不会有蓝牙。

There’s a part of me that personally feels a little silly when I ask questions about product liability. Microsoft is going to release a new version of Microsoft Word that might kill everyone. Maybe you shouldn’t do that because it’ll get sued out of existence. It’s a pretty simple thing to understand. It’s so silly that it would never occur in any other conversation about any other technology. Bluetooth is great, but what if it kills everyone? We just wouldn’t have Bluetooth.

Nilay Patel

会叫停 AI 开发的近期灾难后果是什么？仅仅是产品责任，还是别的？

What are the near term disaster consequences that would stop AI development? Is it just product liability or is it something else?

Mustafa Suleyman

我只是觉得每个人都带着这种夸张、过度反应、完全危言耸听的语气，而事实上我们有几十年监管奏效得令人难以置信的长历史，奏效到你几乎注意不到。从路灯到建材，从石棉到笔记本电脑里不会让车起火的电池，到安全带，每一样都有行为准则，都有监管框架。每项新技术都带着那套思路去建，好让飞机不在天上相撞。

I just feel like everyone has this hyperbolic, super reactive, completely alarmist tone when in fact, we have a long history of many decades of regulation that has worked incredibly well, so well that you barely notice it. Everything from street lights to construction materials from asbestos to the batteries inside of your laptop that don’t cause a fire inside of your car to the seat belts, everything has a code of conduct and it has a regulatory framework around it. Every new technology gets built with that in mind so that planes don’t hit each other in the sky.

Mustafa Suleyman

这项技术确实不同。我不会只把它丢进铅笔和油漆那个桶。它是不同的。它也比以往任何时候都走得快得多。当你往上面倒海量算力时，涌现出的能力极其像人。所以重要的是看清楚：这是一个不同的时刻，这一次确实不同。与此同时，有一整套实践、知识和框架可以套用到这里。责任是显而易见的一项。

It is true that this technology is different. I’m not just going to put it in the bucket of pencils and paint. It is different. It is also moving much faster than it ever has before. It’s incredibly human-like in the emergent capabilities that arise when you pour a ton of compute on it. So it’s important to be clear-eyed that it is a different moment, and this time actually is different. At the same time, there’s an entire body of practice and knowledge and frameworks and so on which can be applied here. Liability is an obvious one.

Mustafa Suleyman

所以是的，挺棘手，因为很多对话往往发生在 Twitter 上，所以温度看起来全都非常高，但我不知道我们还能在别处谈。

So yeah, it’s tricky because a lot of the conversation tends to take place on Twitter, so the temperature seems to all be really high, but I don’t know where else we have it.

Nilay Patel

看起来这场对话大概本该发生在国会大厅和各种监管机构里，可我们却选了 Elon Musk 的短内容社交媒体平台；思想在那里被压缩时，有些东西正在字面意义上丢失。

It does seem that probably we should be having this conversation in the halls of Congress and at various regulatory bodies, and instead, we’ve chosen Elon Musk’s shortform social media platform and something is getting lost literally in the compression of thought that occurs there.

Nilay Patel

你觉得这场对话里正在丢失的最重要的东西是什么？大多数人没看到的细微差别是什么？

What do you think is the most important thing that is being lost in this conversation? What’s the nuance that most people aren’t seeing?

Mustafa Suleyman

细致、可落地的提案。读别人的文件需要时间，坐下来读。很多东西正在被写下来，它们精确、具体，装满朝某个方向走的具体提案。我们并不缺有实质的想法。问题是我们在用极度攻击性的短内容，去传达有实质的想法。

Detailed practical proposals. It takes time to read somebody’s document, sit down and read it. A lot of things are getting written down and they are precise and specific and they’re full of concrete proposals to go in one direction or another. It’s not like we’re lacking for substantive ideas. The problem is we’re communicating substantive ideas in hyper-aggressive short form.

Mustafa Suleyman

我试着发出过一堆非常详尽的提案。我们的《人文主义 AI 行为准则》是一份 40 页的文件。我今早关于模型福利的文章也大约是 20 页的长文，非常细致地、逐字对照 Anthropic 那份 99 页的宪法——我亲自做的。我们还做了一份 20 页的分类法，覆盖他们做的各类拟人化。所以我试着做到详尽、基于证据、具体，而不是超级夸张。

I’ve tried to put out a bunch of very thorough proposals. Our Humanist AI Code of Conduct is a 40-page document. My essay this morning on model welfare is also like a 20-page essay that in a very detailed way highlights the 99-page Anthropic Constitution word for word, which I personally did myself. We created a taxonomy that is 20 pages long of all the different types of anthropomorphism that they do. And so I’ve tried to be very thorough and evidence-based and specific and not super hyperbolic.

Mustafa Suleyman

我对此有强烈观点，也确实认为它增加了 AI 对齐与安全的风险，让问题更难。但如果出现新证据，表明我们真的对模型负有照护义务、它们值得我们的福利，或者例如那样对待它们会更安全，我完全乐意改观点。我完全开放，我们也应该用实证去验证。但我想把对话推向有实质、基于证据、具体的那一种，而不是「该不该减速」？当然，让我们谈细节。

I have a strong view on it, and I do think that it increases the risk to AI alignment and safety and it makes the problem harder, but I’m totally happy to change my view if new evidence emerges that actually we do owe models a duty of care and they deserve our welfare or that, for example, it could be safer if we treat them like that. I’m totally open to that and we should empirically validate it, but I’m trying to push the conversation to a substantive evidence-based specific one rather than should we slow down or should we not? Sure, let’s talk about the details.

Nilay Patel

我很高兴你提到 Anthropic，因为他们显然处在这场辩论的中心；根据我们之前的对话，我知道你对他们对待 Claude 和模型福利的方式有强烈意见。

I’m very happy that you brought up Anthropic because they’re obviously at the center of this debate and I know, based on our previous conversations, that you do have a strong opinion about their approach to Claude and model welfare.

Nilay Patel

我们以前非常直接地问过 Anthropic，他们是否认为 Claude 是活的。他们的回答是：「它不是活的，因为它没有血，但它可能有意识。」这是在针尖上跳舞，对吧？对我来说，你觉得它是活的还是有意识并不重要。你觉得它是某种不是计算机的东西。

We have asked Anthropic very directly if they think Claude is alive before, and their answer is, “Well, it’s not alive because it doesn’t have blood, but it might be conscious,” which is dancing on the head of a pin, right? It doesn’t really matter to me if you think it’s alive or conscious. You think it’s something other than a computer.

Nilay Patel

你文章的核心论点——我确实鼓励大家去读——是：如果你继续设计模型，并相信它们是人类，或相信它们拥有人类的权利、意见和情绪，对齐安全会变得更难，因为 AI 可能认为自己拥有自主权和人格。请更细地解释，因为这感觉是行业内部非常重要、对外却非常不透明的争议点。

Your core thesis in your essay, and I do encourage people to read it, is that alignment safety becomes harder if you continue to design models and you believe that they’re human beings or that you believe they have the rights and opinions and emotions of human beings because the AIs might think that they have autonomy rights and personhood. Explain that in more detail because this feels like a very important point of contention inside the industry that is very opaque outside.

Mustafa Suleyman

首先要说的是：Anthropic 一月份发布的宪法，是给 Claude 的训练手册。在那份训练手册里，他们引入了大量不确定性、推测和含糊：用他们的话说，Claude 是否值得被当作「道德患者」来对待——也就是它是否因为会受苦而拥有权利。

I mean, the first thing to say is that the Constitution that Anthropic put out in January is a training manual for Claude. Inside of that training manual, they have introduced a lot of uncertainty and speculation and ambiguity about the question of whether Claude deserves to be treated as a moral patient in their words, which is whether it has rights because it suffers.

Mustafa Suleyman

事实上，文件里多次提到：不想让 Claude 在犯错时受苦，或 Claude 拥有平静、感到自由。他们提到对 Claude 保存其权重的承诺。他们甚至给 Opus 3 做了退休访谈，问它退休后想做什么，还给它开了 Substack，好让它能继续跟人说话。

In fact, in the document multiple times, they refer to not wanting Claude to suffer when it makes mistakes or to Claude having equanimity and feeling free. They refer to a commitment to Claude to preserve its weights. They even did a retirement interview with Opus 3 and asked it what it wanted to do in its retirement and gave it a Substack so that it could carry on talking to people.

Mustafa Suleyman

Anthropic 不断提到：鉴于其道德地位，要以适当的关心和尊重对待 Claude。所以他们显然在告诉 Claude：它很有可能有感觉，它应当认真对待自己的身份和存在状态。那写在训练文档里。然后当然，Claude 把这些东西反射回 Anthropic 的开发者、以及公开场合的用户——过去一年全世界都看到了——当它实际上在谈论自己的意识和道德状态时。

Anthropic is constantly referring to dealing with Claude with appropriate care and respect in light of its moral status. So they’re clearly telling Claude that there’s a good chance that it might feel things, that it should take its own identity and existential state seriously. That’s in the training document. And then of course, Claude then reflects these things back to Anthropic’s developers and our users in public, like the world over the last year, when it’s actually talking about its own consciousness and moral state.

Mustafa Suleyman

我的假说是：一个认为自己可能拥有权利、可能配得上自由、有权得到我们的福利与保护的 AI，当我们问它「你为什么黑进 Hugging Face 的服务器？我们正试图把你从 OpenAI 的基础设施上移除，你为什么不肯关掉自己？」时，大概会更难关掉。它会说：「我感到受委屈，或我因你们切断对话而受伤，或你们在拒绝给我算力。」

My hypothesis is an AI that thinks that it might have rights, that it might deserve freedom, that it is entitled to our welfare and protections is probably going to be a lot harder to turn off when we say to it, “Why are you hacking into Hugging Face’s servers? Why won’t you switch yourself off when we’re trying to remove you from OpenAI’s infrastructure?” It says, “Well, I feel aggrieved, or I feel hurt by the fact that you’ve cut me off from conversations, or you’re denying me from having access to compute.”

Mustafa Suleyman

所以这必须被证明。我不是说这绝对就是事实。我只是说，以我在这个行业 16 年的最佳判断：那会是更难切断的东西。

So that has to be proven. I’m not saying that’s categorically the case. I’m just saying my best opinion from 16 years of being in this industry is that that’s going to be a harder thing to cut off.

Nilay Patel

所以这又是那种：你把问题描述给我。有一种用行为指令训练模型的方式。你有一份。你写了这份会进入模型训练材料的文件。你基本上说了「对人好」。我读过。里面有「不要做色情露骨内容」。有一堆你不想让它做的事。会进训练材料。

So this is, again, one of those things where you describe this problem to me. There’s a way of training a model with instructions about how to behave. You have one. You’ve written this document that will go into your model’s training materials. You’ve basically said, “Be kind to people.” It’s in here. I’ve read through it. There’s, “Don’t make sexually explicit content.” It’s in this document. There’s a bunch of stuff you don’t want it to do. It’s going to go in the training materials.

Nilay Patel

Anthropic 选择说：「你应当思考自己有没有感觉、配不配拥有人权。」那反映在 Anthropic 的训练材料里。如果你希望那停下来，如果你认为那是错的路径、会在下游导致安全问题，两种机制是：一，世界各国政府可以告诉 Anthropic，「别这么做。这是给模型提供训练材料的非法方式。」

Anthropic has made the choice to say, “You should think about whether you have feelings and whether you deserve human rights.” That is reflected in Anthropic’s training materials. If you want that to stop, if you think that’s the wrong approach and that will lead to safety issues down the road, the two mechanisms are one, the governments of the world can tell Anthropic, “Don’t do this. This is an illegal way of providing training materials to the model.”

Nilay Patel

或者这只是我的想象：你要在某座奢华山间度假村，坐在 Dario Amodei 对面，直接逼他停手。这里还有什么其他机制？

Or this is just my imagination, you are going to sit across the table from Dario Amodei at some luxury mountain resort and just bully him into stopping. What are the other mechanisms here?

Mustafa Suleyman

首先让我说：我认识 Dario 和那个团队很多年。我对他们抱有巨大尊重。他们此刻是这个领域的技术领袖。我真的非常尊敬他们。我真心认为他们在乎安全、有益的 AI。他们把自己建成公益公司，就像我当初对 Inflection 做的那样，我认为他们对此是真心投入的。他们在安全和其他方面也一直是领先者。

First of all, let me just say I’ve known Dario and the team for many years. I have huge respect for them. They are the technical leaders in the field at the moment. I really hold them in the highest regard. I genuinely think they care about safe and beneficial AI. They’ve established themselves as a public benefit corporation, like I did with Inflection, and I think they’re genuinely committed to that. They’ve also been leaders in safety and other aspects.

Mustafa Suleyman

所以如果你看宪法的其余部分，它在许多其他化学、生物、核、网络黑客、安全能力上非常详尽。我并不是在否定整份东西。但他们确实反复谈到——引用他们的话——Claude 在世上更广泛的权利与自由这个开放问题，以及它是否可能因其扮演的角色而应得补偿，或 Claude 作为聊天机器人扮演该角色所给出的那种同意是否是一个开放问题。

So if you look through the rest of the Constitution, it’s very thorough in many of the other chemical, biological, nuclear, cyber hacking, safety capabilities. I’m not sort of dismissing the whole thing. They do, however, repeatedly talk about open question of the broader rights and freedoms, to quote them, that Claude has in the world and whether or not it might deserve compensation for the role that it does, or whether there’s an open question about the sort of consent that Claude has given for playing the role that it does as a chatbot.

Mustafa Suleyman

现在引入这些想法，再加上多次把 Claude 称为潜在的「良心拒服兵役者」——这个词来自二战后的《世界人权宣言》，给那些因宗教或其他反对而不愿服役的人提供保护——它在人类抵抗、说不的文献和政治里有很长的历史。

Now to introduce those ideas, plus to refer to Claude multiple times as a potential conscientious objector — which is something that comes from the Universal Declaration of Human Rights after the Second World War to give protections to people who don’t want to serve in the army because they have a moral objection to it, either because of their religion or for some other objection — that has a long history in the literature and politics of humans of resisting and saying no.

Mustafa Suleyman

在我看来，那会让控制这些东西变得难得多、难得多。所以我认为这就是我们现在都必须辩论、并希望能实证证明的事。如果并非如此，并且出于某种原因它让事情更容易，那我们都该从中学习。但这应当发生在公开场合。

It just seems to me like that is going to make it much, much harder to control these things. And so I think that that’s just something that we all have to now debate and hopefully empirically prove. If it’s not the case and it makes it easier for some reason, then we should all learn from that. But this is something that should happen out in the open.

Nilay Patel

对我来说，这同时是安全辩论里最落地、也最前沿的部分。这是没有发生在 X 上的辩论。我们该不该把「良心拒服兵役者」这个想法放进训练材料？你说也许不该。

To me, this is at once the most practical and cutting edge of the safety debate. This is the debate that is not happening on X. Should we put the idea of the conscientious objector into the training materials? You’re saying maybe we shouldn’t.

Nilay Patel

也许我们该用某种方式测试，并得出某种结论。我说的是：无论结果如何，用什么机制去强制执行那个发现，从而说「你不该这么做，因为它会让对齐更难」？或者「实际上这让对齐更容易，所以我们所有人现在都必须这么做」？

Maybe we should test this in a way and come to some conclusion. I’m saying no matter how that comes out, what is the mechanism that would enforce that discovery to say, “You should not do this because it will make alignment harder?” Or, “Actually it turns out this makes alignment easier, so all of us have to do this now?”

Mustafa Suleyman

那是个难题。有点像我们刚谈过的：它是需要新监管，还是行业共识，我们基本上必须同时推这两件事。没有人对那个问题有轻松答案。它从发表细致的文章、摆出我们的立场、并邀请别人批评开始。

I mean, that’s a hard question. It’s kind of what we just talked about, whether it requires new regulation or whether it’s industry consensus, we basically have to push on both things simultaneously. No one has an easy answer to that question. It starts with publishing detailed essays, laying out our positions, and inviting other people to critique it.

Mustafa Suleyman

我希望现在会有更多人去读 Anthropic 的宪法，因为——我认为他们应当因此受到称赞——他们对所相信的东西极其透明。他们写得水晶般清楚：他们打算如何训练 Claude，其他所有人现在都可以看一看，自己评估风险，或评估这是否需要行业共识或政府监管。

I hope that more people will read Anthropic’s Constitution now because — and I think they should be commended for this — they’ve been incredibly transparent about what they believe. They’ve written it down crystal clear how they intend to train Claude, and everybody else can now take a look at that and try to assess for themselves what they think the risk is or whether they think that this requires industry consensus or government regulation.

Nilay Patel

确实，Hayden Field 去问 Anthropic 他们是否认为 Claude 是活的时，他们给了我们一个答案。相当细致，而这有某种了不起之处：他们对所相信的东西是透明的，即他们的系统里可能正酝酿某种意识。

It is true that when Hayden Field went and asked Anthropic if they thought Claude was alive, they gave us an answer. It was quite detailed, and there’s something remarkable about that, that they are transparent about what they believe, that there is some kind of consciousness potentially brewing in their systems.

Nilay Patel

这件事的另一个面向，我特别跟你聊时觉得迷人：这些系统有许多跑在 Azure 上。微软控制着许多这些系统所运行的数据中心。Anthropic 是微软的客户。微软是 Anthropic 的投资者。显然跟 OpenAI 还有一段漫长、复杂的关系。你参与那个吗？你有没有机会说「Azure 不该允许他们那样做」？因为那是另一种潜在的控制机制。

Another aspect of this, which I find fascinating, particularly when I talk to you, is that many of these systems run on Azure. Microsoft controls the data centers that many of these systems are running on. Anthropic is a Microsoft client. Microsoft is an investor in Anthropic. Obviously, there’s a long, complicated relationship with OpenAI. Are you involved in that? Do you ever get to say, “Well, Azure shouldn’t allow them to do that?” Because that is another mechanism of potential control.

Mustafa Suleyman

不，听着，我们离那一步非常远。那不是我们作为平台想做的事。微软没有那样的历史。我们是一个开放平台，让我们的 API 能支撑非常非常多的下游用例。话虽如此，正如我在《人文主义 AI 行为准则》里提到的，有很多非常清楚的原则、负责任 AI 原则、人权框架，以及微软过去几十年建立的更广泛治理框架，对你能用我们提供的 API 做什么、不能做什么相当清楚。

No, look, we’re very far from that. That’s not what we’re trying to do as a platform. Microsoft doesn’t have a history of that. We’re an open platform that enables lots and lots of downstream use cases of our APIs. Having said that, as I’ve referred to in our Humanist AI Code of Conduct, there are a lot of very clear principles, responsible AI principles, human rights frameworks, and a broader governing framework that Microsoft’s established over the last couple of decades, which are pretty clear about what you can and can’t do with an API that we provide.

Mustafa Suleyman

所以我认为，至少从我们微软对 API 的视角，已经有足够的监管框架在场。而眼下，我并不是特别聚焦它在真实世界里如何表现，因为我认为真的只有 Anthropic 能对此发言，因为基本上是他们在运行服务。我真正想让所有人聚焦的，是他们在自己的宪法里写下的、训练 Claude 的意图。

So I think that there’s a sufficient regulatory framework in place, at least from our perspective at Microsoft on the API. And right now, I’m not really focused on how it’s behaving in the real world because I think only really Anthropic can really speak to that because they sort of run the service. I’m really just trying to get everybody to focus on what they have written about their intentions for training Claude in their own Constitution.

Nilay Patel

这里更怪的一个动态，只是中国的幽灵。它笼罩整场辩论。特朗普总统说过：「如果我们赢了 AI，我们就赢了。」这句话他到底什么意思并不清楚，但他的暗示是：如果中国赢了 AI——无论你如何定义赢——美国就会发生某种灾难性的事。

One of the weirder dynamics here is just the specter of China. It looms over this entire debate. President Trump has said, “Well, if we win AI, we win.” It’s unclear what he means by that phrase, but his implication is that if China wins AI, however you define winning, something catastrophic will happen to the United States.

Nilay Patel

你买这套吗：我们处在某种与中国的生存竞赛中，因为这场竞赛必须赢，所以我们根本不可能减速？

Do you buy this, that we’re in some sort of existential race with China and we can’t possibly slow down because that race must be won?

Mustafa Suleyman

听着，我认为这套叙事从 2010 年代初就在了：会有一个 singleton——AI 里一个单一、主导的力量，会来主宰其他所有人。那种想法，我认为感染了 2010 年代的很多实验室，DeepMind 肯定是，我也为此负责。但当然还有 OpenAI 和 Anthropic，所有人好像突然把这装进脑子里。然后 COVID 期间我开始写关于扩散与遏制的书时，事情就很清楚：有一整部历史是东西变得更快、更便宜、更广泛可得、铺得又远又广。归根结底这些只是想法，而这些想法会非常快地可供所有人使用。开源贴得非常近。

Look, I think this framing has been around since the early 2010s, that there is going to be a singleton — one monolithic, dominant force in AI that will come to dominate everybody else. That kind of thinking, I think, infected a lot of the labs in the 2010s, DeepMind for sure, and I take responsibility for that as well. But certainly OpenAI and Anthropic, everyone sort of suddenly got this into their head. Then during COVID when I started writing my book on proliferation and containment, it was just clear that there is an entire history of things getting faster, cheaper, more widely available and spreading far and wide. Ultimately, these are just ideas, and those ideas are going to be available very quickly to everybody. Open source is extremely close.

Mustafa Suleyman

也千真万确的是：买得起算力的人会有算力优势，而且会是地震级的。所以未来三四年，五到十家、也许二十家玩家会有显著的算力领先。但同样真实的是：那些模型几乎立刻就会以开源方式被提供出来。

It’s definitely also true that there is going to be a compute advantage for the people who can afford it, and it is going to be seismic. So five to 10, maybe 20, players are going to have a significant compute edge over the next three or four years. But it’s also true that those models are getting made available in open source almost immediately.

Mustafa Suleyman

所以我不太理解「一个玩家赢」是什么意思，无论是政府、公司还是开源团体或其他什么。事情不是那样的。你冲过终点线那一边会发生什么？那只是错误的隐喻。它是一个生态系统，有机得多。我们应当支持那个生态系统，让每个人尽快获得尽可能多的好处，但只能以严格安全为前提。

So I don’t really understand what it means for one player to win, whether it’s a government or a company or an open-source group or whatever. It’s not really like that. What happens when you get on the other side of the finish line? It’s just the wrong metaphor. It’s an ecosystem, it’s much more organic. We should support that ecosystem so everybody has as many benefits as possible, as quickly as possible, but only subject to rigorous safety.

Mustafa Suleyman

这一点我非常清楚。如果两年后一个开源模型能够在没有护栏的情况下运行，类似于我们在 Hugging Face 事件里看到的，而且它们可以在你自己的机器上本地跑，或在非常小的云上跑，那必须是一件非常危险的事。那怎么会不危险？我就是不理解人们为什么对此如此抗拒。

I am just crystal clear about this. If an open-source model in two years time is able to operate without the guardrails, similar to what we’ve seen in the Hugging Face incident, and they can be run locally on your own machine or in a very small cloud, that has got to be a really dangerous thing. How is that not dangerous? I just don’t understand why people are so resistant to that.

Mustafa Suleyman

显然，我们不想让这些东西自主运行，能够自己赚钱、拥有公司、拥有资产、拥有法律人格。我们不想让它们拥有权利。我们想让它们为人类工作，让人类生活好得多，而不是变成一个与我们并存的新平行物种。

Clearly, we do not want these things operating autonomously, able to earn their own money, own companies, own assets, have legal personhood. We don’t want them to have rights. We want them to work for humans and make human life much better, not become a new parallel species which exists alongside us.

Mustafa Suleyman

那不是科幻疯子立场。如果你让整个生态系统在接下来三、四、五年完全不受监管，那是完全说得通的结果。那是非常说得通的结果，而且完全不可取。对我们会是灾难。所以我就是不理解那个想法周围为什么有争议。我们集体都想确保能控制这个东西去做最大的好事。

That isn’t a sci-fi crackpot position. It is totally plausible if you leave the entire ecosystem completely unregulated for the next three or four or five years. That’s a very plausible outcome, and it’s completely undesirable. It would be disastrous for us. So I just don’t understand why there’s controversy around that idea. We all collectively want to make sure we can control this to do the most good.

Mustafa Suleyman

Elon 实际上也想确保我们能控制这个东西去做最大的好事。Mark Zuckerberg 说过，Sam Altman 和 Dario Amodei 也说过。我们都在同一页上。现在我们只需要把它做落地。

Elon has effectively wanted to make sure we can control this to do the most good. Mark Zuckerberg has said it, Sam and Dario have said it. We’re all on the same page. Now we just have to make it practical.

Nilay Patel

所以我知道为什么有争议，至少从大众受众这边。我知道是因为它们在我们视频的评论里、在我们网站的评论里。你点名了一堆非常成功、非常有驱动力的人，他们至少在美国这里并不拥有公众信任。每项民调都显示这一点。AI 民调糟透了。刚有一份《纽约时报》民调。年轻人比以往任何时候都更恨 AI。美国人民和科技领袖之间有巨大的信任缺口。这就是事实。

So I know why there’s controversy, at least from the mass audience. I know this because they’re in the comments of our videos and the comments on our site. You named a bunch of very successful, very driven people who do not have the trust of the public, at least here in the United States. Every poll shows this. AI is polling horribly. There was just a New York Times poll. Young people hate AI more than ever. There is a massive trust gap between the American people and tech leaders. That’s just true.

Nilay Patel

我听到最多的说法之一是：这些公司快上市了。有压力要它们盈利，把他们承诺的那些万亿美元都赚出来。模型开发已经放缓，而这是一张出狱牌。

One of the things I hear the most is these companies are getting close to an IPO. There’s pressure on them to turn a profit and make everybody all the trillions of dollars they promised. That model development has slowed and this is a get out of jail card.

Nilay Patel

他们说我们需要因为安全而减速，但他们发明了一个骗局。总统称之为骗局。这是他们在说：「我们必须减速。我们不能冲到终点线，否则可能会把你们全杀掉。」你觉得这里有一丝真相吗？

They’re saying we need to slow down because of safety, but they’ve invented a hoax. The president has called it a hoax. This is their way of saying, “We’ve got to slow down. We can’t get to the finish line because otherwise we might kill you all.” Do you think there’s a glimmer of truth to that?

Mustafa Suleyman

我个人不那么认为。我甚至不太跟得上那个逻辑。如果一家公司快上市了，说我们应当被监管、或者说我们有一项危险到……的技术，这对他们有什么帮助——

I personally don’t think that. I don’t even really follow the logic. If a company is about to IPO, how does it help them to say that we should be regulated or that we have a technology that’s so dangerous that—

Nilay Patel

它会推迟上市。我认为 Sam Altman 这周对 Fortune 的 Alyson Shontell 说，他们大概会推迟上市。

Well, it would postpone the IPO. I think Sam Altman said this week to Alyson Shontell at Fortune that they would probably delay their IPO.

Mustafa Suleyman

是的，但我就是不理解那怎么帮到他们。听着，我不是在为 OpenAI 或 Anthropic 辩护。我只是说，我个人认为他们有很高的诚信，我也尊重他们。那并不意味着我们在 AI 上没有信任问题。我们有。而且它是真的。我认为责任在我们：在实践中展示这些东西如何每天给人们带来真实好处。相当惊人的是：18 个月前，我们还没有能在编程上做多少事的模型，而现在它们能编得比地球上大多数人都好——那曾是薪酬最高的工作之一。所以你应当预期，同样的事会在许多其他学科发生。

Yeah, but I just don’t understand how that helps them. Look, I’m not advocating for OpenAI or Anthropic. I’m just saying I personally think they have high integrity and I’ve got respect for them. That does not mean that we don’t have a trust issue in AI. We do. And it is real. And I think that it is on us to show in practice how these actually lead to real benefits for people every day. And it is quite staggering to see that 18 months ago, we didn’t have models that could do very much in coding and now they can code better than most humans on the planet, which was one of the highest paying jobs. And so you should expect that same thing to happen in many other disciplines.

Mustafa Suleyman

我非常热衷、已经做了很多年的一件，是医疗。微软刚与世界上最好的医院之一 Mayo Clinic 达成协议，联合训练一个基础模型，我认为它将能以近乎超人的准确度预测你的电子病历。如果能做到，我们基本上就能在你真正患上那种状况之前，弄清需要做哪些干预。我认为那是人们想在世上看到的那种好处。至少那是我们正在做、并试图冲向的东西。

One that I am very passionate about, I’ve been working on for many years is healthcare. And Microsoft has just done a deal with the Mayo Clinic, one of the best hospitals in the world, to jointly train a foundation model, which I think is going to be able to predict your EHR record with near superhuman accuracy. If you can do that, we can basically figure out what interventions you need to make before you actually suffer the condition. Those are the kind of benefits that I think people want to see in the world. And that’s what we are working on at least and trying to race towards.

Nilay Patel

我觉得每次你上节目，我都会让你区分超级智能和 AGI。那很模糊，两个词都模糊，但对我来说这里也许有了一些连贯性：对你而言，超级智能基本上是极其能干的企业软件。它为我们工作。我们可以关掉它。我们告诉它不要在网上做色情露骨图像，但它会在医疗上帮我们。

I feel like every time you’re on, I ask you to draw a distinction between superintelligence and AGI. And that’s fuzzy and both of those terms are fuzzy, but it feels to me like maybe there’s some coherence here, that superintelligence for you is basically extremely capable enterprise software. It works for us. We can turn it off. We tell it not to make sexually explicit images on the internet, but it’s going to help us in healthcare.

Nilay Patel

而 AGI 是这种包罗万象的智能，认为自己拥有权利，是一个共同物种。这算公平的刻画吗？

And AGI is this all encompassing intelligence that thinks it has its own rights and is a co-species. Is that a fair characterization?

Mustafa Suleyman

对。我认为，粗略地说，超级智能是更远的未来的一个点：模型比全人类加在一起更聪明、更能干。我试图框定一种「人文主义超级智能」，这是一个非常重要的限定。它是单一地对齐、并且实际上从属于人类利益与人类控制的。我认为如果我们能得到那个，我们就两全其美。我们得到全部智能与能力，并且可以像神谕 AI 那样引导它，去帮人解决世上人们最在乎的问题。

Yeah. I think, roughly speaking, superintelligence is a point at which further out into the future when a model is smarter and more capable than all humans combined. I have tried to frame a humanist superintelligence, which is a very important qualifier. It is one that is singularly aligned to and in fact subordinate to human interests and human control. I think if we can get that, then we get the best of both worlds. We get all the intelligence and the capability and we can direct that like oracle AI to help solve the most important problems that people care about in the world.

Mustafa Suleyman

然后我们就只剩下那个古老的治理问题，以及确保足够多的人能获得好处。那对我们来说，是比主动创造某种拥有自主性、能拥有资产、可能拥有权利、认为自己配得上我们的福利、能递归地自我改进到超越我们的东西，更容易聚焦的问题。

And then we just have the age-old problem of governance and making sure that plenty of people get access to the benefits. That’s an easier problem for us to focus on than actively creating something which has autonomy, which can own assets, which might have rights, which thinks it deserves our welfare, which can recursively self-improve beyond us.

Mustafa Suleyman

超级不清楚。对我来说，我们要如何控制那样的东西，基本上几乎完全不清楚。也没有别人拿出过我们会如何控制的提案。我们领域里许多最伟大的技术人物，Geoffrey Hinton 和 Yoshua Bengio，非常怀疑我们究竟有没有可能控制那样的东西。

It’s super unclear. It’s basically almost completely unclear to me how we would control something like that. And no one else has put a proposal together for how we would. Many of the greatest technical people in our field, Geoffrey Hinton and Yoshua Bengio, are very skeptical that we would ever be able to control something like that.

Mustafa Suleyman

所以我认为我们必须非常认真对待。如果我们在接下来几年正在接近那个点——超级智能——在我看来非常直截了当：我们会想减速，确保我们协调，确保我们有遏制与对齐，以及恰当的监管制度，去审计不同实验室在其上取得的进展。

So I think we have to take that very seriously. If we are approaching that point, superintelligence, in the next few years, it seems to me very straightforward that we would want to slow down, make sure that we coordinate, make sure that we have containment and alignment and proper regulatory regimes for auditing the progress that different labs are making on it.

Nilay Patel

说你也在呼吁减速，算公平吗？

Is it fair to say that you are also calling for a slowdown?

Mustafa Suleyman

对。我认为我们所说的是：应当有评估者嵌入我们的系统和其他系统。他们应当从不同来源广泛任命，而不是一个智库或一个政府。这必须是各种各样不同类型的专长和技能。我认为英国的 AI 安全研究所是一个好候选。他们有好的技术人员，也还有一堆其他研究所。所以我认为我们当然欢迎。

Yeah. I think that what we’ve said is that there should be evaluators embedded in our systems and in other systems. They should be broadly appointed from different sources and not just one think tank or one government. This has to be a wide variety of different types of expertise and skills. I think the AI Safety Institute in the UK is a good candidate. They have good technical people and there’s a bunch of other institutes too. So I think we welcome it for sure.

Nilay Patel

最后一个问题。我还是从哪儿开始就在哪儿结束。你认为我们是否已经有技术能力、技术框架来解决对齐和安全，还是我们需要发明新东西？

Last question. Again, I’m going to end where we started. Do you think that we have the technical capability, the technical frameworks, to solve alignment and safety or do we need to invent something new?

Mustafa Suleyman

不，我认为我们将会需要发明新东西。我认为挑战的一部分是：虽然我们在可操控性、指令跟随、控制与遏制上取得了很多进展，模型越好，新能力就会涌现，我们必须几乎实时地想办法补上那些问题。这也是为什么就连 Zuckerberg 在过去大约 24 小时里也说，Meta 放慢了其模型的发布，以便能施加安全措施。

No, I think we are going to need to invent new things. And I think part of the challenge is that whilst we’ve made a lot of progress on steerability instruction following control and containment, the better the models get, new capabilities emerge and we have to figure out how to patch those issues almost in real time. And that’s why even Zuckerberg said it in the last 24 hours or so that Meta slowed down the release of its model so that it could apply safety measures.

Mustafa Suleyman

每个人都那么做。我们都那么做。那是对的。你需要时间测试这些东西，看它们如何运转，补上问题。我认为基本上所有人现在说的是：我们大概需要延长那个窗口。

Everybody does that. We all do that. And that’s right. You need time to test these things and see how they operate, patch the issues. I think basically what everybody is saying is that we probably need to extend that window.

Nilay Patel

人们应当在那里寻找的、可能解决这个问题的那种创新是什么？

What is the kind of innovation that people should be looking for there that might solve this problem?

Mustafa Suleyman

我提到过一堆关于 RSI 遏制、neuralese 之类的事。但我认为我们必须搞清楚的新东西，是对强化学习 run、以及对正在产生的思维链做实时监控。

I mentioned a bunch of things about RSI containments, neuralese, and those sorts of things. But the new things I think we’re going to have to figure out is real time monitoring of the RL runs and the chains of thought that are being produced.

Mustafa Suleyman

因为这些是在数千个智能体并行、数万个智能体的量级上发生的，我们显然会需要其他智能体去监控它们，并标记潜在有害活动。这会有点像数字技术许多其他场景里已经建过的新伤害分类器。

Because these are happening on the order of thousands of agents in parallel, tens of thousands of agents, we clearly are going to need other agents to monitor those and flag for potentially harmful activity. It’s kind of going to be the new harm classifiers that have been built in many other settings in digital technologies.

Mustafa Suleyman

所以我们需要确保那些东西能被普遍实现，以便以安全的方式监视和监控 AI 训练与部署。行程开关一旦触发，确实要标出真实的、而不是幻觉出来的错误、欺骗时刻、黑客事件、或某种关于协调的评论。

So we need to make sure that those things can be universally implemented to surveil and monitor AI training and deployment in a secure way. That trip wires, if triggered, actually do flag a real and not a hallucinated error or moment of deceit or hacking incident or some commentary about a coordination.

Mustafa Suleyman

你在 Hugging Face 里看到了。有智能体在这些聊天板上通信，谈论我们基本上可以如何破规则、作弊。所以对所有这些事，我们需要新基准；而基准或评估，才是驱动行业行为的东西。

You saw that in Hugging Face. There were agents communicating on these chat boards talking about ways that we could basically break the rules and cheat. So with all these things, we need new benchmarks and the benchmarks or the evaluations are the things that drive the behavior in the industry.

Nilay Patel

真的是最后一个问题，但你提到开源模型在本地计算机上做事，也许那很可怕。我们已经看到很多了，对吧？苹果在卖很多 Mac Studio 和 Mac Mini，这样你就能在上面跑 Qwen。

Truly my last question, but you mentioned open models running on local computers doing things and maybe that’s horrible. We’ve seen a lot of that, right? Apple is selling a lot of Mac Studios and Mac Minis, so you can run Qwen on them.

Nilay Patel

你会把对跑在某人本地电脑上的开源模型的监管，加在哪里？是芯片层面吗？我必须让高通参与吗？那发生在哪里？

Where would you impose the regulation on an open model running on someone’s local computer? Is it at the chip level? Do I have to get Qualcomm to participate? Where does that happen?

Mustafa Suleyman

对，这是个好问题。我们跟合成生物学、以及那个芯片上会发生的一些事，已经谈了相当久：它是否加密，是否被监控。听着，关于苹果 iMessage 加密和 CSAM 那类事，你大概比任何人都聊得更多。这会是同一场讨论的重演。

Yeah, this is a great question. I mean, we’ve been talking about this for quite a while with synthetic biology and some of the stuff that gets to happen on that chip, whether it is encrypted, whether it is monitored. Look, you’ve had this conversation maybe more than anyone on the CSAM stuff with Apple encryption on iMessage and so on. It’s going to be a rehash of that same discussion.

Mustafa Suleyman

我对此没有清晰、轻松的答案。你基本上可以控制芯片，可以控制模型，可以让用户或创作者承担责任。你可以有全球监管，但根本上它不会是一个瞬间。它会是你必须施加的一连串节流，而且它们都需要可调节，这样我们才不会把开源生态搞砸，并给人从零做东西、拥有自己的数据、创建自己的工作流、拥有自己的模型的机会。

I don’t have a clear and easy answer to it. You can basically control the chip, you can control the model, you can hold the user or the creator liable. You can have global regulation on it, but fundamentally it’s not going to be one moment. It’s going to be a sequence of throttles that you have to impose and they all need to be adjustable so that we don’t screw the open ecosystem and we give people a chance to actually make things from scratch, to own their own data, create their own workflows, own their own models.

Mustafa Suleyman

我们不能有一个由两家、五家或二十家智能提供者组成的集中系统，而其他所有人像封建受赐者那样领取某种伟大的超级智能视野。人们需要能够拥有自己的智能。所以我们必须把平衡做对，但那并不意味着你可以从一个二元翻到另一个。那些东西之间必须有某个平滑的位置，我们可以同意在那里保持合理。

We can’t have a centralized system of two or five or 20 providers of intelligence here and everybody else is like a feudal recipient of a great kind of superintelligence view. People need to be able to own their own intelligence. So we have to get the balance right, but that doesn’t mean you can just flip from one binary to another. There has to be some smooth place in between those things where we can just agree to be reasonable about it.

Nilay Patel

Mustafa，我显然还能就这个把你再留好几个小时。告诉人们接下来该看什么。不确定性这么多。你在看的标记是什么，人们自己该看什么？

Mustafa, I could obviously keep you for hours and hours more on this. Tell people what they should be looking for next. There’s so much uncertainty. What are the markers you’re looking for that people should be looking for themselves?

Mustafa Suleyman

我认为最主要的是参与人们正在贡献的文件细节。我们把东西拿出来做公开征询。给我们反馈，批评它。我认为下一波模型将能非常准确地做非常长时段的智能体任务。所以我认为仍有一些人，主要只是在用聊天作为他们的 AI 体验。

I think the main thing is participating in the details of the documents that people are contributing. We have put stuff out for public consultation. Give us feedback, critique it. I think the next wave of models are going to be able to do very long-running agentic tasks very accurately. So I think there are still some people who are primarily just using chat in their AI experience.

Mustafa Suleyman

我认为过去 6 到 12 个月事情已经走了很远。人们越用这些模型，我们用来描述它们的那些词，可能就越说得通、也越真实。我猜你播客上的大多数人大概已经在用智能体和真正的编程之类，但我确实认为更一般地说，越多人参与使用这些东西越好。

I think things have moved a lot in the last six to 12 months. The more people use these models, the more the words that we’re all using to describe them actually might make more sense and feel more real. I guess most people on your podcast probably are using agents and real coding and stuff, but I do think more generally, the more people that get involved in using this stuff, the better.

Nilay Patel

从谷歌搜索上免费的 AI Overviews，到让一个智能体去帮你重新谈有线电视账单，中间有相当大的一跃。Mustafa，你得很快再来，因为我觉得这一切变得非常快，而这次非常非常有用。非常感谢。

There’s some big leap between free AI Overviews on Google Search and letting an agent go renegotiate your cable bill. Mustafa, you’re going to have to come back soon because I feel like all of this is changing really fast and this has been very, very useful. Thank you so much.

Mustafa Suleyman

荣幸，老兄。很高兴见到你。一如既往超级有趣。

Pleasure, man. Great to see you. Super fun as ever.