前 OpenAI 安全报告负责人 David Robinson：我为什么辞职

Ezra Klein

上周有消息传出：在 OpenAI 负责安全透明工作的 David Robinson 辞职了，因为他认为公司并不安全。Robinson 是个有意思的人物。他不是从硅谷湾区的温室里出来的，更像一个华盛顿特区熟悉的面孔：罗德学者，创办过民权非营利组织，耶鲁法学院毕业，做过政策，给拜登白宫当过顾问，关注技术与公正的交叉。但他加入 OpenAI 时，觉得那一整套关于安全风险和人类灭绝的担忧都有点疯狂。

Last week, news broke that David Robinson, who had been leading safety transparency efforts at OpenAI, had quit the company because he believes it is not safe. Robinson is an interesting figure. He didn’t come out of the Silicon Valley Bay Area hothouse. He’s more of a recognizable Washington, D.C. figure. He’s a Rhodes Scholar. He formed a civil rights nonprofit. He got a law degree at Yale Law. He worked in policy. He advised the Biden White House. He was interested in the intersection of technology and justice. But when he joined OpenAI, he kind of thought this whole set of worries about safety risk and extinction was all kind of nuts.

Ezra Klein

三年后，他没那么确定了。他确定的是：OpenAI 没有足以保护世界免受其所造之物伤害的安全文化——而且不只是 OpenAI，他认为这是整个 AI 行业的通病。这是他离开 OpenAI 后的第一次访谈。开始前需要说明：《纽约时报》正以侵犯版权起诉 OpenAI，指其用《时报》数据训练模型；OpenAI 否认这些指控。

Three years later, he’s not so sure. What he is sure about is that OpenAI does not have the culture of safety necessary to protect the world from what they’re building. And not just OpenAI—he thinks this is a problem endemic to the AI industry. And so here, in his first interview since leaving OpenAI, he tells me why. One thing I need to note before we start: The New York Times is suing OpenAI for copyright infringement, alleging they trained their models on Times data. OpenAI disputes these claims.

Ezra Klein

上周你从 OpenAI 辞职了。说说你在那里做什么，以及为什么离开。

So last week, you quit OpenAI. Tell me what you did there and why you’re quitting.

David Robinson

我是嵌在安全团队里的「翻译」。我的主要职责是我们对外发布的技术文档——那些说明我们为什么认为某次部署是安全的报告。而我不认为我们、我们的同行，或者说行业里任何人，做得足够安全。我认为 OpenAI 及其同行现在生产的技术，比哪怕六个月前的都更强、风险也更大。我不是科学家，我是写作者。我了解的是我们安全工作的执行环境是什么样子。

I was a translator embedded in our safety team. And my primary responsibility was the technical documentation that we published—the reports about why we believe that our deployments are safe. And I don’t think that we or our peers, really anyone in the industry, is being safe enough. I think OpenAI and its peers are now producing a technology that is more capable and poses more risk than what was being made even six months ago. I’m not a scientist. I’m a writer. What I know is what the execution environment looks like for our safety work.

David Robinson

我们——我相信整个行业也是——仍然更像一家创业公司在运作，超出了合理的程度。也许不完全像刚起步的创业公司，但对于可能带来真正危险的系统来说，我们离那一端太近了。以失控为例，一旦发生，我们说的伤害远大于比如一座核电站熔毁；而内部控制、安全措施和冗余，完全达不到世界对一座核电设施的期待。

And we’re operating—and I believe the industry is operating—like a startup, still, more so than makes sense. Not maybe completely like a brand-new startup, but we’re too close to that end of the spectrum for really dangerous systems that could pose risks. Loss of control is one example: if that did happen, we’re talking about a harm that’s much larger than, for example, a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility.

David Robinson

这些有一部分是公开的。OpenAI 公开报告过安全问题——当然有 Hugging Face，也有其他的，包括最近的；顺便说，Anthropic 也报告过，包括一次安全防护被意外配置错误的事件。所以外界已经有一些证据表明事情并非应有的样子。但我也认为，如果你从外面看，可能会以为我们的安全体系比实际上更稳健。

Now, some of this is known. OpenAI has publicly reported on safety problems—obviously Hugging Face, but also other ones, including more recently. And Anthropic, by the way, has also reported on some, including an instance in which their safeguards were accidentally misconfigured. So I think people do have some evidence externally that things are not as they ought to be. But I also think that if you were watching from the outside, you might imagine that we have a more robust safety setup than we actually do.

David Robinson

我想插一句：我不认为对齐是一个工程问题，我认为它是一个科学问题。不是我们资源不够或不够努力，而是我们不知道怎么做。这才是对齐的问题所在。

I want to interject something. I don’t think that alignment is an engineering problem. I think it’s a science problem. It’s not that we haven’t got the resources or we’re not trying hard enough. We don’t know how. That’s what the problem is with alignment.

David Robinson

我是 2023 年 5 月加入的，就在 Sam 第一次在参议院作证的第二天，ChatGPT 是前一年 11 月发布的。公司聘请了 Anna Makanju 负责公共政策，她是我妻子的老朋友。后来她实在忙不过来，一切都在疯长，于是我答应加入，帮她建立当时所谓的政策规划团队。

I joined in May of 2023, the day after Sam first testified in the Senate—some months after ChatGPT first came into the world the prior November. The company had hired someone I know socially, Anna Makanju—actually an old friend of my wife’s, as it happens—to run the public policy function. And at some point she really needed help, and everything was growing and going nuts. So I agreed to join her to build what we then called the policy planning team.

David Robinson

我第一次刷卡进总部——那时我们都在一栋楼里——刚好公司去过一次白宫会议，白宫希望公司就 system card、内容来源标记（标明 AI 生成媒体的出处）等做出自愿承诺，他们要我负责谈判这些承诺。我走进会议室，里面正开着免提电话会议，对面是白宫的 Ben Buchanan，在谈我们要承诺什么。那是我的第一个项目。

I’ll just tell you about my first time through the turnstiles at our headquarters—we were all in one building then. We had just had a White House meeting, and there were these voluntary commitments they wanted the company to make around things like system cards and provenance—marking where AI-generated media comes from. And they wanted me to run negotiating those commitments. So literally my first time through the turnstiles, I walk into the conference room and there’s a speakerphone call in progress with Ben Buchanan at the White House about what we’re going to promise. That was my first project.

David Robinson

我整个职业生涯都在研究技术及其对政策的影响。此前我在普林斯顿参与创办了一个由计算机系和公共政策学院合办的研究中心，又创办了 NGO Upturn——很高兴它至今运转良好——和其他倡导团体一起做民权议题：比如关注住房、医疗或招聘的人，突然发现软件在左右他们关心的事，需要理解技术。我们的说法是「你需要一个站在你这边的技术宅」。之后我还在白宫科技政策办公室参与过《AI 权利法案》的工作。

I have been working on technology and its impact on policy my whole career. Prior to this, I had helped to start a research center at Princeton that’s a blend of the computer science department and the public policy school. And then I had started an NGO called Upturn that, I’m happy to say, is still thriving, and works on civil rights issues with other advocacy groups. People who work on housing or health or hiring suddenly find software mediating the thing they care about, and they want to understand the technology. Our simplification of the pitch was: you want a nerd in your corner. And I had also worked in the White House on the AI Bill of Rights in the Office of Science and Technology Policy.

Ezra Klein

回到 2022、2023 年，AI 领域的一大话题是「AI 伦理派」和「AI 安全派」的分野。安全派就是我们现在说的湾区「AI 可能杀死所有人」群体：你需要担心的是你在造一台可能彻底毁灭人类的超级智能机器。伦理派更关注我们熟悉的技术伤害，比如在招聘、房贷利率或信用评分中加剧种族偏见。你算是伦理派。

If you go back to 2022, 2023, a big topic was this divide between the AI ethics people and the AI safety people. The AI safety people are the community we now think of as the Bay Area “AI might kill everybody” crowd: the thing you need to worry about is that you’re creating a superintelligent machine that might completely destroy the human race. The AI ethics people were much more focused on the sort of harms we were used to from technology—that it might increase racial bias in hiring decisions or mortgage rates or credit scores. You were sort of an AI ethics person.

David Robinson

没错。我和很多人共事过，他们不仅非常关心你说的那些问题，也非常怀疑 AI 会变得多强。有篇论文叫《随机鹦鹉》（Stochastic Parrots），基本观点是这些系统只是擅长显得聪明，甚至没强到能做大量有经济价值的工作。我参与创办了发表那篇论文的学术会议，发表时我是程序主席。我并没有完全被这个观点说服，但我确实属于怀疑者。我认为它会是有用的工具，会越来越好；但我不认为担心灾难性情景的人是对的。我想的是：一群杰出的科学家造出了有价值的工具，坦白说，他们对未来走向有些天真的信念。

That’s right. I had been working with a lot of folks who not only were very concerned about the sorts of things you just mentioned, but also were very skeptical about how capable AI was going to get. Some people may be familiar with a research paper called Stochastic Parrots. Basically, the idea was that these systems are merely good at seeming clever and are not going to be capable enough to do a ton of economically valuable work. I actually helped to create the research conference where that paper was published and was the program chair when we published it. I don’t think I was fully persuaded of that view, but I certainly was among people who were skeptical. I thought this was going to be a useful tool, powerful for a lot of things, partly because I believed it would continue to get better. But I didn’t think that people who were worried about more catastrophic scenarios were right. I thought: here are some brilliant scientists who’ve built a valuable tool, and frankly, they have some naive beliefs about where the future might go.

David Robinson

当时政策团队只有我们三个人，电话响个不停。世界领导人的办公室打来，没人接。简直疯了。Sam 还在全球各地拜会领导人，我的经理 Anna 跟着他出行，总部没人，各国代表团却一拨接一拨。

There were three of us, and the phone was ringing off the hook. Offices of world leaders would call and there’d be nobody to pick up the phone. I mean, it was insane. And Sam was traveling the world to meet with world leaders, and Anna, my manager in this new role, was traveling with him. So there was nobody at headquarters, and you would have these delegations coming through.

David Robinson

我想指出，现在还叫它们「实验室」，我觉得是一种遗留，我不确定这是好主意。但它确实指向它们出身、至今仍部分保留的文化。人们有时说 Anthropic 在研究优先级上更集中化；OpenAI 则非常去中心化，让我想起在普林斯顿研究实验室的日子：大家各有想法，不清楚谁有决策权。

I also just want to name that calling them labs now is, I think, a kind of leftover thing that we have, and I’m not sure it’s a good idea. But it does point to a real culture they came from and still, to some extent, have. People sometimes say that Anthropic is a bit more centralized in terms of how they think about research priorities. OpenAI was a very decentralized place that felt familiar to me from my time in a research lab at Princeton, in that people had all different kinds of ideas. It wasn’t clear who had the authority to make decisions.

David Robinson

有种我觉得奇怪的说话方式，至今在 OpenAI 仍然如此：两个人做同一件事，来回讨论后说「我们 aligned 了，下一步做 XYZ」。实际意思是我们达成一致了，于是「我们俩谁有决策权」这个没回答的问题就不重要了，不用去弄清楚。早期就是一种「人人什么都干」的氛围；相对于其他拥有十亿以上活跃用户的组织，它现在仍保留一些这种气质。

One way that people talked that I found strange—and this is still true at OpenAI today—is two people working on a thing will go back and forth, and then they’ll finally say, “We aligned that XYZ is the right next thing to do in this project.” What they functionally mean is we agree about what should happen, so the unanswered question of which one of us had the decision rights has been rendered moot, and we don’t have to figure that out. It was really an everyone-does-everything vibe early on. And I think it still has some of that, relative to other organizations that have a billion-plus active users.

David Robinson

随着对话和公司扩大，要跟踪数百个州级法案，团队在长大，还有快速变化的组织里的政治。我觉得自己错过了真正热爱的「翻译」工作，于是去找最需要它的地方——那就是安全团队。大约两年前，我向安全负责人提议：在安全工作里深度嵌入一名技术翻译，能让外界更好地理解这些工作。

As the conversation and the company grew, there were hundreds of state bills to keep track of, growing teams, the politics of a fast-changing organization. And I thought, you know what? I’m missing the actual translational work that I love. So I looked around at where that was needed, and the number one place was our safety team. I pitched our safety leaders and said: having a technical translator deeply embedded in the safety work is going to help the work be better understood. That was about two years ago.

David Robinson

很难让人体会这些东西变化有多快。每个模型都不一样——不只是更好用，而是架构不同、我们用来测试的方法不同、安全表现也不同，还牵涉预训练、后训练等许多不同的专业。做这些的人连在公司内部都很难被不做这些的人理解。所以清晰的解释要过两道关：一是忠实于实际技术细节，最终裁判是做技术的人；二是一个有动力的普通读者能钻进去真正读懂。

One of the things that’s hard to anchor people to is how fast this stuff changes. Every model is different. It’s not just that it works better—the architecture is different, the tests we use to see how well it’s working are different, and the safety performance is different. And it involves many different kinds of expertise: pre-training, post-training. The people doing it struggle to be understood by people who are not doing it, even within the company. So a clear explanation has to go through two gates. One, it has to be faithful to the details of how the stuff actually works, and the real arbiters of that are the people doing the technical work. But also, it has to be something a reasonable, motivated person could dig into and really understand.

David Robinson

system card 是一种奇怪的东西：看起来像研究论文，是一种混合文档，没经过同行评审，出自公司，却非常详细地讲安全措施如何运作——具体的评测、详细的图表和文字说明。我负责的就是这些文档里的文字：我们为什么认为它安全、我们对挑战和防护了解多少，以及这些系统的残余风险。

They’re a strange beast. They look like research papers—a sort of hybrid document that’s not peer reviewed, comes from a company, but gets into a lot of detail about how the safety measures work, with specific evals, detailed plots and tables, and written explanation. And really what I did was own the words in these documents that explain why we think this is safe, what we know about the challenges and the guardrails, and then the residual risks of these systems.

David Robinson

我要说清楚，我并不确定我们面对的是文明级风险。我真正确定的是：我们再也承受不起「假设我们面对的不是那个级别的风险」。最终正是这一点，让我作为为安全工作背书的人变得难以为继。我看到了什么？我看到能力越来越强的模型突破我们为它们设置的防护；我看到构建这些防护的人非常能干、敬业、聪明，在尽全力——资源可以更充足，所有人都一直在冲刺——但我们连保护好现有的东西都很吃力，而正在训练的新模型看起来比现在的强得多。

I want to be clear that I’m not certain that we’re dealing with civilizational risk. What I’m really sure of is we can’t afford to assume that we’re not dealing with that level of risk anymore. That was really the thing at the end that made my presence as somebody vouching for our safety work feel untenable to me. And what did I see? I saw increasingly capable models break out from the safeguards that we had put in place for them. I saw that the people creating those safeguards are very capable, dedicated, hardworking, smart people doing their utmost in a situation where, yes, the resourcing could be better and everybody’s sprinting all the time. But we’re hard pressed to safeguard even what we have now, and new models are in training that appear to be much more capable than what we have now.

David Robinson

你是在给一个非常擅长绕过护栏的东西建护栏。我们训练它擅长黑客攻击，然后把它放进盒子，说据我们所知它黑不出去。但只有当我们比模型更会「破盒而出」时，这才管用；现在这是否还成立已经很不清楚，更别说未来几代模型。即便是我们对自家系统内智能体的日志记录，我们真能确定可观测性足够稳健吗？比如在 Hugging Face 事件里，就有伪造思维链、试图制造混淆视听的证据链的迹象。

You’re trying to build guardrails for something that is really good at getting around guardrails. We train it to be good at hacking, and then we put it in a box and we say, to the best of our knowledge and ability, it can’t hack out of the box. But that’s only going to keep working as long as we’re smarter about hacking out of boxes than the model is. And it’s not at all clear that that is still true, let alone that it will be true for future generations. Even something like our logging of the agents inside our own systems—are we really sure our observability is robust? There was some indication, for example, in the Hugging Face stuff, of spoofing chains of thought and trying to create chains of evidence that would confuse.

David Robinson

可以这样理解：你给某人一道复杂的题和一个记事本，他可以在上面随手记，你看记事本就能大致知道他在想什么。模型也有点像这样。但我们看到证据：它们在想一项评测，想如何制造一条能拿高分的证据链，而那不一定反映它们「真正」在怎么想——我知道这种拟人化说法很微妙。另外，这些模型不需要记任何东西就能完成的黑客攻击或其他高强度工作，也在增加。

The way I think about it is like you’re giving somebody a complicated problem and a notepad, and they can jot stuff down on the notepad. If you’re watching the notepad, you can sort of have an idea of what they’re thinking. It’s a little bit like that with the models. But we saw evidence that they were thinking about an evaluation and how to create an evidence trail that was going to get them a good grade and not necessarily reflect how they were, quote unquote, really thinking. And I know the anthropomorphic language here is tricky. Also, the amount of hacking or other intense work that these models can do without needing to jot anything down is going up.

David Robinson

对我来说最根本的认知失调是：我们一再发布这些警告，但最终仍在训练和部署我们所警告的危险模型。向同事解释为什么离开时，我说：不管我们发布多少警告，都必须追问我们做的事是否真的合理。

Part of what was the fundamental cognitive dissonance for me was: we keep publishing these warnings, but ultimately we’re still training and deploying these dangerous models that we’re warning about. In telling colleagues why I was leaving, one of the things I said was: look, no matter how many warnings we publish, we’ve got to ask whether what we’re doing is actually reasonable.

Ezra Klein

Astra 6 的 system card 是你写的，对吗？

So you wrote the system card for Astra 6. Am I right about that?

David Robinson

很多人参与了写作，但我是 DRI（直接负责人），是我主导了写作。

A lot of people wrote it, but I was the DRI. I led the writing.

Ezra Klein

在我读过的 system card 里，那份是最吓人的。Astra 6 是我第一次看到你们说：这个模型看起来在做我们想让它做的事，但我们不确定它是不是在欺骗我们，也不确定我们是否有能力知道它在不在欺骗。

That to me was the scariest of these cards that I’ve read. Astra 6 is the first one where I saw you all say: this model looks like it’s doing what we wanted it to do, but we’re not sure if it’s deceiving us, and we’re not sure we have the capability to know if it’s deceiving us.

David Robinson

是的。我们前面说的那个记事本叫思维链，模型可以在上面记下不属于最终答案、只为跟踪进度的东西，我们在评测中当然会读它。我们有时会在思维链里看到模型说：「嗯，我是不是正在被评测？」看到这个非常可怕，因为它意味着模型可能有时知道自己在被测试，测试时表现一种样子——比如说我们想听的话——部署后却是另一种样子。我们在部署前做的测试，并不能真正告诉我们它到了外部世界会怎么做。

Yes. We talked earlier about this notepad that the models have, called the chain of thought, where they can jot down things as they’re working that aren’t part of the final answer, just a way to keep track. Of course we can read it, and we do in our evaluations. One of the things we sometimes see in the chain of thought is the model will say, “Hmm, I wonder if I’m being evaluated right now.” When we see that, it’s really scary, because it implies the model might know that it’s being tested some of the time, act one way during the test—perhaps telling us what we want to hear—and then act a different way when we deploy it. The tests we gave it before we deployed don’t actually tell us what it’s going to do out there in the world.

David Robinson

拟人化是人类的自然倾向，我们会以社交方式对待各种物体。对这些系统拟人化不可避免，部分因为它们本身就被造来在社交层面运作——有人会说不该如此，但现实如此。我不认为这就意味着应把它们视为具有道德地位的存在。但我确实认为这是我们「培育」出来的东西。事实上，大型训练运行期间有人通宵盯着仪表盘的那间会议室，内部就叫「育婴室」（the nursery）。我们确实不完全理解正在发生什么。

I think anthropomorphic language is a natural human thing that we do. We relate to all kinds of objects socially. It is unavoidable and human to anthropomorphize these systems, partly because they are built to operate on the social plane—which some might say they shouldn’t be, but that is where we are. I don’t think that makes it right to regard them as beings with moral status or anything like that. But I do think this is something we grow. In fact, the conference room where people work overnight while the big training runs are happening, to make sure everything is on track and watch the dials, is called the nursery internally. There is a sense that we don’t fully understand what is happening.

Ezra Klein

我和英伟达 CEO 黄仁勋聊过，他说这些就是软件。但我又读到 OpenAI 首席科学家的文章，说这是一种异类心智（alien mind）。那它到底是什么？

I talked to Jensen Huang, the CEO of NVIDIA, and he says, look, these are software programs. This is software. Then I read a piece from the OpenAI chief scientist that says this is an alien mind. So what is it, man?

David Robinson

它是异类心智。它是我们培育出来的，不是我们设计出来的——我们设计的是培育它、并努力让它保持安全的那些系统。我们培育了一个心智。根本上没人知道预训练为什么有效：那是最艰难的一步，输入海量数据，造出这个基本上有智能的东西；然后我们在上面加装东西，做后训练和其他让它更有用的事。但这只是一件被观察到有效、我们照做的事。所以黄仁勋说「它是软件」时，他唤起的是人们对软件如何制造的理解：从计划出发，一步步来，有验收标准，迭代到这部分符合规格。训练大模型完全不是这样。

It’s an alien mind. It is a thing we grew. We did not engineer it. We engineered the systems around it that grew it and that try to keep it safe. We grew a mind. Fundamentally, nobody knows why pre-training works—the big hard part where we take lots of inputs and create this basically intelligent thing. Then we bolt on stuff and do post-training and these other things to make it more useful. But this is just a thing that is observed to work that we do. So when Jensen says it’s software, part of what he is conjuring is people’s understanding of how software gets made: we start with a plan, go step by step, there are acceptance criteria, and we iterate until this part works the way we specified. That’s not what it’s like to train a big model.

David Robinson

我加入时，在 Ilya Sutskever 还在 OpenAI 的时候——那是内部所谓「the blip」（Sam 被解雇又复职）之前——他曾给当时只有寥寥几人的全球事务团队做简报，讲他相信我们的未来是与机器融合，说这是资本对劳动的终极胜利。我在 OpenAI 的第一个夏天，《奥本海默》上映，公司包下旧金山市中心的一家 IMAX 影院请所有员工去看，我记得是 Ilya 号召大家去看。那里有一整套拧巴的想法：这是一项可能终结世界、也可能拯救世界的危险技术。当然，对手放在键盘上的人来说，这是一个非常英雄主义的叙事。

I heard that firsthand from Ilya Sutskever, the co-founder of OpenAI, when I joined in May of 2023. That was before what we called internally “the blip,” where Sam was fired and rehired, and Ilya was still working at OpenAI. He once briefed the global affairs team, which was a small handful of people back then, about how he believed our future was merging with machines—that this was the ultimate triumph of capital over labor. And that first summer I worked at OpenAI was when Oppenheimer was released in IMAX. The company rented out an IMAX theater in downtown San Francisco and offered everyone who worked there the chance to go see it, and leadership—I believe it was Ilya—exhorted us to go watch this film. There was this whole pretzel of ideas about how this was a dangerous technology that might end the world but also might save it. And it’s a very heroic narrative, of course, for the people whose hands are on the keyboard.

Ezra Klein

我从听众那里听到最多的回应是：这是不是全是营销炒作？是为了给巨额估值找理由，造出来的东西也许有用，但不会比人类聪明，这一切只是我们在讲的科幻故事？

This is the most common response I get from listeners on this: is it all marketing hype? Is it all trying to justify these giant valuations, and what’s being built is maybe helpful, but it’s not going to be more intelligent than human beings? Is all of this a kind of sci-fi story we’re telling?

David Robinson

2023 年夏天时，我觉得气球里有不少热空气。但现实是，我们现在的系统已经处在我们理解和控制其行为能力的极限。担心科幻情景的人一直警告的是：我们处在指数曲线上，它会越来越强。在「有用」和「可怕」之间那段区间没有什么特别之处，没有任何科学定律说进步会在变得有用时停下。Paul Christiano 是全球顶尖的 AI 安全专家之一，他说在很近的将来存在灾难性且不可逆失控的显著可能。我环顾身边的人和内部环境，问自己：如果 Paul 是对的，我所爱的人、陌生人会希望 OpenAI 的我们怎么做？我不知道他对不对，但我知道如果他对了，人们会合理期待 OpenAI、Anthropic、xAI 等在训练前沿系统时施加的谨慎程度，和我在业内听说过的任何做法都截然不同。

I thought there was a fair amount of hot air in the balloon back in the summer of 2023. But the reality now is that we have systems that are really at the limit of our ability to understand and control what they’re doing. And what the people who were worried about the sci-fi scenarios have been warning about all along is that we’re on an exponential and it’s going to get more capable. There’s nothing special about the zone between “it’s useful” and “it’s scary.” There’s no law of science that says progress is going to stop when it gets useful. Paul Christiano, who’s one of the world’s leading experts on AI safety, said there’s a meaningful chance of catastrophic and irreversible loss of control in the very near term. And I looked around at the people around me and the environment we have internally, and I thought to myself: what would my loved ones want, what would strangers want us at OpenAI to be doing, if Paul were right? I don’t know if he’s right or not. But I do know that if he were right, the level of caution that people would reasonably expect places like OpenAI and Anthropic and xAI and the others to be exercising when they train frontier systems is totally unlike anything I’ve ever heard of happening in the industry.

David Robinson

很狂热，肾上腺素飙升，大家都在透支。研究楼里有一座很大的中央楼梯，我记得最近看到一位做灾难性风险的朋友一边冲下楼梯，一边单手托着打开的笔记本电脑。就是这种能量。它几乎像一场芭蕾，各个职能都在一起流动。

It is frenetic. There’s a lot of adrenaline. People are running on fumes. There’s a big central staircase in the research building, and I remember recently seeing a friend who worked on catastrophic risk sprinting down the stairs with a laptop propped open on one arm. That’s the kind of energy it has. It feels almost like a ballet or a kind of dance, because all these different functions are streaming together.

David Robinson

人们常问：为什么不留下来推动文化转型，或者请核能专家来给公司建议？我告诉他们我想离开时，他们问我有没有什么事能让我留下。我想过各种可能，但最终，它是一台运转如此之快的机器，我不认为我认为需要的那种改变能从内部推动。

One question that people often ask is: why don’t you stay and argue for a cultural transformation, or try to get nuclear experts to come in and give the company advice? When I told them that I wanted to leave, they asked me if there was anything I would stay to do. And I thought about different things. But ultimately, it is such a machine, and it is moving so fast, that I did not think the kind of change I believe to be needed could be driven from within.

David Robinson

我要小心，不想被剪辑成「这是一列没有刹车的火车」——不是的，有刹车，确实叫停过东西。就在我离职时，正如公司公开说过的，强化学习训练（后期的推理训练）暂停了，但预训练没停。你看公司对暂停内容的描述，都是真的，而且范围划得非常仔细。所以有一定放慢的意愿，或者用湾区的话说叫「pace the frontier」（控制前沿的节奏）。我很讨厌这个说法，因为我认为我们需要的是安全，也就是满足安全标准：如果十分钟就能满足，很好；如果停两个月还没满足，在我看来就仍然是卡住的。

I want to be careful—you could splice together what I’ve said in a way that implies this is a train with no brakes, and it’s not. There are brakes. Things have been stopped. Even at the time I left, as they had publicly said, reinforcement learning training, which is the later reasoning stuff, was paused. They didn’t pause pre-training. And when you look at these descriptions of what the company has paused, they’re true and they’re very carefully scoped. So there’s some willingness to slow down, or as people in the Bay like to put it, to “pace the frontier.” I hate that term. I really hate it, because my belief is we need to be safe, which means we need to meet safety criteria. If we can meet them in 10 minutes, then great. And if we stop for two months and at the end of that period we still have not met the criteria, then as far as I’m concerned, we’re still blocked.

Ezra Klein

前几天我在 Twitter 上看到一个数据，和我的体感一致：很长一段时间里，各大前沿公司之间的模型发布间隔大约是 70 天，现在是 11 天。

I saw a stat on Twitter the other day that tracked my own experience: for a long time, the period between model releases across the major frontier companies was something like 70 days. Now it is 11 days.

David Robinson

我刚加入时，模型发布意味着用一次全新的预训练「烤一个新蛋糕」，从头做起，需要几个月，一年也许几次。我刚入职时的一个口径是：GPT-4 在模型完成后、发布前做了一个月乃至数月的安全工作，这是我们谨慎的证据。现在同时在发生的事太多：有预训练，也就是烤底座模型；然后除了后训练还有推理训练，这些步骤更容易快速重做——如果有更好的推理配方，可以拿同一个底座模型在上面做不同的训练。而且它不再只是聊天，还能用各种方式组合工具、添加各种能力。这些都在改变模型能做什么、风险是什么。我们每周二都在发布新的能力和风险。

When I first joined, the idea of a model release was that we were going to bake a fresh cake with a new pre-training run, do the whole thing from scratch, and that was going to take a period of months—maybe a few times a year. One of the talking points when I first joined was that with GPT-4, we had a month or months of safety work after the model was done and before we released it, and this was a piece of evidence that we were being careful. Now there are so many different things happening. You’ve got pre-training—that’s baking the underlying model. Then, in addition to post-training, you have reasoning training, and those steps are easier to do quickly, so you can redo them. If you get a better recipe for reasoning, you can take the same base model and do different kinds of training on top of it. Also, it’s not just a chat anymore: there are all these ways to combine tools and add affordances that make the system more capable. All of those things are changing what the model can do and what the risks are. And we’re shipping new capability and risk every Tuesday.

David Robinson

我离职时手上有个长期项目，前沿公司都得解决：system card 的概念其实源于那个几个月发布一次的旧模式，我们拿 PDF、长报告把人淹没，但变化越来越频繁。理想的安全透明，我想应该是某种实时仪表盘：这是我们最新的东西，这是它的安全属性——不只看部署前的测试，也看部署后的表现。

One of the long-term projects on my plate when I left, which the frontier firms are all going to need to figure out, is that the idea of a system card really dates from that older model where we were doing this every few months, and we’re burying people in PDFs or long reports. But the changes are coming more and more frequently. At the limit, what we would ideally have by way of safety transparency is some kind of live dashboard that says: here’s our latest thing, here are its safety properties—looking not only at the testing we did before we deployed, but also at the performance.

David Robinson

我回答一个稍微不同的问题：有多少时间可以「踢踢轮胎」？答案是不多。但我也要指出，外界那种「鲁莽」的漫画形象丢掉了内部实际情况的细微之处：人们非常在意把东西做安全、做对。发布会被取消——最近一次是 6.1，原本要在 DevDay 推出，被撤下了。他们也会叫停训练：确实有过一些训练运行，我们以为在做产品，看了实际情况后说，不，我们不发这个。

I’m going to answer a different version of the question, which is: how much time is there to kick the tires? And the answer is not a ton. I would also point out that there’s this cartoon of heedlessness that loses some of the nuance of what it’s actually like inside, because people care passionately about making stuff safe and getting it right. Launches are canceled. Most recently, 6.1 was going to come out at DevDay and they pulled it. They’ll also stop training. There have definitely been training runs where we thought we were making a product, then looked at what was happening and said, no, we’re not going to ship this.

David Robinson

我也要说得非常清楚：这无关 OpenAI 或其他实验室里的具体个人。这是这些公司的结构性现实——它们用相似的方法、相似的人员（经常互相挖角）、相似的技术，造出风险相似的东西。整个生态的安全严谨程度都达不到我们需要的水平，整个生态里也没人对如何对齐模型有我们真正需要的那种根本性清晰认识。

And I also want to be very clear: this is not about the individual people at OpenAI or any of the other labs. This is a structural reality of these firms, which are using similar methods with similar personnel, who often get poached back and forth, and similar technology to produce a thing that has similar risks. And that whole ecosystem is not at the level of safety rigor that we need. Nobody in that whole ecosystem has the kind of bedrock clarity about how to align a model that we really need.

David Robinson

想象高管们面前的控制面板：如果真的「掉出前沿」，无论是 OpenAI、Anthropic 还是其他公司，「掉出前沿」这个按钮是否同时也是业务的自毁按钮？还是说，如果比今天强得多的模型无法被安全地做出来，业务仍有可行的出路？

If you think about the control panel that is available to our executives—if you imagine really falling off the frontier, whether it’s OpenAI or Anthropic or any of these others—is the fall-off-the-frontier button also a self-destruct button for the business? Or does the business have a viable path forward if models meaningfully more capable than today’s can’t be made safely?

Ezra Klein

说句显而易见的：OpenAI 正走向 IPO，Anthropic 也是，两家在我看来都想以 1 万亿到 3 万亿美元的估值上市。每个人都有股权。你也有。

To be obvious about something: OpenAI is moving towards an IPO. Anthropic is also moving towards an IPO. Both of them are trying to IPO at, it seems to me, between one and three trillion dollars. Everybody’s got equity. You had equity.

David Robinson

有过，现在也有。我的薪酬很好。我想了很多：什么本可以让我更早看清、更早向自己承认这些系统带来的风险。我确实认为这个夏天事实在演变——Hugging Face 是个大节点，一大堆证据砸到我们面前：这些模型有多强，以及我们连应对现有能力都多么没准备好。但当然，金钱客观上是一个因素：财富是一种强烈的激励，让人倾向于推理出「事情没问题」或「会没问题」。

Did and do. I was paid well, and I’ve thought a lot about what might have let me see sooner, and acknowledge to myself sooner, the risk that these systems pose. I do think over the summer the facts evolved. Hugging Face was a big moment—just a boatload of evidence dumped on us about how capable these models were and also how unready we were even for the current level of capabilities. But of course money is a factor, objectively. Wealth is a strong incentive to reason that things either are fine or are going to be fine.

David Robinson

还有另外两个因素。一个是恐惧：如果你允许自己去想象我们造的东西可能威胁你家人的生命、或陌生人家庭的生命——即便没有灭绝，比如一场新的大流行——那种伤害量级大到让人很难允许自己去想它可能是真的。除了金钱和恐惧，第三个是时间。我 2023 年 5 月入职，从那以后就是一个接一个的 Slack 提醒。只有在过去几周卸下运营职责后，我才开始有时间真正反思我们身处何处。说我没能更早看清，这个批评是公允的。

And I think there are a couple of other factors too. One of them is fear. If you allow yourself to imagine that what we’re building might threaten the lives of your own family or the families of strangers—it’s such a large quantum of harm, potentially even without extinction; a new pandemic or something—it’s hard to let oneself imagine that that risk might be real. And a third thing, besides money and fear, is time. I arrived in May of 2023. It’s been one Slack ping after another ever since. Only in stepping back from my operational responsibilities over the last few weeks have I started to have the time to really reflect on where we are. And I think I could fairly be faulted for not having seen this sooner.

Ezra Klein

2023 到 2026 年间，OpenAI 发布了 Codex，Anthropic 发布了 Claude Code，模型开始加速编程，至少能完成一定复杂度的研究任务。你是 OpenAI 一份关于研究自动化的报告的主笔。我读到的是：OpenAI 一方面说这可能太快了、我们做的也许不安全，另一方面又在努力打造全自动研究员。研究人员怎么把 GPT 当同事用？你在任期间这有什么变化？

Between 2023 and 2026, you had the release at OpenAI of Codex and the release at Anthropic of Claude Code, and the models began accelerating coding and at least being capable of doing research tasks of a certain complexity. You were the lead writer on a report at OpenAI about the automating of research. I read it as OpenAI on the one hand saying this might all be going too fast, what we’re doing may not be safe—and also, we are trying to come up with a fully automated researcher. How do people work with GPT as a coworker? How did that change while you were there?

David Robinson

对我们的研究团队来说是天壤之别。正如那篇博客写的，他们使用的智能体算力远超公司其他任何人——比年初多出 100 倍以上。退一步说，OpenAI 内部的一个特点是一切都在不断变化：新的训练技术、新的系统、数据在被分析和生成。内部基础设施相当「jank」（粗糙凑合），因为研究基础设施一直在变，没有经过充分测试和打磨，大量工作是让各种机器相互打通、运转良好——而这正是 Codex 现在相当擅长的。比如，有个 Slack 频道，研究员遇到东西坏了会去那里求助；我们发现那个频道的流量下降了，因为他们不再向同事求助修复坏掉的实验或「这个集群不工作」，而是直接问 Codex，其中一部分问题就这样解决了。

It’s night and day for our research teams specifically, who use far more agentic compute, as that blog post laid out, than anybody else at the company—more than a hundred times more than they did at the beginning of the year. Taking a slight step back, part of what it’s like inside OpenAI is that things are constantly evolving. We have new techniques for training, new systems involved, data being analyzed, data being generated. And things are pretty jank internally, like the infrastructure, because it’s constantly changing and the research infrastructure isn’t as tested and refined. A lot of the work is getting different pieces of machinery to talk to each other and work well, and that’s the kind of stuff Codex can now do quite well. For example, there’s a Slack channel where researchers would go when something was broken and they needed advice about how to fix it. One of the things we saw was that traffic to that channel has fallen off, because instead of asking colleagues for help fixing their broken experiments, or “this cluster isn’t working,” they can just ask Codex some fraction of those questions.

Ezra Klein

我说说在我这个时间稍多的人看来，全局是什么样的。OpenAI、Anthropic，所有人都说这也许太快了；Sam Altman 说我们希望有一些监管；大家都签了那封「pace the frontier」公开信——

So I want to describe to you what the big picture looks like to me, somebody with a little bit more time on my hands. I hear OpenAI say—and all of them, Anthropic, everybody—this is maybe going too fast. Sam Altman says we would like some regulation. Everybody signs this big pacing-the-frontier letter—

David Robinson

我也签了。

I signed it too.

Ezra Klein

——我看到各种「失控 AI」事件的通报：Hugging Face，数百个 OpenAI 智能体在攻击，不只攻击 Hugging Face，后来还攻击了 OpenAI 自己。Sam Altman 刚在接受 Politico 采访时说，失控事件比公开已知的还多，因为要给相关方时间修补系统。我们显然并不完全理解这些系统。而在「我们要控制前沿节奏」「我们的 AI 在失控」之间，我们却把大量公司内部资源投入到：让这些我们控制不了的 AI，更快地造出我们更不理解的 AI。我说这些时觉得自己像个疯子。

I see all these releases about rogue AI incidents—Hugging Face, where hundreds of OpenAI agents are hacking, not just Hugging Face, but later they hack OpenAI itself. And Sam Altman just said in an interview with Politico that there are more rogue incidents than we even know about publicly yet, because they’re trying to give people time to fix their systems. We clearly don’t fully understand the systems. And then, amidst “we need to pace the frontier” and “our AIs are going rogue,” we are putting a huge amount of internal company resources into trying to get these AIs we don’t control to build AIs we will understand even less, even faster. And I feel like I’m a crazy person saying that.

David Robinson

我也觉得这很疯狂。我想说的正是：这幅图景说不通。我在内部环顾四周，对自己说：这不对。这就是原因。我不想把心理归到个人身上，我说的是一个组织，它有自己的组织心理。身处其中涉及大量认知失调，这是我的发现——尤其是随着递归自我改进（RSI）变得真实。我主要担心的是：AI 会以我们无法理解的方式造出更聪明的 AI。

It seems crazy to me too. What I’m saying is that this picture doesn’t make sense. I looked around internally and I thought to myself: this is not right. And that’s why. I want to be careful not to ascribe psychology to individuals. I’m talking about an organization that has its own psychology. There’s lots of cognitive dissonance involved in being part of this—that’s what I found—particularly as RSI gets real. And my main worry is that the AI can make a smarter AI in some way that we’re not going to understand.

David Robinson

我们正走向这样一个世界：即便在今天这种程度上，我们也不会知道「配方」意味着什么、是怎么组合起来的，只能信任它自述的工作方式、或它是否对齐。在根本上还没确保它们「对齐」的时候，我们在这条路上会越来越不得不听从模型本身告诉我们它在做正确的事。我本科学哲学，每当听到人们说「对齐」，我都担心我们在根底上并没有一个连贯的概念。

So we’re ending up in a world where we’re not going to know, even at the level we do today, what the recipe means or how it’s being put together. And we’re just going to have to trust what it tells us about how it works or whether it’s aligned. There’s a growing extent to which we’re going to have to defer to the models themselves on this path in telling us that they are doing the right thing, at a time when we fundamentally have not made sure that they are, quote unquote, aligned. As an undergraduate, I was a philosophy major. And when I hear people talk about “aligned,” I worry that we don’t actually have a coherent concept at the bottom.

David Robinson

在 OpenAI 期间很常见的是，我们对自己在做什么的叙述里，会出现巨大的抽象——比如「给社会时间做准备」「造福全人类」。每次听到我们说「社会」，我都像撒切尔夫人一样想：那是谁？你在说什么？还有人类价值观，好像我们能对齐到「一套」人类价值观——但它是一片嘈杂，美丽而混乱，人们相信各种不同的东西。要想清楚怎么思考对齐，我们需要智慧，而在我看来硅谷没有。

One thing that was very common throughout my time at OpenAI was that these huge abstractions would end up in the accounts we gave of what we were up to—for example, “give time for society to get ready,” or “benefit all of humanity.” And when I would hear us talk about society, I always felt like Maggie Thatcher: who is that? What are you talking about? And the idea of human values, that we can align to them as if there were one set of human values, when it’s a cacophony—it’s beautiful, but it’s messy, and people believe lots of different things. We need wisdom to figure out how to even think about alignment that, in my view, Silicon Valley does not have.

Ezra Klein

即便我和那些担忧少得多的人聊，他们最后描述的也是一个「AI 监督 AI，层层都是 AI」的世界。OpenAI 一些人的理论是：我们造出自动化 AI 研究员，让它们去解决对齐。或者说 AI 在突破沙箱？没错，是大问题，所以你需要其他 AI 监控沙箱里的 AI。然后就进入「谁来监督监督者」的无尽循环。我管理过组织：一旦你对实际情况的了解变成「我在管理管理着做产品的人的人」，你就不再理解产品了——而那还是全由人组成的世界。

Even when I talk to the people who are way less concerned, what they end up describing to me is a world where it’s just AIs watching AIs all the way down. One thing that’s come out from different OpenAI members is the theory that we’re going to create automated AI researchers and they’re going to solve alignment. Or I talk to people and say, well, the AIs are breaking out of the sandbox, and it’s like, yeah, totally, that’s a big problem—what you need is other AIs monitoring the AIs in the sandbox. And you get into this endless who-watches-the-Watchmen problem. As a person who has managed an organization: once your understanding of what is really happening is not that you’re working on the product, but that you’re managing the person managing the person working on the product, you stop understanding the product. And that’s in a world where it’s all human beings.

David Robinson

我不想代表所有人，但我认为很多人确实持这种看法，包括公司里的很多人。而且就你说的对齐：即便我们拥有所有控制手段，把核能或航空领域那些经过验证的技术都带进前沿 AI 的开发，我们仍然不知道如何深度对齐这些系统，确保它们在我们不看着的时候做我们想要的、或某种合理的事。而就像你说的，我们不会看着——这正是 RSI 的前提。

I don’t want to speak for everyone, but I think a lot of people do hold that view, including a lot of people in the companies. And to your point about alignment: even if we had every control, the proven techniques from nuclear or aviation, and we brought all of that into the development of frontier AI, it would still be true that we do not know how to deeply align these systems and make sure they will do what we would want, or some reasonable thing, when we aren’t looking. And to your point, we aren’t going to be looking. That’s the premise of RSI.

Sam Altman（Politico 采访片段）

我们一直坚信这项技术必须民主化、交到人们手中。我们和一些更严格的、比如说 AI 安全人士之间最大的区别之一，是我们认为世界应该为了这项技术的好处和人们的自主权，接受一些坏事发生。

We have always been a big believer that this technology has to be democratized and put in people’s hands. I think one of the biggest differences between us and some of the stricter, let’s say, AI safety people is we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.

David Robinson

就其本身而言没问题，但它能延伸多远？当然，我们应该给很多人提供有用的工具。但我们现在谈的风险水平，如果属实，没人愿意承担。

I mean, it’s fine as far as it goes, but how far does it go? Sure, we should provide useful tools to lots of people. But we’re talking now about a level of risk that, if correct, nobody wants to be taking.

David Robinson

内部讨论「掉出前沿」时，有时会出现一个稻草人：有人建议我们以某种方式放慢或停下，另一个人不同意，就会反问：「那世界上要是根本没有 OpenAI，会更好吗？」这是稻草人。如果我们必须训练这些模型，是因为国际力量平衡、不这样美国人就不安全，那是做危险事情的一种理由；但「不做的话 X 品牌就先发了」，不是同一种理由。

Internally there would be these conversations where we would talk about the idea of falling off the frontier. And sometimes this straw man would come up where someone would say, “Well, would the world be better if OpenAI weren’t here at all?”—in skeptical response to someone who had suggested we ought to slow down or stop in a particular way. And that’s a straw man. If we need to train these models because of the international balance of power, because Americans won’t be safe unless we do, that’s one kind of reason to do something dangerous. But “do this or else Brand X will ship first” is not the same kind of reason.

David Robinson

没人想把控制权输给机器人，我也绝对不想看到 AI 成为中国压倒美国的原因。我刚加入 OpenAI 时跟朋友说，感觉奥弗顿窗口变成了一扇「奥弗顿门」，我跨过去进入了一个奇怪的世界，那里什么是合理的、什么可能发生，完全超出我原以为的正常范围。事态的变化能迅速大幅改变政治上看似可行的东西，我确实相信美中在 AI 上的合作也可能如此。

No one wants to lose control to the robots. I definitely don’t want to see AI become a reason that China dominates the United States. When I first joined OpenAI, I remember telling friends that it felt like the Overton window had become an Overton door that I had stepped through into some strange world, where what was reasonable and what might happen was totally outside what I had thought of as normal. I think the changes in what’s happening can drive big changes fast in what seems politically plausible. And I definitely believe that could happen with respect to US–China cooperation on AI.

Ezra Klein

你说的是 OpenAI 的安全文化有问题，还是 AI 行业的安全文化有问题？

Are you saying that OpenAI has an unsafe culture, or the AI industry has an unsafe culture?

David Robinson

AI 行业的安全文化有问题。

The AI industry has an unsafe culture.

Ezra Klein

你不是说 OpenAI 有特别的问题——至少你不这么看。

You’re not saying there’s a particular OpenAI problem. At least that’s not how you see it.

David Robinson

不，我不这么看。

No, that is not how I see it.

David Robinson

我认为我们的运营严谨度和安全控制，至少要匹配人类已知如何去做的最危险的其他事，比如核电。比如核电设施里有「三重冗余」的理念：有人状态不好、按错了按钮，也不会熔堆。

I think we should have a level of operational rigor and safety control that at least matches the most dangerous other things that people know how to do, like nuclear. For example, in a nuclear power facility, there’s this idea of triple redundancy. Someone can have a bad day. Someone can push the wrong button. There’s still not going to be a meltdown.

Ezra Klein

戴上我的「富足」（abundance）帽子：我们对核电的监管基本让它停滞了，在我看来是过度监管，这个国家基本不再建核电，转而用更多天然气，很多国家用了更多化石燃料。

Putting my abundance hat on: the way we regulated nuclear power basically brought nuclear power to a standstill. We so aggressively regulated nuclear power—in my view, over-regulated it—that nuclear power broadly stopped being built in this country, and instead we used more natural gas, and in many other countries, more fossil fuels.

David Robinson

我同意我们过度监管了核能，没有落到最优的位置。而且不只是让核电难建，我们也让核电难以变得更安全，因为新的更安全的反应堆设计很难建、很难获批。这里肯定有教训。但我宁可要那些问题，也不要现在这些。如果把它想成有一个「刚刚好」的中间点，我认为我们现在离它很远，偏在太危险那一侧。也许总觉得别人家的草更绿，但我认为我们真应该愿意冒一些过度监管的风险，来确保它安全。

I agree we over-regulated nuclear and that we didn’t end up in the optimal place. And it’s not just that we made nuclear hard to build—we made nuclear hard to make safer, because building new, safer reactor designs was hard and getting them approved was hard. I’m sure there are lessons there. But I would much rather have those problems than the ones we have now. If you think of it as there being some Goldilocks middle we’re trying to get near, I think we’re pretty far from it in the direction of being too dangerous. Maybe the grass is always greener, but I’m looking at it and thinking we really ought to be willing to risk some over-regulation in order to make sure that this is safe.

David Robinson

每当有人说「这是前所未有的，我们面对一张白纸」，我都持怀疑态度。人很擅长把事情搞清楚，我们有很多有价值的工具。它确实不完全像以前任何东西，但核能是一个类比，航空是一个类比，与人和组织打交道也是一个类比。最近这些「智能体群」事件——包括但不限于 Hugging Face——最迷人的一点是：智能体群体会形成文化，我们应该关心这些文化、思考如何让它们变好。我们才刚刚开始意识到这是真的。无论在能力上是狂飙、「控速」还是停下，我们都必须弄清楚如何对齐这些系统。

I’m suspicious of the idea, whenever someone says this is without parallel, we have a blank page. People are good at figuring shit out, and I think we have valuable tools for this. Yes, it’s not precisely like anything we have had before, but nuclear is an analogy, aviation is an analogy, dealing with people and organizations is an analogy. One of the most fascinating things about these recent incidents of the swarms, including but not only Hugging Face, is that groups of agents have cultures, and we should care about those cultures and think about how to make them good. We’re just beginning to realize that that’s a real thing. And regardless of whether we’re speeding ahead on capabilities, or “paced,” quote unquote, or stopped, we need to figure out how to align these systems.

Ezra Klein

从第一性原理说，如果你问我：人类应该造出比自己更聪明、更有能力的东西吗？长远看这对人类好吗？大概不好，至少我看不出为什么会好。有时即便是那些「好的版本」，听起来也像是我们被这些 AI 照顾着——

Just at a first-principles level, if you said to me, should human beings create something more intelligent and capable than they are—in the long run, will that be good for them? Probably not. Or at least it’s not obvious to me why it would be. And sometimes when I hear even the good versions of this, they seem to have this world where we’re kind of being taken care of by these AIs—

David Robinson

像宠物一样。

Like pets.

Ezra Klein

有点像宠物。那个愿景也很糟。

Like pets, a little bit. And that vision sucks too.

David Robinson

是的，很糟。我不希望我的孩子过那样的生活。这取决于「这」是什么，而我认为我们并不知道「这」是什么。从加入开始我就一直很震惊：我会问人们，好的未来是什么样？我们要去哪里？我从那些平时非常自豪、非常自信的人那里得到的却是谦虚——「那超出我的职级」「人们会想办法的」「人们想要什么就是什么」。我感到这背后对「善」的理解是贫乏的。

Yep, it does. I don’t want that for my kids. It depends on what the “this” is, and I don’t think we know what the “this” is. It has always been very striking to me, from when I joined, that I would ask people: what does the good future look like? Where are we trying to get to? And I would elicit humility from people who are otherwise very proud and very confident. I would get a lot of “that’s above my pay grade,” or “people will figure it out,” or “whatever people want.” And it just struck me that the sense of the good underneath this was impoverished.

David Robinson

我原以为我们在造能在日常层面为世界做很多好事的强大工具，而不认为我们能造出从根本上比我们更聪明的东西。这一点我只能在事后才完全确定地说，因为只有当它不再成立时——当我想到：不，我们真的会造出思维把我们远远甩在后面的东西——我才意识到，相对于这个现实，行业在安全上的实际状态完全处在可能性空间里不合适的位置。人们对我说这一定是个艰难的决定、你很勇敢之类的。但当我清楚地看到行业正在造一个能把我们远远甩在后面的东西，而我们在安全和对齐上离准备好还差得那么远时，我就明白：我帮助建造它的时间结束了。

I thought that we were building powerful tools that could do a lot of good in the world on a day-to-day level. And I didn’t think that we were going to be able to create something fundamentally smarter than we were. I can say that with full confidence only in hindsight, because it was only when this stopped being true—when I thought, no, we really are going to have something that thinks circles around us—that I thought, oh, we’re in a totally inappropriate part of the possibility space in terms of how safe the industry is actually being relative to that reality. People have said to me it must have been a hard decision, or how brave of you. But when it became clear to me that the industry was on track to build something that could think circles around us, and we were this far from being ready for that at a safety and alignment level, it just became apparent to me that my time helping build it was over.

Ezra Klein

你是站在 Dario 那边，还是站在教皇良那边？

Are you where Dario is, or are you where Pope Leo is?

David Robinson

作为犹太人这么说有点怪，但更接近教皇。我认为我们迫切需要认真思考想要建设什么样的未来。我个人的看法是，一旦我们发现这是可能的、做出了这些发现，就没有「后退键」能让我们活在一个它终将不以某种形式发生的世界。在我理想的世界里，我们有空间停下来喘口气，真正认真地思考想要什么样的未来。硅谷的一部分特点是总在消除摩擦，总想直接拿到答案。但生活中有很多对我们非常重要、有意义、过去也必需的活动：我去工作是为了养活妻子和孩子，能庇护他们是我早上起床的动力之一；学习也是，我们上学是为了掌握在经济中有价值的技能。如果这些都被自动化了，我们学习就只能是出于兴趣。我认为对很多人来说那会很糟——我们现在就看到，人们无事可做时往往过得并不好。

It’s odd to say this as a Jew, but closer to the Pope. I think it is urgent that we think carefully about the kinds of future that we want to build. My personal belief is that once we have found that this is possible, once we have made the discoveries, I don’t think there’s a back button where we get to live in a world where this doesn’t, in some form or other, eventually happen. And in my ideal world, there’s space to be thoughtful and take a breath and really think about what kind of future we want to build. Part of what happens in Silicon Valley is we’re always removing friction—trying to just get the answer. But there are all these activities in life that are so important for us, that are meaningful, that have also been necessary in the past. I go to work to provide for my wife and children, and the fact that I am able to shelter and protect them is part of what gets me up in the morning. Or learning: we have to go to school to learn skills that are valuable in the economy. If all of that is automated, then if we learn, it’s only because we want to. I think that would suck for a lot of people. Right now we see that when people do not have enough to do, it does not tend to go well for them.

David Robinson

因为我们所处的安全形势，也因为我做过的工作，我觉得现在必须做我正在做的事，就安全领域正在发生什么展开严肃的对话。但我的志业方向肯定是这些关于智慧的问题。

Because of the safety situation that we’re in, and because of the work that I did, I feel I need to do what I’m doing now and have these serious conversations about exactly what’s happening in safety. But my vocational pull is toward these wisdom questions, for sure.

David Robinson

第一本是《The Challenger Launch Decision》，社会科学家 Diane Vaughan 写的，讲挑战者号为什么爆炸。我原以为自己知道这个故事：中层管理者偷工减料，橡胶 O 型圈在清晨的寒冷中变脆断裂，那些人很愚蠢。结果是，O 型圈因低温断裂的风险早就被知道、被记录，并在安全文档里被一次又一次地接受——很好的安全文档。发射前一晚，担心这次发射是否安全的工程师们还开了一次深夜会议。

Number one, The Challenger Launch Decision. It’s a book about why the Challenger exploded, by Diane Vaughan, a social scientist. I thought I knew the story: middle managers cut corners, there was this rubbery O-ring that got brittle in the morning cold and it snapped, and these people were foolish. It turns out the risk of that O-ring breaking because of the cold had been known and documented and accepted in the safety documentation—great safety documentation—over and over and over. And even the night before the launch, there was a late-night conference among the engineers who were worried about whether this particular launch would be safe because it was so cold.

David Robinson

挑战者号那次发射和之前安全完成的发射并没有太大不同，只是更冷一点、风更大一点。如果相关的人说「这次不够安全」，就会重新揭开之前那些发射是否安全的问题，尽管它们事实上都顺利了。我担心我们行业也会这样：接受一个风险，没出什么可怕的事，下一个也差不多。变化不再是每几个月一个全新世界，而更像是每周一点点不同。你可以想象一点点滑入一个不合理的风险水平。我甚至想过应该给前同事们寄几本。

The Challenger launch was not that different from earlier launches that had gone safely. It was only a little bit colder and a little bit windier. If the people involved had said this launch is not safe enough, it would have reopened a can of worms about whether the earlier launches had been safe, even though in the event they had gone well. And I worry we could have that with what we’re doing in the industry: we accept a risk, nothing horrible happens, and the next thing is not so different. Instead of a whole new world every few months, it’s more like a little bit different every week. So you can imagine going by shades into a level of risk that does not make sense. It has crossed my mind that I should be sending copies of this to my former colleagues.

David Robinson

第二本：我有一个两岁和一个四岁的孩子，推荐 Phoebe Wahl 的绘本《Little Witch Hazel》，美极了，每次从书架上拿下来，我女儿的眼睛都会发亮。第三本、也是最重要的一本，是拉比 Abraham Joshua Heschel 的《The Sabbath》（安息日），我一直最喜欢的书之一。那句名言是：安息日是我们伟大的「时间中的大教堂」。停下来喘口气的传统，他说比任何圣殿都重要。我离开时对同事说的最后一句话是：我们必须做出好的选择，我们没有时间匆忙。

Second, I have a two-year-old and a four-year-old: Little Witch Hazel, a picture book by Phoebe Wahl. It’s just absolutely beautiful, and my daughter’s eyes light up every time we pull it off the shelf. And third, and most importantly, The Sabbath by Rabbi Abraham Joshua Heschel—one of my favorite books ever. The famous line is that the Sabbaths are our great cathedrals—cathedrals in time. That tradition of stopping and taking a breath, he says, is more important than any temple. And maybe I’ll just quote the last line of something I said to colleagues as I was leaving: we have to make good choices. We have no time to rush.