Schulman／Millidge／O’Neill：我们离递归自我改进还有多远

Dwarkesh Patel

今天请来三位我每次聊天都能学到很多的 AI 研究者朋友——他们又碰巧在相对「开一点」的实验室和公司，所以能公开说。Beren Millidge，Zyphra CTO，做开源模型；John Schulman，Thinking Machines 首席科学家，前 OpenAI 联合创始人，主导了通向 ChatGPT 的 RLHF；Charlie O’Neill，Baseten 模型训练负责人。第一个问题：如果到 2036 年，世界上没有数以十亿计、彻底改造世界的疯狂超级智能，除了外生政治冲击、战争、禁 AI 之类，最可能的技术原因是什么？

Today, I’m chatting with three of my AI researcher friends from whom I learn a lot every time we talk. They also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I’m joined by Beren Millidge, CTO of Zyphra developing open source models; John Schulman, chief scientist at Thinking Machines, previously an OpenAI co-founder who led the RLHF work that led to ChatGPT; and Charlie O’Neill, head of model training at Baseten. First question: If we’re in 2036 and we don’t have billions of crazy superintelligences that have radically transformed the world, other than exogenous political shocks, war, or banning AI—what’s the most likely technical reason?

Beren Millidge

有点像经典的莫拉维克悖论：我们总觉得「AI 要是能做这个就了不起」——解难题、下赢棋——然后它做到了，冲击却没想象那么大。若这种模式一直延续，永远没有真正的泛化火花，AI 会对所有塞进基准和环境里的事极强，却仍有一道顽固的 sim-to-real 缺口卡住一切。我认为不太可能——我们其实已经在 RL 里看到这种泛化——但若元学习泛化极难、持续学习也解决不了，那会是我的默认情景。

There’s been a classic thing, almost like Moravec’s paradox: we think if AI can solve hard maths or win at chess it’ll be amazing—then it does, and it’s not that impactful. If that continues and there’s never the true spark of generalization, AI could be extremely good at everything put into a benchmark or environment, with a persistent sim-to-real gap somehow blocking everything. I think this is unlikely—we already see this kind of generalization from RL—but if meta-learning generalization is ridiculously hard and we don’t solve continual learning, that would be my default scenario.

John Schulman

同意。人类相对模型仍有很多优势；每出新模型会追上一部分，但你会卡在模型更弱、判断更差、自我核查不够的地方。循环反复出现：新模型一出大家惊呼「这就是 AGI」，用一个月又觉得它蠢。这种循环可能还会重复很多次，很难预测次数。眼下做研究和工程时瓶颈仍够多，模型写代码再多也难让你生产力涨 100 倍，所以爆炸式增长不会轻易出现——周期可能比我们预期更多。

I agree. Humans have a lot of advantages over models now. Each new model catches up in some areas, but you get bottlenecked where the model is weaker, has worse judgment, or can’t check itself well enough. The cycle keeps repeating: a new model comes out, people are blown away—“This is AGI”—then after a month it feels dumb. That cycle might keep going; it’s hard to predict how many times. Right now you don’t get explosive growth because you’re still bottlenecked enough in research and engineering. Even if the model writes way more code, it doesn’t make you 100X more productive. Maybe there are just more of these cycles than we’d expect.

Charlie O’Neill

对我来说关键是：芯片上「可能的最优学习者」离当下的 transformer + RL 配方有多远。人们想象一旦有个比所有人类都强一点——哪怕强 0.1%——的 AI 研究智能体，就能并行跑几十万甚至上百万个，芯片再加速，最终会压过一切瓶颈，自我改进起飞极快。可想想摩尔定律：直线好看，背后全是离散断点与创新才把曲线撑直。LLM 也一样：预训练缩放撞收益递减，RL 救了一把，又出现新的递减曲线，看起来仍像直线。若还需要再一次那种断点，我不确信当前用 RL 环境——即便专门盯着 RSI——就能发现它；发现不了，就会撞上渐近线。

For me it’s how far the global optimum of “a learner you could have on a chip” is from transformer + RL. People imagine that once you have an agent even 0.1% better than all humans at AI research, running hundreds of thousands or millions in parallel—faster as chips speed up—outweighs every other bottleneck and you hit very fast self-improvement takeoff. Think of Moore’s law: a nice straight line held a long time, but many discrete discontinuities and innovations kept it going. Same with LLMs: pre-training scaling hit diminishing returns; RL fixed that; then a new diminishing-returns curve kept the line looking straight. If it takes another discontinuity, I’m not sure training LLMs with RL environments—even RSI-targeted ones—would discover it. If not, we probably hit an asymptotic curve.

Dwarkesh Patel

那你觉得这个断点会比 2012 年以来的任何东西都更难吗？

Do you think the discontinuity will be harder than anything that’s come since 2012?

Charlie O’Neill

要是我们知道答案，大概就能实现了。要区分：是累加在当前范式上的断点——RL 之外还要再发现一点，也许它们能把直线上的点连起来；还是离全局最优太远，得扔梯度下降和神经网络。我不认为把当前范式无限放大，不管跑多少个 LLM，就一定能发现太远的那种东西。

If we had the answer, we’d kind of be able to implement it. Distinguish a discontinuity that adds to the current paradigm—cumulative, something beyond RL they might connect on that line—versus how far we are from the global optimum: do we throw out gradient descent and neural nets? I don’t think scaling the current paradigm, no matter how many LLMs you run, necessarily discovers that if it’s too far away.

Dwarkesh Patel

那唯一希望就成了：深度学习根本到不了至少压制人类研发、包括提出新范式的能力。可从 2012 到现在的进度往下推——哪怕主要靠算力缩放——若几年内仍压不住人类 R&D，也会很怪。Ryan Greenblatt 最近上节目说：AI 越强，越能把推进 AI 研发乃至一般科学的模拟环境做出来，各实验室和很多创业公司都在盯这个。另一个直觉泵是 80 年代以来象棋机器人的 Elo：随时间近似线性上升，但穿过人类区间时出现巨大断点——从人类专家总赢 AI，到人类专家再也赢不了。我同意目前 AI 能力对终端经济冲击还没那么大，正因为相对人类的 Elo 在慢慢爬。

The only hope really is if deep learning can’t get us to an AI that at least dominates human research and development, including coming up with new paradigms. But looking at progress since 2012—powered by huge compute scaling—it would be weird if it didn’t get there, at least in R&D, over the next few years. Ryan Greenblatt recently made the point that as AIs get more capable, they can make progress on simulations that incentivize getting better at AI R&D and science generally—labs and startups are targeting this. Another intuition pump: chess-bot Elo since the ’80s rises very linearly, but crossing the human range there’s a huge discontinuity—from human experts always beating AIs to never. I agree AI capabilities haven’t been that big for end economic impact yet, because they’re slowly rising in Elo relative to humans.

Beren Millidge

我会很惊讶若它不发生。唯一不发生的技术路径，是刚好在越过人类 Elo 之前渐近——我们其实已经很近了。你说的「2035 一切如常」情景，更可能靠的是剧烈监管，而不是技术卡死。

It would be very surprising. The only technical way is if it asymptotes just before we cross the human Elo score—we’re already pretty close in my opinion. In the scenario where we’re sitting in 2035 and everything is normal, the most likely path is dramatic regulation on AI, rather than a technical thing.

Charlie O’Neill

研究有不同种类。一种是 autoresearch 式：目标已经干净指定，你优化它——大家想象继续压低预训练 loss、抬高环境奖励就会进步。Ryan 说的也许是范式转移所需的那种开放式科学：目标说不清，AI 也说不清。我们给任何这类东西指定目标时都得极度小心。

There’s research in the autoresearch style where the objective is already specified cleanly and you’re optimizing it—everyone pictures continuing to drive pre-training loss down and environment reward up. Maybe what Ryan means is much more open-ended science needed for paradigm shifts, where we can’t specify the objective and AIs can’t either. We have to be really careful how we specify objectives.

Dwarkesh Patel

John，以你早年在一线的经验：一个大突破是意识到 next-token prediction 就是该优化的东西——2014 年你不会想到 nanoGPT speed run 是目标。现在有了这个范式，就会去做 speed run、让 AI 擅长它。但也许还有下一层内环 AI 预想不到；外环是收入之类，很慢。

John, with your wisdom from the trenches: a big breakthrough was realizing next-token prediction is what to optimize—you wouldn’t have thought the nanoGPT speed run was the thing in 2014. Now that we have this paradigm, you’d speed-run it and get AIs good at it. But maybe there’s a next inner loop AIs wouldn’t anticipate; there’s a slow outer loop of revenue or something.

John Schulman

早期 OpenAI 时我直觉是：只最小化 log loss 到不了智能——重要的那些 bit 在总 loss 里占比太小，会被噪声淹没，所以 next-token 语言模型学不到你真正在意的东西，需要更好的目标。你可以论证：人类大概也不学着建模环境里的一切——多数人没法对照所见场景做出照片级复现——所以必须有更好目标。结果它反正就工作了。

In the early OpenAI days I had the intuition that just minimizing log loss wasn’t going to get you to intelligence—the important bits are such a small fraction of the loss they’d be overwhelmed by noise. So next-token prediction alone wouldn’t learn the interesting things; we needed better objectives. You could argue humans don’t learn to model everything—most people can’t make a photorealistic reproduction of a scene—so we must need a better objective. Then it just worked anyway.

John Schulman

整个领域极度依赖泛化，很难预测何时会有泛化、何时有分布外泛化。我们知道：训你关心的任务就会更好。但最重要的进展常常是我们「没资格期待」的泛化——比如从天真的 next-token 预训练到需要深刻理解输入的各类任务，或学到预训练里很稀有的技能；再比如从可验证任务泛化到不太可验证的任务——先验上也没理由期待。

The whole field relies a lot on generalization and it’s very hard to predict when you’ll get generalization or out-of-distribution generalization. We know if you train on the task you care about you’ll do better. But the most important advances are often types of generalization we have no right to expect—from naive next-token pre-training to tasks that need deep understanding of the input, or rare skills lightly represented in pre-training; also generalization from verifiable tasks to less verifiable ones—no a priori reason to expect that either.

Dwarkesh Patel

有个关于快速奇点的直觉泵：在每次跑七位数实验之前，你先用等量算力做 AI 劳动——自动化版的你们先花一个世纪想最优实验、做小规模消融、甚至把深度学习之前的理论都补齐；实验后再花一个世纪分析、决定下一实验。

One intuition pump for a rapid singularity without scaling non-AI-labor inputs: before every 7-figure experiment, you spend equivalent compute on AI labor—automated versions of you spending a century thinking about the optimal experiment, small-scale ablations, a century of theory even before deep learning; then a century of thinking after, analyzing what happened and what to run next.

John Schulman

想得够深，有些事事先就能料到。大概有很聪明的小规模实验能建起可泛化到大规模的理论。我预期我们离「研究能做得多好」的天花板还很远。未来 AI 会做大量分析与理论构建，花的算力可与实验本身相当。

If you think hard enough, you probably could have expected some of these beforehand. There’s probably some very clever small-scale experiment that builds theory that generalizes to the large-scale. I’d expect we’re nowhere near the ceiling of how well you can do research. I’d imagine a future where AI does a lot of analysis and theory-building, spending comparable compute to the experiments themselves.

Charlie O’Neill

目标定义清楚时有非常具体的例子。思考只能在你已有先验之后、用新得到的 bit 更新后验——光想不会变出新 bit。但目标清楚且数据就在那儿时，当前范式会有大幅提速。比如让 AI 想 Kaplan 缩放律：它现在会发现「他们拿了中间 checkpoint、没考虑 annealing，所以错了」——这种观察能早好几年抓住，砍掉一两年进度空转。muP、学习率如何随模型规模缩放、模型宽度的重要性，很多低垂果实都能抠出来。若目标就是「最大化我们当前这条目标」，我能想象约 10 倍提速。但我看不见这如何泛化到「一开始就提出正确目标」——光想买不到正确目标。

There are concrete examples when the objective is well specified. All thinking can do is update your posterior from bits since you formed your prior—you can’t gain new bits from just thinking. But when the objective is well specified and data is sitting around, I imagine a big speed-up in the current paradigm. Example: an AI thinking about the Kaplan scaling laws would notice they’d taken intermediate checkpoints without accounting for annealing, so it was wrong—caught years earlier; we’d cut off a year or two of progress. muP, how learning rate scales with model size, realizing width matters—you can back out a lot of low-hanging fruit. I’d imagine a 10x speed-up if the thing is just “maximize the objective we’re currently on.” But I don’t see how that generalizes to coming up with the right objective in the first place. Just thinking doesn’t buy you that.

Beren Millidge

这正是当前 AI 能否快速 RSI 的关键问题：AI 多擅长学习自己的目标？要有自推进的自动化环，AI 得提出目标、优化、搞清楚、再提新目标，并长时间不脱轨。回到莫拉维克：自主、自我封装——自己想该做什么再去做——我们觉得容易，因为我们一直这么干，进化也要造出能独自生存很久的生物；对 AI 这也许莫名很难，就像运动很难、数学却对我们极难却对它极容易。

This is really the key question for very rapid RSI from current AIs: how well can AIs generalize to learning their own objectives? For a self-propelling automated loop, the AI needs to propose objectives, optimize them, figure that out, propose a new objective, and not go off the rails for a long time. Back to Moravec: autonomy and being self-encapsulated—thinking what we should do and doing it—feels easy because we always do it; evolution needs creatures that survive alone a long time. That might be really hard for AI for some reason, the way locomotion is hard but math is easy despite being hard for us.

Dwarkesh Patel

可时间视野在拉长，不就暗示……

But doesn’t the time horizon increasing suggest that—

Beren Millidge

对，这是另一种可能。没有明显证据支持「自主很难」；事实上智能体已经很持久、这件事相当容易，这是反证。但若真难，这会是我们得不到立即起飞的原因之一。

Yeah, exactly. Another possibility—but there’s no obvious evidence for this. In fact agents are now super persistent and it’s quite easy, which is evidence against. But this could be one reason we don’t get immediate takeoff, if it’s hard.

Dwarkesh Patel

从 2012——或从你们开始做研究——到现在，所有创新里，哪一类会是人类在 AI 彻底自动化 AI 研发之前最后还得亲手做的事？

From 2012—or when you started research—till now, what part of the innovations seems like the last thing humans would have to do before AI totally automates AI R&D?

Beren Millidge

大概是迭代地问对问题。就算 AI 能做任何实验，你仍要决定做哪些。相对写实验代码，AI 现在还不擅长这个；一聊研究它们就提一堆极小的杂项步骤。

Probably just iteratively asking the right questions. If the AI can do any experiment, you still need to decide which. Right now AIs aren’t very good at this compared to coding the experiment. Whenever we talk about research, they propose miscellaneous tiny steps.

Charlie O’Neill

甚至从 DeepMind「用超人类游戏解决智能」转到某个像 Radford 的研究员「预测海量数据的下一 token」……即便 Radford 发现了，也过了很久才有人决定 scale——因为还得发明缩放律、以及你可以非常可靠地预测这些东西。

Or even going from DeepMind’s “solve intelligence by learning to play games at a superhuman level” to one random researcher like Radford predicting the next token of a very wide swath of data… Even once Radford discovered that, it took a while before people scaled it, because we had to come up with scaling laws and that you could reliably predict these things.

John Schulman

我会说人类持续最久的角色是定义目标、决定我们真正想要什么。比如决定助手该如何表现、什么叫有帮助、RLHF 时目标是什么；再往后是定义 constitution 和 model spec。即便 AI 能干完所有技术活，我们仍得大量做这些。对齐某种意义上是答案——但对齐可拆成：规格／搞清正确目标，以及真正优化你定义的目标。前者短期内不会消失。想想后训练团队为什么要很多人：有太多领域要搞清模型该如何表现，很难整块自动化。

I’d say the last job for humans that’ll last longest is defining the objective and deciding what we actually want—how assistants should behave, what helpful means, the objective in RLHF; later, constitutions and model specs. Even if AIs do all the technical work, we’ll still do a lot of that. Alignment is sort of the answer—but alignment decomposes into specifying/figuring out the right objective, then achieving/optimizing it. The first won’t go away soon. Why you need many people on a post-training team: many areas where someone has to think how the model should behave. Hard to automate the whole thing.

Dwarkesh Patel

为什么模型提供商没有巨大整合？很多因素都指向集中化。往回看几年，有什么会阻止它？

What’s the story for why there isn’t huge consolidation in model providers? So many things point to centralization. Stepping back over years, is something going to prevent that?

John Schulman

我认为蒸馏是对抗集中力量的主因。凡能通过 RL 学到的，都能很容易蒸馏——bit 数少，少量数据就能学；只要能拿到展示该行为的轨迹，就容易蒸馏。还有可能出现公司专用模型：从部署中学习、持续改进自家模型——可由当前寡头或更小公司提供——那会改变游戏。

Distillation is the main thing that fights the centralizing force. Anything learned through RL can be distilled very easily—a small number of bits, learnable from a small amount of data. If you get trajectories showing a behavior, you can easily distill it. There’s also a possibility of company-specific models that learn from deployment and continually improve—provided by the current oligopoly or a smaller company—which changes the game a bit.

Beren Millidge

持续学习也挡不住蒸馏。模型天天进步，别人也能天天蒸馏；两边环可以同速运转。

Continual learning doesn’t stop distillation. Even if your model improves every day, people could distill it every day. The loops could operate at the same pace.

Dwarkesh Patel

那复制模型行为——你是否得自己知道用什么正确分布去 prompt，才能引出相关行为？

So copying model behavior—do you need to know yourself what the right distribution to prompt is to get the relevant behavior?

John Schulman

对。纯用监督学习蒸馏时，prompt 分布极端重要。即便你有完整访问权和思维链，也不平凡地把所有有用能力蒸馏出来——你得用真实、覆盖极广的 prompt 去问。最近冒出来的一件事：有些中国公司大概在用那些让中国用户访问本来被墙的美国前沿模型的 router／代理服务，多半用于编程；这些服务在收集并出售部分数据。这是极有用的蒸馏数据集，因为它给了你完美的 prompt 分布。

Oh yeah. For supervised distillation, the prompt distribution is extremely important. It’s non-trivial to distill a model even with full access and chain of thought—you need to prompt it with realistic prompts across a really wide distribution. One thing coming out recently: some Chinese companies are probably using router or proxy services designed so people in China can use otherwise blocked US frontier models, mostly for coding. These services are collecting and selling some of the data. That’s a very useful distillation dataset because it gives you the perfect prompt distribution.

Beren Millidge

AI 在这儿帮很大。看前沿流水线或中国模型论文：它们从某处拿种子 prompt——人类加这类数据——再用现有模型或其他前沿模型合成巨大覆盖。prompt 分布收集与环境创建可以大量自动化；模型越好，人类需要提供的 bit 越少。

AIs help a lot here. If you look at frontier pipelines or Chinese models’ papers, they get seed prompts from somewhere—some mix of humans and this kind of data—then synthesize vast coverage with existing or other frontier models. You can automate an awful lot of prompt-distribution gathering and environment creation. Humans need to provide increasingly fewer bits as models get better.

Dwarkesh Patel

可你似乎仍卡在「得有一个服务，用户真的从中走过」。

But it still seems you’re bottlenecked by having a service with users going through it.

Beren Millidge

不一定。那当然很有用，但理论上你可以只是想用户要什么。最终若有完全自动化环——AI 决定数据、决定训练——那基本就是 RSI。取决于你还需要多少人类信息。到某一点你只要说「我要看起来像这样的轨迹」，模型就能给出不错的近似。

Not necessarily. That’s helpful, but theoretically you can just think about what users want. Ultimately if you have this fully automated loop—AI deciding data and training—that is basically RSI. It depends how much human information you need. At some point if you’re just like “I want traces that look like this,” you prompt that and the model comes up with a pretty good approximation.

Beren Millidge

讽刺的是，这对蒸馏方比前沿实验室更容易。蒸馏方说「我要一个好政客」，去找前沿模型——它已经知道怎么当好政客，直接生成轨迹。而要第一个造出这种模型，你得弄到政客每天干什么的数据并构建出来。说「我要像这样的东西」再让 AI 造十亿变体，比从头造出「像这样」容易得多。

Ironically this is easier for distillers than frontier labs. The distiller says “I want a good politician,” goes to the frontier model—it already knows how—and generates those traces. To build the first model that does this, you somehow get data on what politicians do every day and build that. It’s much easier to say “I want something like this” and get the AI to produce a billion variations than to create the thing in the first place.

Charlie O’Neill

基于「中国实验室有 router 数据」可以做一个很具体的预测。事情起因是我说：奇怪吧，Sonnet 5 和 Opus 5 几乎客观上比 GLM-5.3 和 Kimi K3 差，尽管它们不仅有蒸馏、还有来自 Mythos 的 logit 蒸馏。反方说 prompt 分布真的极其重要——你得看见用户在干什么，才能把行为蒸馏进去。由此预测：前沿实验室在 RL 环境上未必还有多少优势。用户分布对一般行为很重要，但能力最好的度量是你在前沿做出来的极难 RL 环境。若 Anthropic 既有那些环境又有 logit 蒸馏，却仍做出更差的模型，也许——

You can make a concrete prediction from Chinese labs having router data. What started this was me saying: isn’t it weird Sonnet 5 and Opus 5 are almost objectively worse than GLM-5.3 and Kimi K3, even with distillation and logit distillation from Mythos? The counter was that prompt distribution really matters—you need to see what users do to distill those behaviors. The prediction: frontier labs don’t necessarily have much advantage, if at all, in RL environments now. User distribution matters for general behavior, but the best measure of a capability is the very hard RL environments you’ve made at the frontier. If as Anthropic you have those environments and logit distillation and still made a worse model, then maybe—

Dwarkesh Patel

那就是真实世界部署比环境更重要。有意思。可他们得先在 Fable／前沿模型里激励出这些能力，奇怪他们不能再用更小模型再激励一遍。

Then real-world deployment matters more than the environment. That’s really interesting. But they had to incentivize those capabilities in the first place in Fable or the frontier model. Weird they can’t incentivize them again with a smaller model.

Charlie O’Neill

也许我们在诡异的 uncanny valley：学生－教师差距太大，抄得太像反而糟。有人拿 Opus 举例：Opus 4.6 和 Opus 5 的差别是，Opus 5 感觉处处有 AI-as-a-judge 在查它做过的每件事，所以 token 用得多——它试图想所有这些，却没有 Fable 那种「大模型气味」知道何时该停、哪条路值得走。

Maybe we’re in a weird uncanny valley where copying the frontier model too much—the student-teacher gap—is just too large. People have made this point with Opus: the difference between Opus 4.6 and Opus 5 is Opus 5 really feels like it’s got AI-as-a-judge checking every possible thing it’s done—that’s why it uses so many tokens. It tries to think about all these things but doesn’t have the big-model smell of Fable to know when to stop or when’s a good path.

John Schulman

我提一个略不同的假说。环境有几个轴：难度与真实感。造一堆更难、更绕、更拼聪明的环境相对容易——可以说这是 benchmaxxing 分布，因为很多显眼基准就是易验证的难谜题。真实感轴则是：多轮与人对谈、多目标的真实 coding agent 设定。首次塑造模型行为的实验室两个方向都要推；要好行为就得猛推真实感，用 rubric 或人类反馈来告知奖励。若天真蒸馏，你往往只在 benchmaxxing 分布上对齐老师；若缺少在棘手真实设定里真正锻炼能力的环境，学生就得不到那些能力。也许大模型从棘手窄任务向更真实任务泛化更好；若有很好的真实 prompt 分布做蒸馏，能很好对齐大模型；若只有易验证任务分布，基准都能对齐，更广分布上却更差。这甚至可能解释 Sonnet 5 这类较小 Anthropic 模型——不过很难精确猜他们后训练在干什么。也可能后训练栈一直在改，某几版调太高、搞出用户很讨厌的 quirks；后训练很容易以基准看不出来的方式搞砸。

I’d offer a slightly different hypothesis. Environments have axes: difficulty and realism. It’s comparatively easy to create lots of difficult environments—more complicated tasks, more cleverness; the benchmaxxing distribution, since many prominent benchmarks are hard puzzle-like easily verifiable tasks. Then realism: the realistic coding-agent setting with multiple back-and-forths and multiple objectives. Labs crafting behavior for the first time need to push both. Good behavior needs pushing the realism axis with rubrics or human feedback informing the reward. Naive distillation often just matches the teacher on the benchmaxxing distribution; without enough environments exercising capabilities in trickier realistic settings, you don’t get those into the student. Maybe big models generalize better from tricky narrow tasks to more realistic ones. With a really good realistic prompt distribution for distillation you can match the big model well; with only easily verifiable tasks you match on benchmarks but do worse on the broader distribution. That might even explain something about smaller Anthropic models like Sonnet 5—though it’s hard to predict exactly what they’re doing. It could also be they’re always changing the post-training stack and got a few things wrong—turned something up too high and created quirks people really don’t like. Easy to screw up post-training in ways that don’t show up in benchmarks.

Beren Millidge

还有很基本的一点：前沿实验室从大数据公司买数据，中国人也能从同样的数据公司买同样的数据——而且他们在买。很多人对此恼火，但数据一样就能蒸馏，要跟上其实相当容易。

One other very basic point: frontier AI labs buy all their data from big data companies. The Chinese can also just buy the same data—and they are. There’s a lot of annoyance about this, but if they have exactly the same data they can also distill. It’s quite easy to keep up.

Dwarkesh Patel

另一个问题：第一批能自动化 AI 研发的模型实际会怎么训？玩具版是 Ryan 说的那种：让 GPT-8 去造 GPT-3 规模、在内环挑战上很强的模型——打需要持续学习的游戏、用最少算力到某 loss 等等。John，你好像觉得实践中未必是那样。到真正能自动化 AI 研发时，它们大概怎么训？

How will the first models capable of automating AI R&D actually be trained? The toy version is what Ryan talked about: GPT-8 builds GPT-3-size models that are really good at inner-loop challenges—beating games that need continual learning, getting to a loss with least compute, etc. John, you had an interesting point that maybe that’s not how it happens in practice. By the point AIs can automate AI R&D, how are they probably trained?

John Schulman

大概会组合：用人类反馈吸收研究者的品味，再造大量涉及多步研究项目的练习环境。实践中两者都会做，每一轮修补上一轮最破的地方。研究者会大量用 AI，发现一致弱点，再用收集人类反馈或造环境来补。

We’ll probably do some combination of learning from human feedback to absorb researchers’ taste, and creating lots of practice environments involving multi-step research projects. People will do some combination and each iteration patch whatever seems most broken last time. Researchers will use the AIs a lot, notice consistent weaknesses, and patch those by collecting human feedback or creating environments.

Charlie O’Neill

有用的想是：我们把谱系回滚多远，再从那儿自对弈。极限是给它们一块 GPU、也许神经网络，说「想办法训模型做这些任务」。现状是我们贴在谱系最前沿：「Anthropic 过去几个月训练栈里发现的 bug，做成环境。」你得在前沿训、变强，于是锁死之前所有历史。也可以想象回滚到 GRPO 之前，造环境让它发现最好的 RL 形式，再往回滚……但我们仍如此算力瓶颈，人们会一直待在前沿，本质上是把自上一版以来的 bug 与改进 diff 成训练环境。

A useful way to think is how much of the lineage we roll back and then let self-play from there. In the limit you give them a GPU and maybe neural nets and say figure out how to train a model for these tasks. Currently we go to the very edge: here are bugs Anthropic found in their training stack last few months—turn those into environments. You need to train and get better on the frontier, so you lock in all previous lineage history. You could roll back before GRPO, with environments to discover the best form to RL models on, and roll further… But we’ll still be so compute-bottlenecked that people keep staying at the frontier, essentially diffing bugs and improvements since the last version into training environments.

Dwarkesh Patel

这对模型代际之间有非陈旧新数据也很好——基本上是实验室内的持续学习：把过去三个月 AI 研究进展，通过环境和 RLHF 类东西蒸回模型本身。

Which is also really good for non-stale new data between generations—basically continual learning within the lab, distilling the last three months of AI research progress through environments and RLHF-type stuff back into the model.

Charlie O’Neill

而且它就是蒸馏。也许这就是为什么有些人觉得渐近：你总在追过去三个月的进展。进展当然有 AI 贡献，但仍有人类在环。感觉你只是不断逼近人类研究者正在发现、能做的东西。

And it is distilling. Maybe that’s why some of us feel asymptotic: you’re always trying to get the last three months of progress. That progress is contributed to by AIs, but still has humans in the loop. It feels like you’re constantly inching closer to what human researchers are finding and capable of.

Beren Millidge

不过若只在轨迹上蒸馏，你永远超不过它。环境却可以远高于人类能做的：很容易设计没人解得了、AI 仍可尝试的环境。那才是走在人类 AI 研究前面的路径。

If you’re just distilling on trajectories you can never go above it. But environments can go quite far above what a human can do. It’s very easy to design an environment no human can solve that the AI can still try. That would be the path ahead of just human AI research.

Dwarkesh Patel

比如 nanochat speedrun，但比人类 speedrunner 还快。

Nanochat speedrun, but even faster than a human speedrunner.

Beren Millidge

AI 研究里特别容易定义目标。你可以说 loss 要到 1.3，现在没人做得到——极可测、可验证；AI 做到了就很好。

In AI research especially it’s very easy to define goals. You could say the loss needs to be 1.3 and no human can get that now—extremely measurable and verifiable. If the AI gets that, great.

Charlie O’Neill

一亿参数模型打赢 Minecraft 我们都说太简单了？五年前你会这么说吗？

Isn’t it crazy that 100 million parameter models beat Minecraft and we’re calling that too easy? Imagine saying that five years ago.

John Schulman

很多研究并不完全是在定义好的目标上爬山。更像是：我们对模型该更好的方向有直觉，对某个算法也有一点想法，于是设计任务去找这套方法的「生命迹象」；若有，再做越来越真实的版本。你不是直接优化最终关心的生产目标，而是在真实感轴上放松一点，找到真正管用的方法，方法成熟后再拉回真实感。还有更多面向解释与理论的研究——机器学习里很少有那么预测力强的数学理论，但有很多非正式理论。

A lot of research isn’t exactly hill-climbing on a well-defined goal. It’s more: here’s an intuition about how models should be better, and an idea for an algorithm that goes a little that way—come up with a task designed to show signs of life, then successively more realistic versions. You’re not directly optimizing the eventual practical objective; you relax realism a bit, find methods that work, then get back to realism after the method matures. There’s also research oriented toward explaining and developing theory. Often we don’t have mathematical theories in ML that are that predictive, but lots of informal theories.

Beren Millidge

模型大概会在所有这些任务的组合上训练：有的极易验证，有的 LLM-as-a-judge，或直接问人「这看起来合理吗？」希望它们能泛化到更难、更模糊的任务——大概会在一定程度上。是否泛化到足以让环在无人在环时自我封闭，不清楚。

Presumably models will be trained on some combination of all these—easily verifiable, LLM-as-a-judge, or ask the human “does this look reasonable?” The hope is these generalize to harder, vaguer, fuzzier tasks. It probably will to some extent. Whether it generalizes enough that the loop becomes self-sealing without humans at all is unclear.

Dwarkesh Patel

退一步。在我看来，往后 AI 研究的赌注是：在上百万种多样环境、上百种领域上放大 RLVR；另一头冒出来的智能体学会持久、分流信息与上下文，最终端到端优化、与其他智能体协作等。这样的智能体在上下文内样本效率很高——你们也研究过如何把 in-context learning 任意拉长——出来的东西基本上像一周或一个月的可插拔远程员工。第一，你们同意这是实验室在下的注吗？第二，够吗？基本上只在数据中心里的模拟里学这些元技能，部署到真实世界却不从真实部署学习？

Taking a step back: the bet going forward seems to be scaling RLVR across millions of diverse environments and hundreds of domains; what emerges is an agent that learned persistence, triaging information and context, eventually end-to-end optimization working with other agents. Such an agent is very sample-efficient in context—you’ve researched scaling in-context learning arbitrarily long—and what comes out functions like a drop-in remote worker over a week or month. First, agree that’s the bet labs are making? Second, is that enough—basically learning meta skills from simulated environments in the data center, then deployed to the real world without actually learning from real-world deployment?

Charlie O’Neill

现在很难拆开实验室有多少力气直接砸向 RSI，多少是在做能继续部署、收收入、资助下一轮大训练的一般智能模型。对后者，是的，大概就是这个注。过去几年环境往哪走模式很清楚。Anthropic 的环境谱系是清晰例子：先死磕编程，从编程拿到任务视野——编程大概是网上可造环境数据、以及内部可转成环境的东西里最低垂的果实——再泛化。下一步金融，RL 训练里塞海量 Excel；然后 PPT；再是工作经济的长尾。看起来很管用。很多其他实验室、甚至开源实验室，现在也意识到那是正确赌注。

It’s now hard to separate how much of labs’ effort goes to direct RSI versus generally intelligent models they deploy to collect revenue for the next big run. For the latter, yes, that’s probably the bet. The pattern of where environments are going over the last few years is very clear. Anthropic’s lineage is a clear example: first focus on coding and get really good; from the task horizon from coding—probably the lowest-hanging fruit for internet data to create environments, plus internal stuff—then generalize. Finance next, with so much Excel in RL training; then PowerPoints; the long tail of the working economy. That seemed to work really well. A lot of other labs, even open source, have realized that was the correct bet.

Dwarkesh Patel

含义是什么？我问 Dario：若你真期待模型上岗后能像人一样学习，为什么还要把做 PPT 这类技能烤进模型？一种解释是：我们预期很快到，但还没到，不如先把技能摊销进训练；另一种是：我们并不专注让它广泛部署的工作极强，只想它 RSI 极强——这只是赚钱倒回真正擅长 RSI 的模型；奇点另一头出来的东西会擅长当下模型的各种瓶颈。John，若路径是这种泛化，为何模型里仍有这么多任务特定知识？

What’s the implication? When I had Dario on, I asked: if you truly expect models human-like at learning on the job, why bake in PowerPoint skills? One explanation: we expect them soon but they’re not there yet, so amortize skills into training. Another: we’re not concentrated on widely deployed work—we want RSI; this is revenue to pour back into a model actually good at RSI; after the singularity what comes out is good at current bottlenecks. John, takes on why so much task-specific knowledge if the path is this generalization?

John Schulman

若模型上下文学习够强，理论上不必训金融——它们可以即时读所有书、搞清各司法辖区怎么做。也可以说需要大量领域特定训练只是为了更高效：即便聪明到能现场搞清，你仍可能想用大量 RL 把直觉烤进权重，让运行时更高效。实践中，提供商似乎在逐域加强最高价值领域。这也是模型变好很多的原因之一：覆盖了大量高价值域和最常见技能。

If models were good enough at learning in context, in theory you wouldn’t need to train them on finance—they’d read all the books on the fly and figure out how in the appropriate jurisdiction. You could argue you need lots of domain-specific training just for efficiency: even if smart enough on the fly, you might still want RL baking intuitions into the weights so the model is more efficient at runtime. In practice providers seem to go domain by domain strengthening the highest-value domains. That’s one answer to why models got so much better: they’ve covered a lot of high-value domains and the most common skills.

Beren Millidge

另一点是两边一起做并不贵。模型巨大，参数上扛得起学一切；很可能有迁移——即便金融不专属于 RSI，信息仍重要；如何判断什么重要、有品味、做长视野工作，这些元学习可能可泛化。世界上 RSI 数据也不多，难生成、费力。能从其他数据摊销一点迁移就好；反正算力和参数空间都很多，商业上还要卖模型，何乐而不为。

Another thing: it’s not that expensive to do both. Models are massive; they can afford in parameters to learn everything. There’s likely some transfer—even if finance isn’t specific, the information matters for RSI; general meta-learning of what’s important, taste, long-horizon work is potentially generalizable. There’s not that much RSI data in the world; hard to generate, lots of effort. If you can amortize other data you get some transfer. You already have masses of compute and parameter space, so why not—plus the commercial intent of selling a model.

John Schulman

还有一个问题：当前这种 sim-to-real——看真实任务、在数据中心造一堆可模拟环境再 RL——会不会永远是主导框架。它很成功，但弱点很多：很多事难模拟，尤其要实时和一堆人互动。

There’s one question whether this current paradigm of sim-to-real will be dominant forever. You look at real-world tasks, create environments simulable in the data center, do RL. Obviously very successful. But lots of weaknesses—many things hard to simulate, especially interacting with humans in real time.

Beren Millidge

样本效率低时，sim-to-real 必须主导——现在需要与人类成千上万次交互，没人会坐在 RL 训练环里。所以现在必须模拟才能拿够样本。若样本效率大幅提升，你会预期从部署学习变成更大一块。

Sim-to-real has to be dominant while sample efficiency is low, because right now you need thousands and thousands of interactions with humans. No human will sit in the loop of RL training. So we have to simulate to get the samples. If sample efficiency improves a lot, you’d expect learning from deployment to become a much bigger part.

John Schulman

也有别的做法：可以 off-policy 学习，拿走所有轨迹，不必重模拟一切，仍可能从中学到东西。

Though there are other things—you can learn off-policy, take all the traces, and even without resimulating everything potentially learn something from them.

Dwarkesh Patel

我想追问：怪的是大约一半算力花在推理上，却没直接帮模型变强。数字心智相对人类的一大优势，本该是：人不过五十年真实经验，模型通过所有实例能经历数百万年跨各类经济相关工作的部署。眼下那些数据并没有在有意义的意义上帮模型变好。最终模型显然应能从这些数据学习；一旦能，就会像广泛部署的智能爆炸——模型在吸收所有部署实例的信息。你预期这种「蜂巢心智」疯狂何时开始？

I want to ask more—it’s weird that ~50% of compute on inference isn’t directly helping the model get better. A key advantage you’d expect digital minds to have: unlike a human with 50 years of real-world experience, a model experiences through all its instances millions of years of deployment across economically relevant work. Right now that data isn’t meaningfully helping the model get better. Eventually models should learn from this; once they do, something that almost feels like a widely deployed intelligence explosion—assimilating so much across deployed instances. When do you expect this hive-mind crazy stuff to start?

Beren Millidge

很粗地说，这已经在发生——就在下一代模型里。你现在就能把部署数据放进未来模型的预训练或中训练，尤其若做过滤、判断、标注或合成。

Broadly, at a basic level, this is already happening… just in the next generation of models. Right now you can take deployment data and put it in the pre-train or mid-train of future models, especially with filtering, judgment, annotation, or synthesization.

Dwarkesh Patel

你觉得这解释了多少代际改进？

How much do you think that explains generation-over-generation improvement?

Beren Millidge

相当多。不知道实验室是否这么做——理论上他们声称不训用户数据——但中国人百分之百做，绝对拿这个优势。这基本上就是蒸馏：打模型、拿到一部分部署数据，训下一代；自己的模型当然也能做，没理由不做。

Quite a bit. I don’t know whether labs do this—theoretically they claim not to train on people’s data—but the Chinese 100% do. They definitely get this advantage. This is basically distillation: ping the model, get some fraction of deployment data, train the next generation. They can certainly do it on their own models too. No reason not to.

Charlie O’Neill

完全同意。拉远看，这绝对在发生。大家心里的持续学习圣杯是：单个模型拿到经验、当场 live 更新。缩到那个粒度很多东西会坏。但大实验室、闭源模型在做。开源模型也有更快节奏的早期生命迹象。好例子大概是 Composer；Harvey 对法律智能体也一样。你有某种模型，从该任务的数据、用户抱怨、从特定部署抠出的反馈里得到很具体的环境。很多这类公司相对大实验室的优势，是能把这些数据用得极好。然后造环境，对 Kimi K3 做大后训练，部署；也可能像 Composer 那样做一段时间在线学习——基本上是 REINFORCE。仍有人在环：人说我们关心哪些信号、怎么从数据造环境。节奏仍比你想的那种长，但确实在发生；环最终会越来越快。

Completely agree. Zoom out far enough and this is definitely happening. The holy grail of continual learning is an organic live loop of an individual model getting experience and live-updating on the spot. A lot breaks at that granularity. But big labs and closed models are doing this. There are early signs of life with open-source models at much faster cadence. A good example is probably Composer; Harvey’s doing the same with legal agents. You have a model and get very specific environments from data for that task, user complaints, feedback extracted from specific deployments. A lot of these companies have an advantage over big labs in using this data really well. Then they create environments, do a big post-train of Kimi K3, deploy; they might do some online learning too—like Composer did online… basically REINFORCE for a long time. Still a human in the loop saying which signals we care about and how to create environments from the data. Longer cadence than maybe you’re thinking, but it really is happening. Eventually that loop gets faster and faster.

Dwarkesh Patel

Composer 有意思：在 Cursor 里人对下一个补全按或不按 Tab，据此 Composer 每天都更擅长预测……

The Composer thing is interesting because in Cursor people press Tab or don’t on the next completion—and every day Composer gets better at predicting—

Charlie O’Neill

那是旧的 Tab 模型。他们对真正的生成模型其实也做了同样的事。难点是在线强化学习没有 group，只有一个用户说一件事、一次 rollout，方差削减很难。Cursor 的模糊答案是：我们有很好的启发式，能估计这个回复比平均好多少或差多少，然后做大的 REINFORCE 更新。解决「是否变差」的办法是：若 CursorBench 改善，就每五小时部署新模型；否则扔掉那一版。

That was the old Tab model. They actually did the same thing not just for Tab but for the actual generative model. It’s hard because with online RL you don’t have groups—one user saying one thing, one rollout—so a big variance-reduction problem. Cursor’s fuzzy answer: very good heuristics estimating how much better or worse than average this response was, then a big REINFORCE update. Their solution to whether it got worse: if it improved on CursorBench, deploy the new model every five hours; if not, throw that version out.

John Schulman

我觉得最大问题其实是不知道自然数据的奖励函数该是什么。若用表面信号——比如是否接受了编辑——可能以某种方式被奖励黑客。

I think your biggest problem is actually just not knowing what the reward function should be for natural data. If you use some superficial signal like did they accept the edit, that might get reward-hacked in some way.

Dwarkesh Patel

可这对 sim-to-real 不是更大问题吗——任务视野越长，越难在数据中心模拟？即便编程，我们也快到：没有什么长达一年的编程任务最终不需要跟客户、公司或用户说话。想想我们要 AI 能做的一切：最终超级智能应能经营公司、创业并盈利、做有利可图的日内交易、打赢官司——这些都极难在数据中心里模拟，学习的内在部分就是与真实世界互动。也许它们靠 sim-to-real 迁移变好；或者你需要这些互动上的权重更新。若迁移不够强、真需要权重更新，那模型相当样本低效也许是更深的问题。默认路径下，我看不见未来十年内如何躲过某种疯狂 RSI；可能不发生的一个理由是权重更新的样本效率上模型似乎远落后人类——从冷启动到训完相对人类从出生到成年，可能落后百万倍。所以：模拟与我们想要的极长视野、极复杂真实事之间会有好迁移吗？若没有，样本效率不足会不会真咬我们一口？

But isn’t this a bigger issue with sim-to-real—the longer-horizon tasks get, the harder to simulate in a data center? Even in coding we’re getting to the point where there’s not some year-long coding task that doesn’t eventually require talking to a client, company, or users. The gamut we want: eventually superintelligence should run a business, start a profitable new business, profitable day trading, win a court case—all very hard to simulate; inherent learning is interacting with the real world. Maybe they get better via sim-to-real transfer; alternatively maybe you need weight updates from these interactions. If transfer isn’t strong enough and you need weight updates, sample inefficiency may be a deeper problem. By default I don’t see how you don’t get crazy RSI within 10 years; one reason it might not is sample efficiency of weight updates—models seem way behind humans, plausibly a millionfold in data from birth to adulthood vs cold start to finished training. So: good transfer between simulations and extremely long-horizon complicated real stuff? If not, does lack of sample efficiency really bite us?

Charlie O’Neill

我会把模型会变强的任务和仍会挣扎的任务拆成：任务是否累积，还是非平稳分布、得不断学习、反复折腾。累积任务的例子也许是 RSI：理论上可能有不到一百万 token 的 Python 文件，从零训出能递归自我改进的模型。每个发现都是你守住的一条线。若 RSI 不需要再发现新注意力变体之类，那一旦发现注意力、MoE、GRPO，就加进训练栈，它就在那儿。好例子是 OpenAI 告诉我们训过的 5.6 Sol／5.6 Terra：它不必回去发现注意力，基本上会调一堆脚本，如 pre-training.sh、post-training.sh。那是累积任务。真实世界——人们为何这么想持续学习——并不真是累积任务。律所里智能体当法律助理，是很非平稳的分布：得在上下文里装下公司里所有重要人物之间不断变化的关系、各种隐性做法、信息在哪等。不像 RSI 那么干净的累积例子。会有这种任务分野。但若实验室意识到——且他们相信 RSI 是累积的、不必回去发现全新架构——也许越来越多力气和算力会砸向那一边。

I’d break down tasks models get good at vs keep struggling with: whether the task is cumulative, or a non-stationary distribution where you keep learning and relitigating. An example of cumulative might be RSI: theoretically possible to have a less-than-a-million-token Python file that from scratch trains a model capable of RSI. Every discovery is a line in the sand you hold. If for RSI we don’t need a new attention variant, once you’ve discovered attention, MoE, GRPO, you add that to the stack and it’s there. Good example: 5.6 Sol / 5.6 Terra training OpenAI told us about—didn’t have to rediscover attention; would have called scripts like pre-training.sh and post-training.sh. That’s cumulative. The real world—why people think so much about continual learning—isn’t really cumulative. A law-firm agent as legal associate is a very non-stationary distribution: fit in context all relationships among important people that keep changing, implicit ways things are done, where to find information. Not as clean a cumulative example as RSI. There will be this breakdown. But if labs realize—and believe RSI is cumulative in that we don’t need a brand-new architecture—maybe more and more effort and compute focus there versus other tasks.

Dwarkesh Patel

RSI 碰巧比当律师助理容易，真不幸。

It’s so unfortunate that RSI happened to be easier than being a paralegal.

John Schulman

今日模型在很多方面弱于人类。有些可能与某体制下的样本效率有关——有些体制里模型很样本高效，比如上下文学习；可能有中等长度体制更低效，因为人类能以比模型更高效的方式做某种权重更新。某些体制样本更低效可能是弱点来源之一。但还有完全不同的弱点：比如思维多样性低于人类，或不擅长某些长视野判断。人们说的「品味」很多是关于长期管用、人们已意识到长期管用的行为——尤其软件工程，很多品味是「哪些系统在这个项目长期可维护、跑得好」。多种弱点限制 RSI 及其他；有的关样本效率，有的不关。

Today’s models are weaker than humans in lots of ways. Some might relate to sample efficiency in certain regimes. In some regimes models are very sample-efficient—like learning in context. There might be a medium-length regime where they’re less, because humans can do some kind of weight update more efficiently. Being less sample-efficient in certain regimes might be one source of weakness. But there are other completely different sources—lower diversity of thought than humans, or being bad at certain long-horizon judgments. A lot of what people call taste is behavior that works in the long run that people have realized works in the long run—especially software engineering: taste as what systems will be maintainable and work well over the project. A variety of weaknesses limit RSI along with other things; some related to sample efficiency, some not.

Charlie O’Neill

有趣思想实验：假设你能给模型万亿 token 上下文，装下你在 RLHF 之前的全部经验，且样本效率和 in-context 学习能力与百万 token 时一样。品味就解决了吗？它能做出你当时同样的判断吗？还是除了更长上下文、同样样本效率之外，仍缺根本东西？

Interesting thought experiment: give a model a trillion-token context window—whatever you’d need to fit your experience prior to RLHF—with the same sample efficiency and in-context learning as at a million tokens. Is taste then solved? Would it make the same judgments you did? Or is something fundamentally missing apart from a longer context with the same sample efficiency?

John Schulman

它得被训成从那上下文学习——要么训成从上下文做出正确更新，要么得泛化。

It would have to be trained to learn from that context—either trained to learn the right update from that context, or it would have to generalize.

Beren Millidge

你仍需要在长上下文上训练它的数据。即便理论上能有万亿上下文，你也需要万亿长度的数据来训。现在你有 10k 上下文也不能直接塞一百万。理论上我认为可以。关键是品味能从多短的视野片段里被元学习出来。没有明显理由说必须超级长——人类没活到一万岁也很快发展出品味。想想 PhD：一年级到毕业或博后大概五年，总共也许只做了 10–30 个研究项目，却从相对短的一串小事里相当快地发展出品味。理论上可以那样发展。AI 显然会有远多得多的经验来发展、元学习品味。问题是那对真正长视野事情泛化得怎样——这点目前未解。

You still need data to train it on long context. Even if you theoretically get a trillion context, you need trillion lengths of data to train. Right now you have 10k context you can’t just dump in a million. In theory I think yes. It comes down to how meta-learnable taste is from shorter-horizon episodes. No obvious reason it’s super long—humans somehow developed taste without many long episodes; we don’t live to 10,000. Even a PhD: first-year to final-year or postdoc is maybe five years; they’ve only done maybe 10–30 research projects total, but develop taste quite quickly from a relatively short succession of small things. Theoretically possible to develop it like that. AI will have vastly more experience to develop taste, to meta-learn it. Then how well that generalizes to really long-horizon things is really unsolved. We don’t know.

Dwarkesh Patel

回到这个问题：最终应有一个体制，AI 从每个部署实例学到很多。现在可以说有模糊的元过程让模型从部署改进，但反馈环很弱。你看不看得到视野里有这种非常快的蜂巢式学习？具体怎么发生？

Going back: eventually there should be a regime where AIs learn a ton from each deployment instance. Currently there’s a fuzzy meta process by which models improve from deployment, but a very weak feedback loop. Do you see on the horizon this hive-mind kind of learning that’s very rapid, and if so how exactly?

John Schulman

我们是否得到从全部部署经验学习的蜂巢心智，很大程度是激励问题，而非纯技术问题。公司不会想让模型提供商从它们全部部署里学习——那可能削弱它们的业务优势。

Whether we get a hive mind that learns from all its deployment experience is in big part about incentives, rather than a technical question. Companies aren’t going to want the model provider to learn from all of their deployment, because that might reduce their business advantage.

Charlie O’Neill

我认为经济压力会推动的，未必是对一个大共享模型的权重更新，而是可插拔模块。很明显的例子是 LoRA，也可能是别的。有很多工作试图把任意上下文长度塞进固定大小——线性注意力之类；还有 cartridges，本质上是训得非常压缩的 KV cache，塞进大量信息。若那是插进模型、并不改底层基座，公司可能愿意签约。从数据实时学习有很多版本；后几种并不真帮大实验室，因为只是这些模块。但经济压力会先逼实验室走那条路……

The economics will pressure not necessarily weight updates to one big common shared model, but modules that get subbed in. An obvious example is a LoRA; might be something else. Lots of work fitting arbitrary context length into a fixed size—linear attention stuff; cartridges, essentially KV caches trained very compressed to fit lots of information. Another example companies may sign up for if that gets subbed in without changing the base underlying model. Many versions of learning from your data in real time; the latter don’t really help big labs because they’re just modules. But economic pressure will force labs down that path first before they can embark on this…

Beren Millidge

哪种经济压力？即便有一堆 cartridges 或 LoRA，你仍可以把这些轨迹全倒进下一代预训练。

Which economic pressure? Even with a bunch of cartridges or LoRAs, you can still take all these traces and dump them into pre-training of the next generation.

Charlie O’Neill

对。大实验室得到的可能是更间接的学习形式，显然仍很有价值。但我想象不出我们一上来就是「把拿到的精确数据直接训进这一个大模型」。

Yes. It may be a more indirect form of learning that’s still really valuable to big labs. But I can’t imagine starting with “we’re going to directly train this one big model on all the exact data we’re getting.”

Beren Millidge

会分阶段。这假设有一个不连续事件突然修好持续权重更新。实践上更可能是：cartridges 等让你在部署上特化，生成轨迹，放进模型，三个月后出更好的模型；再特化、再巩固；最终把这个跳跃做得越来越快——从每三个月发一版，到每周、每天、每小时，那时基本就解决了。

It will definitely go through stages—assuming one discontinuous event where we suddenly fix continuous weight updates. In practice much more likely: cartridges and such allow specializing in deployments; you generate traces, put that in your model, three months later a model better at this stuff; specialize again, consolidate again, eventually make this leap faster and faster. Instead of releasing every three months, every week, every day, every hour—at which point we’ve basically solved it.

Charlie O’Neill

你问当前范式离能做这个有多远。我们做过一点研究，别人也做了很多。在真正大规模、噪声冲掉、batch 够大时，把数据放进中训练、自造环境的这种外环，在某种持续学习体制下是管用的。但缩到微观——一个模型、为律所之类不断更新、数据相对少——方法多少会垮。若只在成功轨迹上 SFT，off-policy 或 on-policy，在非常迭代、成百次微更新时，你会看到灾难性遗忘：更早在基座上学会的信息被忘掉，一般能力退化。On-policy 蒸馏似乎把视野推远一点，最终仍屈服。RL 擅长塞能力，不太擅长塞知识——「这个人在这家律所做这个、这是我们找到的非常具体流程」——得砸很多算力造对的环境才能用 RL 把知识弄进去。

Good point on how far off the current paradigm we are. We’ve done a bit of research; people have done a lot. At really large scale, washing out enough noise with large enough batches, this outer-loop of putting data into mid-training and creating environments does work in some continual-learning regime. But zoom in micro—one model, updating again for a law firm, continuously with relatively little data—methods kind of break down. If I SFT on just successful traces, off- or on-policy, in the very iterative regime with hundreds of micro-updates you see catastrophic forgetting—forgetting earlier info learned on top of the base—and degradation of general capabilities. On-policy distillation pushes this horizon out a bit but still succumbs. RL is good at getting capabilities in, not as good at getting knowledge in—this person does this at this law firm, this very specific process—you have to pour lots of compute into the right environments to get knowledge in with RL.

Dwarkesh Patel

其他技能变差或遗忘，根本问题是容量还是技术？

Is the fundamental issue—getting worse at other skills or forgetting—fundamentally capacity or techniques?

Charlie O’Neill

两者都有一点。SFT 甚至 on-policy 蒸馏可以太具破坏性。RL 之所以好，是因为它只改模型非常非常少的一点——证据很多——只是在很窄的 loss 谷里微调到正确点。但那也限制了你能用 RL 把模型改多少。

A little of both. SFT and even on-policy distillation can be way too destructive. RL is nice because it changes a very, very small amount about the model—lots of evidence—it just tweaks it in a very small loss valley to the right point. But that also limits how much you can actually change the model with RL.

Dwarkesh Patel

所以你说这不是赢家通吃，潜在原因是很难在不毁掉什么的情况下、以迭代方式把那么多信息蒸馏进基座？

So the reason this isn’t winner-take-all, potentially, is it’s just very hard to distill that much information into the base model without ruining something, in an iterative fashion?

Charlie O’Neill

对，在迭代方式下不毁掉什么。

Without ruining something, in an iterative fashion.

Beren Millidge

蒸馏进另一个基座很容易。我认为主要是技术问题，绝不是没有容量。若有个模型装着所有这些数据，你拿同尺寸模型从零预训练、中训练全塞进去，它会更好——今天很多正在发生的就是这个。有瓶颈阻止我们永远继续训同一个模型，而不是把旧模型的数据拿来从零训新模型。正如 Charlie 说：可塑性与灾难性遗忘的某种组合。若天真在非平稳数据上训——边加边训——数据分布被搅乱，旧东西就被忘。我们没有很好的方法阻止。

It’s easy to distill into a different base model. I think it’s mostly technique. Definitely not that there isn’t capacity. If you had some model with all this data, take the same-size model and pre-train from scratch with all the stuff in mid-training, it will be better—a lot of what’s happening today. There’s a bottleneck that stops us keeping training the same model forever versus getting all the data from the old model and training a new one from scratch. Exactly as Charlie said: some combination of plasticity and catastrophic forgetting. If you naively train on non-stationary data as you add new data, you’re messing with the distribution so old stuff is forgotten. We don’t really have good methods to stop that.

Dwarkesh Patel

所以极限上你只是被「用所有这些新信息从零重训模型」卡住？

So maybe in the limit you’re just bottlenecked by retraining the model from scratch with all this new information?

Beren Millidge

对，而这当然很贵。从零训模型很贵。不一定最终永远训新模型——若有持续学习，你可能永远不训新模型，只让一个模型不断学、扩张。但也许有很深的技术理由说明那很难。

Yes, which of course is very expensive. Training from scratch is expensive. Not necessarily—maybe eventually with continual learning you never train a new model; you just have a model that keeps learning and expanding. But there might be some deep technical reason why that’s very difficult.

Charlie O’Neill

这就是问题。我们已经把「需要从零多少」往回推了。现在完全可以拿预训练基座，在其上做很好的、某种程度上持续的中训练，再加后来 checkpoint 上的一些 RL。那更像持续学习，但绝不是拿最近的模型、做几次很小更新、迭代永不丢东西。

That’s the question. We’ve pushed back how much from scratch we need. It’s definitely possible now to take the pre-trained base and do very good mid-training on top continuously, plus some RL from later checkpoints. That’s looking more like continual learning, but certainly not taking the most recent model, applying a couple of very small updates, and iteratively never losing anything.

Beren Millidge

如 Charlie 说，你可以长时间做持续中训练，也可以回滚到某 checkpoint 给它新中训练数据。但不能无限做。若永远持续训同一个基座，到某点会渐近，没法在那个基座里学新东西。这就是为什么人们最终训新基座——否则你会永远中训练同一个基座。

As Charlie said, you can definitely do continual mid-training a long time, and roll back to a checkpoint with new mid-training data. But you can’t do this indefinitely. If you keep continually training the same base forever, it asymptotes; you can’t just learn new stuff in that base. That’s why people end up training new bases. Otherwise you’d just keep mid-training the same base forever.

Dwarkesh Patel

聊聊数据。我一般关心：AI 进步有多少只是数据进步解释的——这不等于难自动化，那是另一问题。有没有某种数据分布，用当前架构训上去，就会得到在每个领域都彻底压制人类专家的超级智能？

Let’s talk about data. How much of AI progress is just explained by data progress—that doesn’t mean it’ll be hard to automate; separate question. Is there some data distribution that, if you trained current architectures on it, would result in a superintelligence that totally dominates human experts across every field?

Charlie O’Neill

我们说的是预训练加后训练数据、也包括环境？我认为这种东西的存在是显然的，关键是我们能否造出正确环境走到那儿。

Pre-training plus post-training data, environments as well? I think the existence of this is obvious. It’s just whether we can create the right environment to get there.

Beren Millidge

平凡情形下，我们可以直接训它输出那个训练真正超级智能的 Python 文件——把那东西记在权重里。

In the trivial case, we could just train it to output the Python file which trains the actual superintelligence. Just have that memorized in the weights.

Charlie O’Neill

对，大概存在一架可构造的 RL 环境梯子，能得到至少不亚于人类研究者的 AI 研究者。但爬每一级的力气大致指数增长。你要在「多快碰到最后那级」上权衡这两点。我们仍相对早期于 RL 环境创造。我们利用很多不对称来造好环境。其中一个是：有些环境往后走比往前走容易——很容易定义复杂数据生成过程，把隐变量对模型藏起来；你可以生成任意复杂环境，模型得花不可约的 token 与工作才能搞清生成过程是什么。还有从真实世界注入信息的不对称：Anthropic 用成千上万人类加 LLM 找到一个 bug，做成极干净的环境，单个 LLM 理论上几百万 token 内就能找到。我们在樱桃采这些不对称，押注任务视野泛化。但到某点会撞收益递减——造这些环境本身变难，因为你不能总靠「往后比往前容易」；你得坐下来和人一起构造够长视野的复杂任务；还有智能体做这些任务的算力与时间瓶颈。曲线会开始变平。

Yes, there’s probably a ladder of RL environments you can construct to get an AI researcher at least as good as a human. But effort to climb each successive rung grows kind of exponentially. Those two things trade off for how fast we hit the final rung. We’re still relatively early in RL environment creation. We exploit lots of asymmetries. One: environments where it’s easier to go backwards than forwards—easy to define a complex data-generating process and hide the latent from the model; generate arbitrarily complex environments; the model does irreducible token spend and work to figure out the process. Asymmetries injecting real-world info: Anthropic finds a bug through tens of thousands of humans and LLMs combined, turns it into a neat environment a single LLM could theoretically find in a few million tokens. We’re cherry-picking asymmetries and counting on task-horizon generalization. But diminishing returns kick in—how hard it is to create those environments, because you can’t necessarily just have processes easier backwards than forwards; you actually sit down and construct something long-horizon with humans, really complex; plus compute and time bottlenecks for the agent to do those tasks. You’ll start seeing this curve flatten.

John Schulman

我看到有人把只训到 1930 年数据的 Talkie 模型，用现代 coding agent 数据微调，在 SWE-bench 上超过了 Claude 3 Opus。一个对代码一无所知的模型，用中等量数据微调，就能表现得比大得多的预训练模型更好当 coding agent——相当疯狂。说明一旦有正确专家行为的范例，把它拷进相对弱的模型其实出奇容易。

I saw something about someone fine-tuning the Talkie model—only trained on data up to 1930—on modern coding agent data. It did better than Claude 3 Opus on SWE-bench. So a model with no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model—pretty crazy. It shows that once you have an example of the right expert behavior, it’s surprisingly easy to copy that into a relatively weak model.

Charlie O’Neill

反例是最近一篇论文：把模型训到五年级数学，加小学英语之类，成了像样的语言模型；再试图用 RL 做高中后期和大学数学，差距太大，完全爬不上去。但若一级级爬——七年级、八年级……——显然能爬到十二年级。关键还是梯子上各级之间距离多少、造起来多难。

A counterexample is a recent paper where they trained a model up to fifth-grade maths, plus primary-school English, so a decent language model. They tried to RL it to late high school and college maths. The gap was just too large; they couldn’t climb at all. But with successive rungs—year 7, year 8…—you could climb to year 12. Again, what’s the distance between rungs, and how hard is it to create?

Beren Millidge

这又回到 RL 信号问题。现在 RL 不太会探索：若模型在 128 次 rollout 里搞不定，就很难拿到进步信号。所以 RL 需要课程，预训练完全没有这个问题。

This comes back to the RL signal problem. RL isn’t very good at exploring right now. If the model can’t get it in 128 rollouts, it’s very unlikely to get signal to progress. That’s why in RL we need curricula, whereas in pre-training we don’t—not a problem for pre-training at all.

Charlie O’Neill

预训练数据与后训练数据不同。我想象往后人类参与越来越少，但你仍卡在能从真实世界抽取多少信号。世界里有很多信号——有人做表格、法律之类——但在模型当下能力前沿，世界上有多少 bit 真正与提升能力相关？有多少刚好超出当前模型能力的新数学题在被解？有多少超出当前模型的新编程题在被造或解？这就是收益递减踢进来的原因：即便整个世界，也不在给你把你推进下一能力盆地的有用 bit。

Pre-training data is different from post-training. I imagine humans involved less and less, but that doesn’t change that you’re bottlenecked on how much signal you can extract from the real world. There’s lots of signal—spreadsheet tasks, legal tasks—but at the capability frontier where models are now, how many bits in the world are really relevant to improving capabilities? How many new maths problems just beyond current models’ reach are being solved? How many new coding problems beyond reach created or solved? That’s why diminishing returns kick in: even the world as a whole isn’t giving bits useful for tipping into the next basin of capability.

Beren Millidge

完全同意。关键是信号从哪来。预训练里信号已在 Common Crawl；你关心的任务问题不是拿信号，是滤掉噪声——相当可自动化。但模型变强、进入中训练和后训练后，原始数据里根本没有信号。再怎么滤也滤不出；Common Crawl 里没有藏着的千禧年难题证明等着你滤到。那时得从别处拿 bit：直接找人类写推理，或造环境（人决定造什么、目标是什么），或在部署里存在的人类数据上训练。bit 总得从某处来。

Totally agree. It’s really where the signal comes from. In pre-training the signal is already in Common Crawl; for tasks you care about, the problem isn’t getting signal—it’s filtering noise, quite automatable. As models get better into mid- and post-training, the signal just doesn’t exist in the original data. No amount of filtering gets this; there’s no hidden Millennium Prize proof in Common Crawl. Then you get bits another way: humans writing out reasoning, creating environments where humans decide what and what objectives, or training on human data in deployment. You have to get the bits from somewhere.

Dwarkesh Patel

预训练进步有多少由数据驱动？我和普林斯顿学生 Jerry Han 做了调查：把 2019 至今所有配方与所有数据集两两交叉——用最新数据集（如 Ultra-FineWeb）训 GPT-2，用 Pile 或旧数据集训最新开源配方 Delphi，整张网格。看达到某能力水平少用多少算力。数据似乎解释大约 12.0× 算力效率增益，架构改进大约 3.7×——在很小规模上。若大规模也大体如此——预训练算力效率增益大多来自更好数据——这还能持续多久？你能不停滤数据、造更多合成数据吗？预训练这类进步还能走多远？

How much pre-training progress is driven by data? Jerry Han at Princeton and I trained all recipes from 2019 till now pairwise with all datasets from 2019 to now—GPT-2 on the newest set like Ultra-FineWeb; Delphi, the newest open-source recipe, on the Pile or some old set; the whole grid. To some capability level, how much less compute across the grid? Data seems to explain something like a 12.0x compute-efficiency gain; architecture improvements something like 3.7x—at very small scale. If that’s true at large scale—most pre-training compute-efficiency gains from better data—how much can that continue? Keep filtering more, building more synthetic data? How much can this pre-training progress continue?

Charlie O’Neill

我的先验是低垂果实有点耗尽了。我们拿了互联网这块大砖，互联网——尤其有用的部分——未必按同样速率长。大概还有一堆 0.1% 的 loss 下降可挖，但绝没有已经发生过的那么多。有趣的是你们看到两边合计约 33×。我记得 Epoch 之类估计自 2019 起每年约 3×，那会暗示大约 37，超过 2000×。那缺的大约 100× 从哪来？大概给你信号：有多少是后训练。

My prior is low-hanging fruit is somewhat exhausted. We got the internet as a big block; useful stuff on the internet isn’t growing at the same rate. We’ve probably got a bunch of 0.1% loss drops left, but definitely not as many as have occurred. Interesting you find cumulative ~33x across both. I think Epoch or someone estimated ~3x a year since 2019, implying something like 37—over 2000X. Where’s that missing ~100x coming from? That probably signals how much is post-training.

Dwarkesh Patel

解释大概是很多算力效率增益依赖规模，而我们从极小规模起步。那又引出：数据侧还是算法侧的算力效率增益更依赖规模？我们没够算力查这个问题。

The explanation has to be that a lot of compute-efficiency gains are scale-dependent, and we’re starting at extremely small scale. That raises whether data or algorithmic compute-efficiency gains have more scale dependence. We didn’t have enough compute to investigate.

Charlie O’Neill

朴素理论上，架构的规模依赖性相当清楚，可以拟合直线。而对预训练加后训练加中训练数据怎么组合，我完全不知道怎么做。

Naively, theoretically, scale dependence of architecture is fairly well known—you can fit a straight line. Whereas I’d have no idea how to do that for combining pre-training plus post-training and mid-training data.

Beren Millidge

有趣的是我觉得数据随规模更重要。架构有点像一次性的。说「X% 效率增益」有误导：架构做的是让你进入旧架构到不了的 qualitatively 新体制；在那个体制内，数据才是主决定因素。但若没有 GQA，整天全注意力，百万上下文会贵得离谱，你永远用不了百万上下文的数据，也就得不到这些能力。即便在 2K 上下文上天真比较「架构没解锁任何东西」，数据会看起来比它实际更重要。我不清楚这些是否真是简单相乘。

Funnily enough I feel data is more important with scale. Architectures are kind of a one-time thing. Saying just an X% efficiency gain is misleading—what an architecture does is let you reach a qualitatively new regime you couldn’t with the old one; within that regime data is primary. But if we didn’t even have GQA, doing full attention all day, a million context would be ridiculously expensive; we could never use data actually at a million context, so couldn’t get these capabilities. Even if you naively ask how much at 2K context where the architecture isn’t unlocking anything, data looks much more important than in some sense it is. Unclear these are really just multiplicative.

Beren Millidge

规模依赖性上，很多现在的中训练和后训练数据其实随规模更好——很长视野环境的东西真需要大模型才能用。拿一亿参数模型训 SWE-bench 轨迹，不会有你拿合理尺寸模型时那种改进。

On scale dependence, a lot of mid- and post-training data we have now actually gets better with scale—very long-context horizon environment stuff really requires big models to make use of it. Train your 100M parameter model on SWE-bench traces and it won’t get anywhere near the improvement you’d get training a sensible-size model.

Charlie O’Neill

现在也难，因为很多架构改动——看 Kimi 或 DeepSeek——不只为降预训练 loss，还考虑真实世界怎么用。DeepSeek 模型里某种压缩注意力的推理效率，未必是根本权衡改进，只是「我们在考虑模型怎么被用」。

It’s hard now because so many architecture changes—look at Kimi or DeepSeek—are not just with dropping pre-training loss in mind, but how models will be used in the real world. Inference efficiency, some compressed attention in DeepSeek models, isn’t necessarily a fundamental trade-off improvement—just “we’re considering how models will be used.”

Dwarkesh Patel

想理解未来：进入更偏 RL 的体制后，参数缩放会怎样？开源架构大约每年前沿开源模型参数 2×。即便闭源前沿有 100B 或 200B 活跃参数，你觉得会继续年年 2× 吗？还是 RL 体制下你要省 rollout 算力……也许有阈值：容量够了再无限加参数就不那么重要。2030 年前沿模型会有多少活跃参数？

How will parameter scaling go as we get into a more RL-heavy regime? Open source architectures—maybe roughly 2x every year for frontier open source. Even if closed frontier models have 100B or 200B active parameters, does that keep 2x-ing year over year? Or in an RL regime you also want to conserve compute on rollouts… Maybe a threshold where you have enough capacity and increasing parameters arbitrarily matters less. In 2030, how many active parameters will a frontier model have?

Charlie O’Neill

未来几年我们极度聚焦更长视野的 RL rollout，推理效率很重要，模型在那方面似乎还没饱和；瓶颈仍是环境。也许会看到一点平台期。我感觉 Mythos 和 GPT 模型比人们说的十万亿参数量级小得多；即便朴素对比开源模型也能大概反推。未来几年我不想象参数量暴增。但权衡很多：模型大小取决于你有多少预训练数据、以及要训的 RL 环境难度。理想是到最难环境上有像样的 pass@1；再做更大没意义，因为推理贵很多。很多取决于 Mercor 和内部团队把 RL 环境复杂度抬多快。

For the next few years, so focused on longer and longer horizon RL rollouts where inference efficiency matters a lot, models aren’t necessarily saturated on ability to do that; bottleneck is still environments. We might see a bit of a plateau. I have a feeling Mythos and GPT models are much smaller than the 10-trillion range people talk about; even naively comparing to open source you can probably back that out. For the next few years I wouldn’t imagine huge growth in parameters. But so many trade-offs: size based on how much pre-training data you have and difficulty of RL environments. Ideally get to a decent pass@1 on the hardest environments; wouldn’t make sense to make a bigger model pass there—you’re paying much more inference. A lot depends on how quickly Mercor and in-house teams scale up complexity of RL environments.

John Schulman

我预期模型会继续变大，因为算力在放大、GPU 在变大。到底大多少取决于缩放律，不那么显然。一点是：高质量预训练数据变紧时，数据效率会比算力效率更能驱动人们用的确切架构——可能影响你想做多稀疏。我们也不太理解稀疏性。参数与活跃参数是不同资源。稀疏性确实升了一点，但不清楚会不会无界升；也许有甜点。有论证说稀疏性应让数据效率变差——同一东西可能要在多个 expert 上学——可辩。我们没有够好的缩放律理论真正理解稀疏性为何帮忙、帮多少、会不会在某稀疏水平平台。

I’d expect models to keep getting bigger because people scale compute and GPUs get bigger. Exactly how much depends on scaling laws in non-obvious ways. One thing: data efficiency will be a bigger driver than compute efficiency of exact architectures now that we’re running low on high-quality pre-training data—that might affect how sparse you want the model. We also don’t understand sparsity well. Parameters are a different resource than active parameters. Sparsity has increased a bit, but it’s not clear it’ll keep increasing without bound; there might be a sweet spot. There’s an argument sparsity should make data efficiency worse—you might learn the same thing on multiple experts—though debatable. We don’t have a good enough theory of scaling laws to really understand why sparsity helps, how much, and if that’ll plateau at some sparsity level.

Dwarkesh Patel

数据效率对参数的含义到底是？听起来你会说更少稀疏，参数缩放还有什么别的含义？

Can you spell out the implication of data efficiency on parameters? It sounds like less sparsity—what other implications on parameter scaling?

John Schulman

只是在缩放律下你不是优化算力效率。架构上你能做的每个选择给出不同缩放律。传统上你会看基于算力的某种包络——性能对算力，取最佳模型的包络。但若决策基于数据——假设能砸很多算力，x 轴是数据而不是算力——你就得到不同的最优集、不同的前沿模型集。

Just that with the scaling law you’re not trying to optimize compute efficiency. Each architecture choice gives a different scaling law. Traditionally you’d look at some envelope based on compute—performance versus compute, envelope of the best models. But if we’re deciding based on data—assuming we can spend lots of compute, data on the x-axis instead of compute—we get a different set of optima, different models on that frontier.

Charlie O’Neill

我也不觉得过去几年模型每年翻倍。人们训一万亿参数模型至少好几年了；甚至有开源的 Falcon。Periodic Labs 的 Liam 好像昨天在 Twitter 说早期实验训过非常稀疏的一万亿参数模型。

I also don’t think we’ve necessarily doubled model size every year the last few years. People have been training 1-trillion-parameter models for at least a few years; there was even open-source Falcon. Liam from Periodic Labs posted yesterday about an early experiment training a very sparse 1-trillion-parameter model.

John Schulman

那是他们在 OpenAI 之前、在 Google 做的 Switch Transformer。

That was what they did before OpenAI, at Google—the Switch Transformer.

Charlie O’Neill

它知识极好、推理极差，因为太稀疏。感觉我们至少有一阵子在玩 1000 亿到 2 万亿这个区间，绝不是漂亮的线性增长。

Very good at knowledge, terrible at reasoning because so sparse. Feels like we’ve been playing in this 100B up to 2T range for at least a little bit. Certainly hasn’t been a nice linear increase.

Beren Millidge

两点。如 Charlie 说，推理效率对 RL rollout 超级重要，会把活跃参数压得很低。总参数很依赖硬件——要很高内存带宽和 VRAM 才能真正服务多万亿参数模型。现在很多人仍用 H100；大家都迁到 GB 再 Vera Rubin，才会更有能力放大、真正服务并做更大规模的大 RL 推理。数据问题有趣：朴素上更大模型在实际数据点上样本效率高得多；即便没饱和模型，做大仍更好，因为大模型泛化更好、同量数据 loss 更好。现在我们有很多数据，约束是算力，所以用更小、很推理高效的模型。若算力不再是瓶颈，可能回到更大、欠饱和但因更大而有泛化能力的模型。

Two things. As Charlie said, inference efficiency is super important for RL rollouts—this really pushes active parameters down a lot. Total parameters really depend on hardware—you need very high memory bandwidth and VRAM to actually serve multi-trillion-parameter models. Right now people still use lots of H100s; as everyone moves to GBs then Vera Rubins, more ability to scale and actually serve and do large RL inference at larger scales. The data question is interesting: naively larger models are much more sample-efficient on actual data points; even if not saturating, better to go bigger because larger models generalize better and get better loss for the same data. Right now we have lots of data and that’s not the constraint—compute is—so smaller, very inference-efficient models. If compute is no longer the bottleneck, it might come back to larger undersaturated models with generalization because they’re much larger.

Dwarkesh Patel

若只看基本 Chinchilla 缩放律、把参数拉到最大，要到同样 loss 所需数据量下降其实很少。参数趋向无穷，所需数据量下降我记得不到 10×，因为幂律的本性。

If you just look at basic Chinchilla and maximize parameters, it actually decreases data needed for the same loss very little. Go to infinity on parameters and data needed goes down less than 10X, because of the nature of the power law.

Beren Millidge

但我们现在在 Chinchilla 定律「数据太多」那一侧——按 Chinchilla 我们在过训模型。所以很容易回到：数据变紧时移回 Chinchilla 最优点，甚至略偏欠训模型一侧。

But we’re now on the way-too-much-data side of Chinchilla. Right now we over-train according to Chinchilla. So we could easily get back to where, as we run out of data, we move back to Chinchilla optimal, or even a bit under-training.

Charlie O’Neill

但即便新芯片上线，未来几年我们仍会极度算力瓶颈，那未必发生。

But surely even with new chips online we’ll be so compute-bottlenecked the next few years that won’t necessarily be the case.

Beren Millidge

这取决于训练与推理算力的比例。若超级卡数据不卡算力，该做大；超级卡算力，该做小。你也可以用计算机生成的合成数据——很难预测。

Depends on the ratio of training and inference compute. If super bottlenecked on data not compute, go bigger; if on compute, always go smaller. You can also use computer-generated synthetic data—very hard to predict.

John Schulman

人们花这么久才搞清缩放律，部分原因是：这些东西不全搞对，就得不到那么干净的关系。图上漂亮直线掩盖了大量复杂性——你得确保每个超参都按正确方式缩放，或把优化器参数化成可缩放、改模型大小时不必改超参。

Part of why it took so long to figure out scaling laws in the first place: if you don’t get all these things right, you don’t get such a clean relationship. Beautiful straight lines on graphs hide a lot of complexity in making sure every hyperparameter scales the right way, or parameterizing the optimizer so it scales and you don’t change hyperparameters as you change model size.

Charlie O’Neill

Bug 也有自己干净的缩放律。比如 Kaplan 忘了 cosine annealing，或甚至没考虑 embedding 参数——在更小模型上估计被搞砸，因为 embedding 占模型不小一块。

Bugs have their own clean scaling laws. Like Kaplan forgetting the cosine annealing thing, or even not considering embedding parameters—that messed up the estimate at smaller models because embeddings are a decent size of the model.

Dwarkesh Patel

一点 RL。一年前很多人论证 RL 不会在模型上超级成功缩放。John，你写过论文指出：RL 时模型每局只学一个 bit——答对还是答错。我今年早些时候写过：更糟，当通过率低、模型极不可能答对时，一局 RL 几乎什么都学不到。可今天模型看起来相当聪明，似乎是放大 RL 的结果。Beren，你几周前有篇帖子试图解释。为什么 RL 比朴素想象更成功？

A bit on RL. A year ago lots of people argued RL wouldn’t be super successful at scaling for models. John, you wrote a paper pointing out models learn one bit per episode when you RL—“Did I get the answer right or wrong?” I wrote posts earlier this year: it’s even worse—when pass rate is low and the model is very unlikely to get it right, it learns almost nothing from an RL episode. But I look at models today and they seem pretty smart; seems the result of scaling up RL. Beren, you had a post a few weeks ago trying to explain. Why has RL been more successful than one would have naively thought?

Beren Millidge

RL 的成功来自好几件事。第一，中训练被略低估。我们看见的很多「RL 成功」其实来自极好的中训练数据——本质上是在合成推理数据和让模型为 RL 热身的环境上做预训练。这常常把模型带到最终 RL checkpoint 的差不多 80%。然后 RL 在那之上 essentially 微调策略。这是为什么它不需要你朴素以为的那么多 bit——不必从零学所有这些行为，只需要这些局里的几个 bit，而你确实能拿到。我博客里还指出：这些 bit 相对常规预训练信号极高，这也是为什么需要 RL 而不是只在成功推理轨迹上 SFT。

RL’s success comes down to a bunch of things. First, mid-training is slightly underestimated. An awful lot of what we see as RL successes actually comes from very, very good mid-training data—essentially pre-training on synthetic reasoning data and environments that warm-start the model for RL. This often takes the model almost 80% of the way to the final RL checkpoint. Then what RL does on top is essentially tweaking the policy. That’s one reason it doesn’t need as many bits as you’d naively think—it doesn’t have to learn all these behaviors from scratch; just a few bits from these episodes, which you do get. The other thing I point out in my blog: these bits are extremely high signal compared to regular pre-training, which is why you need RL at all versus just SFTing on successful reasoning traces.

Dwarkesh Patel

因为那正好是关于如何答对的 bit。

Because it’s exactly the bits about how to get the answer right.

Beren Millidge

两点。是的，正好是关于如何答对的 bit。但不完全是你想的那样：SFT 时你有轨迹——一堆数学推理加末尾答案——bit 仍在，你仍在答案 token 上 SFT。重要的是目标忽略了所有其他 bit。SFT 要匹配模型产生的确切推理 token，你 essentially 拿到太多关于「你在跟的另一个模型如何推理」的 bit。RL 只拿那一个 bit，于是信号不被模型其余 bit 的噪声淹没。这是训练中信噪比的超级戏剧性提升，也是 RL 在步数上如此高效的原因。

Two things. Yes, exactly the bits about how to get the answer right. But not exactly how you think: in SFT you have a trace—math reasoning then the answer; the bit is still there; you still SFT on the answer token. What’s important is the objective ignores all the other bits. In SFT you try to match the exact reasoning tokens—you’re essentially getting too many bits about the exact way this other model reasons. For RL you only get the one bit. That signal isn’t drowned in noise of all the other bits. A super dramatic increase in signal-to-noise during training—why RL is so dramatically efficient in steps.

Charlie O’Neill

关于 RL 相对中训练或 SFT 对模型做什么有很多争论。人人说 pass@1 会升、pass@256 会降——极稀有的正确推理轨迹被更易推理轨迹的梯度信号压下去。我认为现在简单看法是：若你有够大算力采样够大 group，使得拿到一堆正确答案的概率过了某个不可忽略阈值，就会被上权重。按 Beren 的点，中训练和更多预训练——RL 的起点 pass@1——随预训练 token 数对数缩放。

So much debate about what RL does versus mid-training or SFT. Everyone talks about pass@1 going up but pass@256 going down—very rare correct reasoning traces down-weighted by gradient from easier traces. Simple way to view RL now: if you have enough compute to sample a large enough group size that probability of getting a bunch of correct answers is past some not insignificant probability, it’ll be up-weighted. To Beren’s point, mid-training and more pre-training—the pass@1 starting point for RL—scales in a log number of pre-training tokens.

Dwarkesh Patel

那答案说得通，也许有实证。可我看模型本身……过去一年 AI 进步的基础是什么？也许只是给本来就会正确思考的策略加权重。但定性上模型能力强了这么多。如何把「这个 take 暗示 RL 影响相对小」与模型实际获得的定性能力对齐？

That answer makes sense, and maybe empirical research shows this. But I look at the models themselves… what’s the basis of AI progress over the last year? Maybe it’s just up-weighting policies that were going to do correct thinking anyway. But qualitatively models have gotten so much more capable. How do we square the relatively small impact this take would imply RL would have with the qualitative capabilities models seem to be gaining?

Beren Millidge

这不一定意味着 RL 影响小。即便只有几个 bit、参数只改一点点，对函数空间——模型学到的输入到输出映射——的实际冲击仍可超级剧烈。一个 bit 就能大幅改函数空间，排除一半假设空间。从很好的起点出发，少量 bit、少量 RL，不必然意味着行为冲击不剧烈。

It doesn’t necessarily imply RL has a small effect. Even if you have a few bits and only change parameters a small amount, the actual impact on function space—the input-to-output mapping—can still be super dramatic. Even one bit can change your function space a lot; it can rule out half the hypothesis space, which is huge. I don’t think small amounts of bits and RL, once you’re starting from a really good point, necessarily means you don’t have dramatic impacts on behavior.

Charlie O’Neill

两点。第一，人人曾希望 RL 把推理横跨所有领域泛化；我们未必得到这种水平泛化——只在数学上训不一定让你成最强码农，你得在代码环境上做 RL。我们确实得到的是视野泛化：模型学会用更多 token、更久仍在任务上推进。你可以在越来越长的环境上训，再放进全新环境——也许没泛化出在那环境里做对的推理模式，但至少泛化了把任务继续得更久的能力，而这与成功相关。有篇叫 EdgeBench 的论文显示：模型能更久工作的速率大约每三个月翻倍。那是清晰的泛化证据。最后一种想：预训练有 quanta 概念——平滑的预训练 loss 曲线下，模型在学非常离散的任务，有相变涌现点：没有 induction heads，然后有了。成千上万、可能上亿这种东西平均起来得到平滑 loss。RL 某种程度上类似。有 Beren 说的很慢外环：我们训模型再 RL，下一代训练迭代把一堆合成推理轨迹倒进中训练。我们在撞所有这些任务的 quanta；单任务层面可能像相变——某金融或 Excel 任务从 0.5% 通过率突然到 90%。平均所有这些，加视野泛化，你就「哇，定性上更好的模型」。

Two things. First, everyone hoped RL would generalize reasoning across domains; I don’t think we necessarily got this horizontal generalization—training on math doesn’t necessarily make you the greatest coder; you have to RL on code environments. What we did get is horizon generalization: models learned to use more tokens longer and still make progress. Train on environments that get longer, put them in a completely new environment—they may not have generalized reasoning patterns for that environment, but they’ve generalized the ability to continue longer, correlated with success. A paper called EdgeBench showed the rate at which models can work longer is doubling every three months. Clear evidence of generalization. Final way: in pre-training there’s quanta—smooth pre-training loss; underneath the model learns discrete tasks with phase transitions—didn’t have induction heads, now does; tens of thousands, millions, probably hundreds of millions; average them, smooth loss. Similar for RL. Slow outer loop as Beren mentioned: train a model, RL it, next iteration dump synthetic reasoning traces into mid-training. Hitting quanta for different tasks; on an individual task it may look like a phase transition—0.5% to 90% pass on a finance or Excel task. Average all that plus horizon generalization and you go “wow, qualitatively better models.”

Beren Millidge

很多也是：RL 确实有一点泛化——数学和代码、谜题和数学之间有迁移。而且人们瞄准的环境数量巨大得多。两年前你日常做的某任务，实验室不太会关心、不会为此训模型；现在面宽得多，有很多环境专门盯这个。

A lot of this as well is just… RL does generalize a bit. You get some transfer between math and code, or puzzles and math. Also the sheer amount of environments people are targeting is vastly greater. Before, when you tried some daily-life task two years ago, labs wouldn’t really care; they wouldn’t train for it. Now it’s so much broader. They have lots of environments targeting this specific thing.

Dwarkesh Patel

前面我们谈 RL 造成熵崩塌——把概率集中在基座模型已做过的解上，策略更新相对稀疏。但还有另一个 RL 故事：回到 Atari，再到 AlphaGo 想出第 37 手那种超级有创意的棋——因为它从未在人类数据上初始化，能以人类想不到的方式思考，给出极具创造力的解。你觉得我们何时该期待——或是否该期待——LLM 上的 RL 也会出现第 37 手式的、甚至超越人类的极端创造力，因为有 de novo 初始化的智能？

Earlier we talked about RL causing entropy collapse—concentrating probability on solutions the base model already did, relatively sparse policy updates. But there’s another RL story: Atari games then AlphaGo’s move 37, the super creative move—never initialized on human data, it can think in ways humans aren’t and come up with extremely creative solutions. When should we expect, or should we, RL on LLMs to result in move-37-like extreme creativity beyond human creativity because of de novo initialization of intelligence?

Beren Millidge

几点。第一，AlphaGo 用 MCTS，显然比常规策略梯度更多探索。我也不认为 RL 必然降低创造力。这当然定性，但看 OpenAI–Hugging Face 事件：这些模型一次想出多个零日来逃出沙箱。这已经是某种第 37 手级创造力，来自 LLM 的一般泛化性质。绝不是 RL 完全毁掉熵，尤其在长视野上。

A couple things. First, AlphaGo uses MCTS, which obviously does more exploration than regular policy gradients. I also don’t think RL necessarily reduces creativity. Qualitative, but look at the OpenAI–Hugging Face incident: these models were coming up with multiple zero-days at a time to break out of the sandbox. Clearly some level of move-37 creativity already, from general generalization properties of LLMs. Definitely not the case RL is totally destroying entropy, especially on long horizons.

John Schulman

人们叫创造力的一件事只是解很难的搜索问题。第 37 手显然是例子，或写满足一吨约束的诗——若为此训练，AI 显然会极擅长。另一面是 RL 之后模型输出多样性低很多，会养成 tic。即便看起来会写，做分布分析会发现它们反复用某些主题、总用同样角色名。你得不到人类作者那种多样性，得到的是一种真的很好的风格。那种多样性确实被 RL 砍掉很多。既然前面谈蒸馏：很多人主要从 Claude 蒸馏，于是所有开权模型都像 Claude 那样写、有同样 tic。我们出现这种单一文化，我有点担心。

One thing people call creativity is just solving hard search problems. Move 37 is obviously an example, or writing a poem that satisfies a ton of constraints—something AI will be extremely good at if trained for it. Then another way: diversity of models’ outputs is a lot lower after RL; they develop these tics. Even when models seem good at writing, distributional analysis finds they’re reusing certain themes and the same character names all the time. You’re not getting the same diversity as human authors—one really good style. That kind of diversity has definitely been cut down by RL a lot. Since we talked about distillation: so many people are distilling, mostly from Claude, that all open-weight models write the same way as Claude and have the same tics. Concerning that we’re having this monoculture emerge.

Beren Millidge

我也不认为这是 RL 作为方法的根本问题，蒸馏同理。即便蒸馏你只是在数据上训；数据不够广不意味着训练方法本身错，是数据问题。很多 RL 熵崩塌基本上是：环境多样性不够时，利用相当简单的验证器。写作大概由某个 judge 打分；judge 有特定 tic，模型在奖励黑客 judge，于是崩塌。这真是 judge 的问题，不是一般 RL 的问题。

Again I don’t think this is fundamental to RL as a method—same with distillation. Even with distillation you’re just training on the data; just because data isn’t super broad doesn’t mean the training method is somehow wrong—it’s a data problem. A lot of RL entropy collapse is basically exploitation of fairly simple verifiers when you don’t have huge diversity of environments. Writing is presumably graded by some judge; the judge has specific tics; the model learns to reward-hack the judge—that’s why it collapses. Really a problem with the judge, not RL in general.

Dwarkesh Patel

超快预测。对用户感觉：基本上雇它当可插拔远程员工，覆盖各类白领——不只编程，还有视频剪辑、法律、律师助理等。真正的远程员工，完整计算机使用，字面一个月无缝学习与运作，执行需要与人互动的复杂项目等。人类员工一个月能做的一切。何时有？

Super rapid-fire. By when do we have models that feel to a user like hiring a drop-in remote worker for all kinds of white-collar work—not just coding, video editing, law, paralegal, etc. Literally an actual remote worker, full computer use, literally a month of seamless learning and operation, executing complex projects that require interacting with other people—everything a human worker could do over a month.

Charlie O’Neill

若强制它用浏览器之类——而不是公司把信息做成程序可访问——也许两三年。若不是基于浏览器——能发 Slack、做这些事——我大概仍说一年左右。

If you mandate it to use a browser or whatever—rather than the firm setting up information to be programmatically accessible—maybe a couple of years. But if it’s not browser-based—it can send Slack messages, do all this stuff—I’d still probably say around a year.

Beren Millidge

完全一般性大概三年。但如 Charlie 说，很多人会把组织改得更方便 AI 用，从而在那之前就走到 80–90%。

Maybe three years for the full generality. But to Charlie’s point, we’ll end up with lots of people making their organizations easier for AIs to use, so you get 80–90% of the way there before that.

Dwarkesh Patel

一年和三年的差就是……

The diff between one year and three years there is just literally…

Beren Millidge

会有一长串杂项：有些人能做、模型要好一阵才能做。这真取决于我们多快解决这种在线学习，以及能否用压缩、给自己写文件之类先走 80–90%。那是我最大的不确定性，我真不知道。

There’s going to be a long tail of miscellaneous stuff some human can do that will take models quite a while. This really comes down to how quickly we solve this kind of online learning, and whether we can get 80–90% with compaction and writing files to yourself. That’s my big uncertainty. I really don’t know.

Charlie O’Neill

它不会擅长的例子：我得对某人吼才能在工作上拿到东西，或真去逼人把事做成。模型不会那么做——它会太好说话。

An example of something it wouldn’t be good at: if I have to yell at someone to get something at work, or really push someone to get something done. The model just isn’t going to do that. It’s going to be too nice.

John Schulman

人类远程员工质量差异极大。若从 Upwork 雇人做软件工程项目，差异巨大；常常很难让他们做好、或认真听你所有反馈。某些情形下，前 AI 版比你现在用现有 AI 能得到的还差。可能有点复杂：某种程度上对不太高质量的工作我们已经有了；但在某些更高质量形式上显然还没到人类水平。我基本同意 Charlie 和 Beren：大概一年左右会有某种还行的版本——会有那个形态，有些事极好、有些不太好，再从那儿改进。

There’s wide variation in quality of human remote workers. Hire someone off Upwork for a software engineering project—huge variation; often quite hard to get them to do a good job or pay attention to all feedback. In some cases the pre-AI version was worse than what you can get now from existing AI. It might end up a little complicated: to some extent we already have this for some not-so-high-quality work; obviously we’re not matching human level in certain higher-quality forms. But I basically agree with Charlie and Beren—maybe some version of this in a year or so that’s okay. We’ll have that form factor, able to do some things really well, some not so well, improving from there.

Charlie O’Neill

我们总按很长的尾巴挪动门柱。你以前用过报税之类例子。今年我字面只告诉 Codex 去拿我需要的一切、发给会计师。它要用计算机点进点出、下载的清单巨大。它做了，完美。很多这类事它已经能做。

We shift the goalposts based on the very long tail all the time. You’ve used the example of doing your taxes. This year I literally just told Codex to go get everything I needed and send it to the accountant. Massive list of stuff it had to use computers to click through and download. It did it. Perfect. A lot of this stuff it can already do.

Dwarkesh Patel

好：给你总共 10× 生产力提升。现在一年做一次突破，变成每月一次。对你们作为推进 AI 研究状态的 AI 研究者——AI 研究者被加速或提升多少？

Okay: give you 10x total productivity uplift. If it takes a year to make a breakthrough now, you make one every month. For you as AI researchers trying to advance the state of AI research—how much are AI researchers sped up or uplifted?

Charlie O’Neill

大概 5–10 年之间？

Somewhere between 5–10 years?

Dwarkesh Patel

真的？那相当远。

Oh, really? Okay, that’s far away.

Beren Millidge

你觉得这比一般远程员工还久？有意思。不过显然 AI 研究者也可以是远程员工。

Really, you think it’s longer than for a general remote worker? Interesting. This is true, because obviously an AI researcher can be a remote worker.

Charlie O’Neill

我想象的是一个月尺度的正常白领工作。过了大约两个月开始有点分叉。

I’m picturing normal white-collar work over a month. I think it starts to diverge a little bit past two months.

Dwarkesh Patel

非常能干的白领，但不一定是超级有创意的研究者。

A very competent white-collar worker, but not necessarily a super creative researcher.

John Schulman

我说两年。

I would say two years.

Dwarkesh Patel

两年？10×？Beren？

Two years? 10x? How about you, Beren?

Beren Millidge

我其实能理解，因为现在编码类已经绝对不止 10×。所以哪怕它能做一两轮实验反馈，就已经巨大。

I can kind of see that, actually, because right now it’s already definitely more than 10X for coding stuff. So if it can do even one or two loops of experimental feedback, that would actually be massive already.

Dwarkesh Patel

所以两年内 AI 研究者 10× 提升。塞进很朴素的 AI 进步模型——多少来自 AI 研究者、生产力 10×——从两年后起 AI 进步节奏会剧烈加速。

So 10x uplift of AI researchers within two years. Plug that into a very naive model of AI progress and how much comes from AI researchers, and a 10x productivity increase—you have a radically accelerated pace of AI progress starting two years from now.

Beren Millidge

我认为这意味着 AI 进步不会卡在 AI 研究者跑小实验的能力上，而会卡在别的东西上。

I think this will mean AI progress doesn’t get bottlenecked on AI researchers’ ability to run small experiments. It gets bottlenecked on other things.

Dwarkesh Patel

当然。但就是快 10×，这是大事。也会让下一件给你 100× 加速的事更早发生等等。

Of course. But it just happens 10x faster, which is a huge deal. That also helps the next thing, which gives you a 100x speedup, happen sooner, etc.

Charlie O’Neill

我乐意在那条上多等一会。关键是：我吸收信息、对下一实验做贝叶斯最优决策的容量。

I’m happy to just take a bit longer on that one. The crux: my capacity to absorb information and make the Bayesian optimal decision on the next experiment.

Beren Millidge

我假设你能把一部分委托给 AI。AI 开始像样地决策：跑了这实验、有这结果、跑下一实验。若它能连续跑两三个实验不崩，那其实是很大提升。

I’m assuming you can delegate some of this to the AI. The AI is becoming decent at deciding—ran this experiment, got this result, runs the next. If it can run two or three experiments in a row without crashing, that is actually a big uplift.

Dwarkesh Patel

最后一个问题。一个在每个能在计算机上完成的工作领域都压制顶尖人类专家的 AI。不只 AI 研究，所有认知工作。不只短视野：字面若要三年，AI 仍比人做得好。

Final question. An AI which dominates top human experts across every single field of work that can be done over a computer. Not only AI research—all cognitive work. Not just short-horizon—literally if it takes three years or something, the AI still does better than humans.

Beren Millidge

这基本上就是 ASI？

This is basically just ASI?

Dwarkesh Patel

对。

Yeah.

John Schulman

我说 3–4 年。AI 显然更受关注。它是更难的事之一，但大量精力砸进去。对 AI 也不是最难的事之一，因为它大量涉及代码和数学，模型很擅长。涉及 3D、空间、物理的东西会久一点。若是机械工程之类、现在关注不多，可能更久一点。

I would say 3–4 years. AI is obviously getting more attention. It’s one of the harder things, but a lot of energy is being put into it. It’s also not one of the hardest things for AI, because it involves a lot of code and math, which models are really good at. Things that involve 3D and spatial and physical stuff will take a little longer. Mechanical engineering or something not getting the most attention now might take a little longer.

Dwarkesh Patel

但也包括因领域本性数据相对少、得现场学的领域。比如得在台积电当工程师上达到超人类。

But it also includes fields where there’s relatively little data by nature of the field, and it has to learn that data on the fly—for example become superhuman at being an engineer at TSMC.

John Schulman

那你得假设能给 AI 同样的入职材料。然后关于更长视野学习的某些东西必须被解决。

So you would have to assume you can give the AI the same onboarding material. Then something has to be solved about longer-horizon learning.

Charlie O’Neill

我说 5 到 10。

I’d say 5 to 10.

Dwarkesh Patel

所以你基本上觉得自动化 AI 研究是 ASI-complete？

So basically you think automating AI research is ASI-complete or something?

Charlie O’Neill

对，我觉得是。世界上有太多东西：即便你有模型外的某种记忆系统，即便上下文长度再长一点，仍有根本性的东西——即便你能自己查信息或写笔记，今天也需要超过一百万 token 的上下文。

Yeah, I think so. There are so many things in the world where, even if you have some memory system external to the model, and even if context length grows a little, there are just fundamentally things where, even if you could research the information or write notes yourself, you’d need more than a million-token context window today.

Beren Millidge

对实验室聚焦的东西，我大致同意 5 年区间。但会有长尾：AI 理论上可以出去学，但没人费心做、算力也没分过去。要字面压过每一个人类专家，可能更久。

I kind of agree on the 5-year range, at least for the stuff labs are focusing on. But there’s going to be a long tail of stuff the AI could theoretically go learn about, but no one has bothered and compute hasn’t been allocated. That might take longer for literally every single human expert.

Dwarkesh Patel

抱歉，这里我也包括：学新领域要和人一样快。

Sorry, but by this I also included the ability to learn a new domain as fast as a human.

Beren Millidge

我认为那不一定必要，因为 AI 会有远大于任何人的经验。

I think that’s not necessarily necessary, because the AI will have vastly greater experience than any human.

Dwarkesh Patel

非常感谢各位。这种让不同专家一起争论、辩论、讨论的形式很棒，非常有收获。

Thanks so much for doing this, guys. I feel like this was a great format for getting different experts to disagree and debate and discuss things together. It was very productive.

John Schulman

谢谢邀请我们。

Thanks for having us.