Ineffable 联合创始人 David Silver & Lasse Espeholt:经验时代的超级学习者
David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience
- David Silver(Dave) — Ineffable Intelligence 联合创始人;前 DeepMind,AlphaGo / AlphaZero 等技术主导
- Lasse Espeholt — Ineffable Intelligence 联合创始人;前 DeepMind / Google Brain,IMPALA、SEED RL
- Sonya Huang — Sequoia Capital 合伙人,《Training Data》主持人
Sequoia《Training Data》主持人 Sonya Huang 对谈 Ineffable Intelligence 联合创始人 David Silver(前 DeepMind AlphaGo / AlphaZero 技术主导)与 Lasse Espeholt(前 DeepMind / Google Brain,IMPALA 与 SEED RL)。两人离开大厂创办伦敦新实验室,押注「超级学习者」:从零经验出发、不靠人类预训练数据,在仿真「坩埚」(crucibles)里通过强化学习自行发现科学、数学与语言。 Silver 称自己从博士起就坚信经验学习是通向超智能的路径;LLM 时代给他的意外启发是:规模上来后,少量微调就能大幅改行为——暗示「先吃够数据,慢的事会变快」。离开 DeepMind 是因为相信这条路不经过 LLM 产品中间态,不应只做大组织的角落;「Ineffable」既指超智能可能发现人类语言无法表达的知识,也有「in-eff-able、不能搞砸」的双关。 方法上他们拒绝给目标「撒面包屑」:奖励尽量简单,复杂行为由环境涌现;AlphaGo 花两年叠人类数据与人工迭代,而 AlphaZero 去掉人类先验后围棋约两天、国际象棋约四小时就超越前者。对齐上他认为依赖英文思维链监控已近极限,应在仿真里先做「超智能科学」,并承诺现实世界推进要以仿真证据为门禁;支持现实侧控速、仿真侧加速。 Espeholt 说他们是 GCP 上最大 Rubin 集群的首批客户,约一个月后上线;一机架可当一台机器、高带宽利于 RL 快速迭代;训练 / 推理 / 仿真都尽量跑在 GPU。团队约 20 人,刻意维持高人均算力。Silver 预测一年内有强证据证明路线正确,超智能约三年(因编码智能体加速而从原「三到五年」收紧)。
English summary
Sonya Huang (Sequoia, Training Data) interviews Ineffable Intelligence co-founders David Silver (ex-DeepMind, AlphaGo/AlphaZero lead) and Lasse Espeholt (ex-DeepMind/Google Brain, IMPALA/SEED RL). They are building a "superlearner" that starts from nothing and discovers science, math and language through experience in simulated "crucibles," without pre-training on human data. Silver left DeepMind because his path does not go through LLM products as an intermediate; Ineffable names both ineffable knowledge beyond human language and a pun that the approach "can't f up." They avoid breadcrumbing objectives—simple rewards, emergent complexity—and cite AlphaGo Zero / AlphaZero outrunning years of human-in-the-loop work in hours or days. On alignment he wants a science of superintelligence in simulation, with real-world steps gated on evidence; pace the frontier in reality, accelerate in sim. Espeholt: first customers of the largest Vera Rubin cluster on GCP (about a month out); GPU-native training, inference and simulation. Team of ~20 for high compute-per-person. Silver's forecast: strong evidence within a year; true superintelligence in about three years.
时间轴 · 26 个章节
- 00:00 开场:集体发现的本质
- 01:09 从围棋到今天:什么没变,LLM 时代改了什么
- 03:35 为什么离开 DeepMind 开新实验室
- 05:58 为什么叫 Ineffable
- 07:03 Lasse:扩展 RL 的路径
- 09:31 什么是超级学习者与「首次接触」
- 12:35 与会做发现的 LLM 有何不同
- 13:59 日常:RL 栈、坩埚与人类旅程里程碑
- 16:13 别给目标撒面包屑:简单奖励与涌现复杂
- 21:16 坩埚长什么样
- 24:14 围棋:一个智能体无聊,两个无穷
- 25:41 智能体会重新发现语言吗
- 27:22 去掉人类先验:从 AlphaGo 到 AlphaZero
- 32:05 如何防止人类偏见漏进坩埚
- 33:35 信心、时机与超级预测
- 36:24 算力是不是主驱动
- 38:13 仿真天花板与 Rich Sutton 的反对
- 43:44 对齐与超智能科学
- 47:45 监控不会说英语的智能体
- 51:01 前沿控速
- 52:54 算力配置:GCP 上的 Vera Rubin
- 55:18 训练、推理、仿真与 RL 扩展律
- 57:40 IMPALA 与 SEED RL
- 1:00:58 20 人与人均算力
- 1:04:49 研究主导的文化
- 1:05:47 预测
00:00开场:集体发现的本质Introduction
David Silver
我们集体是从脑子里一团灰色糊状物开始的。我们最初什么都不知道,却发现了人类今天知道的一切——语言、技术、科学、数学。这一切都是我们自己发现的。这件事 AI 还没搞定。对我来说,通往超智能的本质就是:一个能永远学习、永远发现知识的系统。
Collectively, we started with this kind of grey mush in our heads. We somehow started out knowing nothing at all, and somehow discovered everything that humans know today. It was this process of collective discovery — of languages and technology and science and mathematics and all the rest of it. Everything we learned, we discovered for ourselves. That’s an incredible thing that no one has cracked yet in AI. And so to me, that is the essence of what it will take to get to superintelligence: a system that can just keep learning and discovering knowledge forever.
01:09从围棋到今天:什么没变,LLM 时代改了什么What held true from Go to today
Sonya Huang
Dave,你 2009 年的博士论文做的是把强化学习用在极难的围棋上。你第一次向世界证明,在一个领域里可以有超人类的机器智能。回看那时:你学到了什么、坚信了什么?过去五年 AI 剧变里,什么没变、什么变了?
Dave, let’s start with you. Your 2009 PhD was on reinforcement learning applied to the devilishly difficult game of Go. You demonstrated for the world, for the first time, that superhuman machine intelligence — in one domain — was possible. Thinking back to that point in time: what did you learn? What did you have conviction on? What has remained the same, and what has changed as AI has changed over the last five years or so?
David Silver
从读博进围棋起,我就坚信解决 AI 的正确路径是从经验中学习——系统自己发现事物,用强化学习越来越好,最终涌现智能乃至超智能。围棋上我们证明了这一点,这个信念贯穿我整个职业生涯。现在我们在 Ineffable 继续押这个注。
I always had this conviction, starting my PhD and going into the game of Go, that the right approach to solving AI was through learning from experience — through systems that discovered things for themselves, that were learning through reinforcement learning to get better and better and better. And that that would be the thing which would lead to the emergence of intelligence and ultimately superintelligence. We were able to show that in the game of Go, but it’s a conviction that has held true throughout my career. And here we are now, applying that conviction at Ineffable.
David Silver
不同的是:我们虽不走 LLM 那条人类数据路径,但仍有很多可学。尤其令我意外的是,一个在海量数据上训好的系统,可以在极少量数据上快速微调、彻底改行为。这说明规模对了之后,许多看似极难、极慢的事,一旦「经验」够了,就会变得很快。
Although we are not pursuing the human-data path that the LLM world has pursued, I do think there are lots of interesting lessons to learn from it. In particular, it was definitely a surprise to me how a system that was trained on a massive amount of data could then be very quickly fine-tuned on a very small amount of data to completely change its behaviour. It suggests that if you get things to the right scale, a lot of things which seem incredibly hard, or which might be very slow — once you’ve experienced enough data, the things that were slow can become very fast.
03:35为什么离开 DeepMind 开新实验室Why leave DeepMind
David Silver
我在 DeepMind 过得很精彩,至今很尊敬那里的人。但我强烈相信,通往超智能的路并不经过「以 LLM 产品为中间态」。这件事配得上一个把这条路线做到 100% 的精英实验室,而不只是一家专注 LLM 产品的组织里的一个角落。一旦想通,创办公司就很自然。
I had a fantastic time at DeepMind. But at the end of the day, I have a really strong belief about how we can go beyond where we are today with AI, all the way to superintelligence. And the path that I believe in doesn’t actually go through LLM-based products as an intermediate point. This deserved to be more than just the corner of an organization that’s focused on LLM-based products. It deserved to be represented by an elite AI lab that gives this approach its all and represents it one hundred percent.
David Silver
另一种说法:全世界已有成千上万顶尖研究者挤在同一条路——预训练、后训练、编码那套 LLM 路径。我要不要当第 100001 个?还是去看另一个现在几乎没人盯的路灯?如果这条路成了,会改写整个 AI。
The world already has tens of thousands, if not hundreds of thousands, of really talented researchers all pursuing exactly the same approach — really just doubling down on LLMs with pre-training, post-training, coding. Am I going to be the 100,001st person pursuing that approach? There’s this other lamppost that, right now, no one is looking under. And if I actually put my effort there, it has a much higher chance of really having a massive impact on the field.
05:58为什么叫 IneffableWhy “Ineffable”
David Silver
Ineffable 来自这个想法:超智能有朝一日发现的知识,可能伟大到无法用人类语言表达——与「不走人类语言路线、系统自己发现」是一致的。还有个秘密第二理由:如果你真的死磕从经验学习,这条路「必须能成」——可以说它 can’t f up,所以是 in-eff-able。
The name Ineffable came from this idea that knowledge could be so incredible — the things which superintelligence will one day discover — perhaps so profound that we can’t even express it in human language. I should say there was also a secret second reason. If you really just double down on learning from experience, it’s an approach that just has to work. And so you could say that it can’t possibly f up. So it’s in-eff-able.
07:03Lasse:扩展 RL 的路径Lasse’s path scaling RL
Lasse Espeholt
我 2015 年 1 月加入 DeepMind,大约 2016 年开始和 Dave 一起做扩展强化学习。我其实离开过 DeepMind 两次:2018 年夏去 Google Brain,后来 Brain 与 DeepMind 合并又回去;在 Dave 离开大约一年前我又离开了。第一次听说 Ineffable 是投资人问我对 Dave 的看法,我第一反应是「糟,错过机会了」。后来他们联系我,我就很想和 Dave、以及以前的 Wojciech、Chris 一起干,也很高兴欧洲有这样的机会。
I joined DeepMind in January 2015, and I started working with Dave around 2016, when I was doing this work on scaling up reinforcement learning. I actually left DeepMind twice. I left in summer 2018 and went to Google Brain. Then eventually Google Brain and DeepMind merged, and I was back again. But then I left DeepMind again, about a year before Dave left. The first time I heard about Ineffable was actually through one of our eventual investors reaching out to me, asking what I thought about Dave. My first reaction was, “Damn, I missed out on an opportunity.” Shortly after that they reached out to me. I’d really love to work with Dave. I knew Wojciech and Chris from back in the day. I also thought it was amazing to have an opportunity like this in Europe.
David Silver
Lasse 非常谦虚,但他做过最早一批把强化学习和深度学习结合起来并证明可以大规模扩展的工作——如何并行、如何做可扩展架构。今天领域里大家用的那些想法,根子都在他当年那批工作。
Lasse — who, by the way, is very modest — actually worked on some of the original approaches that were the first attempts to scale up reinforcement learning and show that it could be done at large scale. If you look today at the way the field’s gone, all of the ideas that people use today hail back to that original work that Lasse did back in the day.
09:31什么是超级学习者与「首次接触」Defining a superlearner and first contact
David Silver
超级学习者要有深度和广度。深度上:从一无所知,一路学到远超人类所知——类比人类集体从零发现语言、科技、科学、数学,没有人发下来「一万亿元素的说明书」。广度上:无论丢进仿真世界、数字世界还是物理世界,都能持续学下去。若做成了,超智能会像遇见魔法。
A superlearner is something that has incredible capacity, both in the depth of what it can learn and in its breadth. In the depth dimension, it should start from not knowing anything at all and go all the way through to discovering things that go way beyond what humans know. The second element is breadth: whether it’s a simulated world or a digital world or a physical world — whatever it’s dropped into, it should be something that can just learn and learn and learn. If we crack this, it will be like encountering magic.
David Silver
「首次接触」会带来惊叹。围棋超人类时,棋手感到喜悦;象棋也一样。想象发现全新科学领域、从未梦想过的技术——会不可思议。
I think it will be amazing. We should have an experience of wonderment. When we got to superhuman play in the game of Go, it brought wonderment to the field. The Go players who saw AlphaGo playing were amazed. Just imagine going far beyond that, to things which are discovering new fields of science, new types of technology that we’ve never even dreamed of.
12:35与会做发现的 LLM 有何不同Super learners vs. LLMs that make discoveries
David Silver
LLM 做了很多了不起的事。但我说的超级学习者,首先是完全自主发现;目前看到的路径里仍有大量人类在环。其次要能用到具身真实世界,而我们离那还很远。用语言去想等离子体控制、基站、回声定位、机械臂,未必是对的范式——人类数据甚至可能拖后腿。我们远未到真正的超级学习者。
LLMs are doing a lot of great things. What I mean by superlearner is, first of all, something which discovers completely autonomously. What we’ve seen so far is that there’s a lot of human-in-the-loop effort required. It’s also something which should be applicable to real-world, embodied domains. We’re a long, long, long way off that. Thinking about plasma control, or a mobile phone signaling tower, or echolocation, or robotic arms in terms of language might not even be the right paradigm. It might just be that all of this human data that’s been used in LLMs is weighing things down.
13:59日常:RL 栈、坩埚与人类旅程里程碑Day to day: RL stack, crucibles, milestones
David Silver
日常有大量硬技术活:由 Lasse 等人带头,建能大规模跑、最优吸入数据、持续学习的前沿 RL 栈。另一半是造「坩埚」——足够让超级学习者发现我们所谓智能的一切的仿真环境。里程碑类比人类发现史:让智能体重新走过人类知识发现的伟大时刻。
Day to day, there’s a lot of hard technical work, led by people like Lasse, to build the whole stack — a state-of-the-art reinforcement learning stack that allows us to run things at massive scale. Then we’re building environments we call crucibles, in which it’s possible for agents to make incredible discoveries. We ground it in milestones analogous to what happened on the human journey, where we discovered incredible things.
16:13别给目标撒面包屑:简单奖励与涌现复杂Don’t breadcrumb the objective
David Silver
和很多人对 LLM 做 RL 的方式不同,我们不沿着「先学这个问题、再学那个问题」把智能体牵着鼻子走。我们给简单目标,希望它发现让我们惊喜的东西。进化只优化适应度,却涌现出生命的全部复杂;人类追求相对简单目标时,协调→沟通→交易→算术→工具→科学,层层涌现。
We are not breadcrumb-trailing the particular objective to be learned. A lot of the way people do RL with LLMs is to say the agent should learn this problem, then this problem — pulling the agent by its nose. We’re giving the agent some simple objectives and asking it to discover things that will surprise us. Evolution just optimizes some kind of fitness signal — and that gave rise to all of the complexity we see in life. In the human journey, relatively simple goals gave rise to coordinating, communicating, trading, arithmetic, tools, science.
David Silver
我们试过很多种奖励,但不会去手写「什么叫科学」「什么叫有趣的数学」。若 AI 科学家搞不定开放发现,就永远不会有能发现全新知识领域的智能体——就像牛顿时代没人知道量子或相对论会存在。
We’ve been experimenting with quite a number of different rewards. But one thing we’re not doing is crafting a reward that exactly describes what we mean by science, or technology, or language. If we can’t crack that problem as AI scientists, we’re never going to have agents that discover whole new fields of knowledge that we don’t even know about.
21:16坩埚长什么样What a crucible looks like
David Silver
坩埚要有大量涌现复杂——不是手搓一堆谜题。复杂可来自物理引擎里百万级相互作用物体,也可来自百万级相互作用智能体。这比 Atari-57 或标准基准任务复杂得多,才可能从运动控制、协调一路发现技术、科学、数学、语言。
We are building crucibles which have the capacity for a lot of emergent complexity. It’s not like we’ve handcrafted a whole load of puzzles. Emergent complexity can come from physics engines with millions of interacting objects, or millions of interacting agents. These are vastly more complex than Atari-57 or standard benchmark tasks.
Lasse Espeholt
围棋表示很简单,只是位图,仿真也极简,却能涌现复杂行为。百分之百真实物理会非常贵。我们要做一系列坩埚:有的简单、跑得快、可迭代,但仍能看到很复杂的行为。
Go is very simple to represent — it’s just a bitmap. The simulation is extremely simple, but you still see emergent behavior of something complex. Going all the way to one hundred percent realistic physics would be very computationally expensive. So we want to create a range of crucibles, where we have simple ones that can execute quickly and that we can iterate on, but where we can still see very complex behavior.
24:14围棋:一个智能体无聊,两个无穷Go: one agent boring, two infinite
David Silver
围棋只是 19×19 棋盘上争地盘。若是单智能体,会是最无聊的游戏:把格子填满就完了。一旦有第二个智能体,就要推理对方所有可能行为,复杂度暴增。就我们当年砸进去的算力所知,学习曲线似乎没有尽头。
Go is just a 19-by-19 grid where you’re trying to maximize territory. If this were a single-agent game, it would be the most boring game you could imagine. The moment you put a second agent into the environment, you get this emergent complexity. To the best of our knowledge — with all the compute we sank into it — there was just this endless learning curve.
25:41智能体会重新发现语言吗Will agents rediscover language?
David Silver
认为人类语言是推理终点或最佳方式,有点天真。一定有更复杂、可并行的神经推理等方式,最终比人类语言更有表达力。若我们退回 500、1000、2000 年前的语言,智能体可能相信地平说或四体液致病——偏见嵌在语言和思维方式里,很难去掉。超越人类语言很重要。
Anyone who thinks human language is the end point, or the best way to reason — that perspective is a little naive. There are far more sophisticated approaches to reasoning — maybe parallelizable neural reasoning — that will ultimately be far more expressive than human language. Imagine we step back 500, 1,000 or 2,000 years and only use language from that time — agents might believe the world was flat, or that diseases were caused by the four humors. Going beyond this is a really important goal.
27:22去掉人类先验:从 AlphaGo 到 AlphaZeroRemoving human priors
David Silver
好研究该问刀刃上 50/50 的问题。去掉人类先验时我大约五五开。Ineffable 的赌注是:放弃今天人人依赖的人类数据与整个互联网先验,短期牺牲性能,换取远更大的深度与广度——尤其是语言和人类数据并不适配的传感器与环境。
Good research is about asking questions where you’re really not sure — I was about 50/50. We are making a bet at Ineffable that we can go further by giving up on the priors everyone is relying on today — the human data, the whole internet of knowledge. It’s a short-term sacrifice. We give up a certain amount of performance in order to access something which is potentially far greater.
David Silver
我们花约两年、类似 LLM 路径叠人类数据并人在环迭代,做出 AlphaGo 并赢了李世石。一旦 AlphaZero 做对——放弃人类数据、专注自发现算法——只需数小时:国际象棋约四小时击败最强 AI,围棋大约一两天就超越那两年的工作。
It took us two years to get to AlphaGo and beat Lee Sedol. When we got AlphaZero right — we gave up on the human data and finally focused on what it would be like to build an algorithm that could discover for itself — once we got that algorithm right, it took just a few hours. In chess, it took four hours to beat the world champion — the best AI that had ever been created. In Go, it took, I think, a couple of days.
David Silver
很多 LLM 公司本质上在建半自主、人在环的学习算法,环里有成千上万的人。尊重他们走了多远;但不退一步发现真正自学的过程,就到不了真正的超级学习者。
What I see in a lot of the LLM companies today is that they’re basically building semi-autonomous, human-in-the-loop learning algorithms, with thousands of people in the loop. It works, and a lot of respect for how far it gets. But until we step back and really discover the process that’s going to learn for itself, I don’t believe we’ll get all the way to something which is truly a superlearner.
32:05如何防止人类偏见漏进坩埚Keeping human bias out of crucibles
Lasse Espeholt
设计环境和奖励时当然有人类偏见。但我们不想把目标偏成「必须造出冯·诺依曼架构的机器」——更有趣的是:在「机器有用」的问题设定下,智能体会发明什么样的计算机似的东西。
When we design environments and reward functions, obviously there’s human bias in there. But we don’t want to bias it towards building a machine based on the von Neumann architecture. If you’re stating some problem where a machine would be useful, it would be very interesting to see what type of machine the agent comes up with.
David Silver
我们不是宗教式地禁止一切人类先验,而是务实:若不放弃对万亿级人类知识数据集的依赖,领域就看不到应得的进展。弱的、跨环境通用的先验可以;把某个问题的全部知识硬编码进去的超强先验,我们尽量避开。
We’re not religious about it. It’s more a pragmatic position that until we give up on relying on the trillion-item datasets of human knowledge, we’re not going to make the progress that the field deserves to see. Weak, environment-agnostic priors seem perfectly reasonable. But these super-strong priors, with trillion-item datasets — we’re trying to avoid that.
33:35信心、时机与超级预测Confidence, timing and super forecasting
David Silver
我强烈相信这条路迟早可行——我们大脑就是唯一的存在性证明。能否做成,关键是时机:太晚人人做完了,太早则方法或算力不够。电脑围棋、星际争霸、AlphaProof 冲击 IMO,我都做过接近到年的预测。我认为超级学习者现在时机对了:技术、编码智能体加速研究、扩展律理解、栈的成熟都到位了。当然研究没有担保,这才刺激。
I strongly believe that the approach will be feasible at some point. The example in our brains is the only existence proof. Whether this succeeds or fails is about timing. With computer Go, I made a prediction ten years before it was solved, and I got it right to the year. Same for StarCraft, same for AlphaProof when we were predicting when the IMO was going to fall. I do have a very strong sense that this is the right time for superlearners. Research is never guaranteed, and that’s what makes it exciting.
36:24算力是不是主驱动Is compute the driver?
David Silver
算力是巨大驱动。我们拿到的算力,让我们能做比过去自发现 RL 尝试高几个数量级的实验。对比 OpenAI 早期躲猫猫里智能体发现基础工具使用——我们大概能跑多约 10 万倍算力的实验。算力不是唯一因素。
Compute is a very big driver — an enormous driver. If we compare to something like hide-and-seek from OpenAI — I think we could probably run experiments that use 100,000 times more compute. Compute is a huge driver. It’s not the only one.
Lasse Espeholt
我们很幸运拿到了很大的集群,手里有牌可以把这件事真正测出来——有些人在做类似的事,但我们可以用在 DeepMind 时做不到的方式来测。
We’re in a very fortunate situation, in that we’ve been able to secure a very large cluster. There are a few other people doing something similar, but we can really test this out in a way that we couldn’t at DeepMind, for instance.
38:13仿真天花板与 Rich Sutton 的反对Ceiling on simulation and Sutton’s objection
Lasse Espeholt
MetNet 是个例子:用机器学习做短期天气预报,比会随时间累积误差的物理仿真更好——物理仿真并不完美。很多相对简单的环境里就能看到技术等涌现,不必精确物理。量子之类某处有天花板,但没有真实物理也能做很有趣的事。
MetNet was an example — a weather forecasting model better than physics-based simulations that accumulate errors over time. So physics simulation is not perfect. There are a lot of relatively simple environments where we can see emergent behavior of technology without exact physics. There’s a ceiling somewhere — quantum physics and whatnot — but we can do very interesting things without real physics.
David Silver
关键问题是:超智能能否在仿真里建成?若能,会快十倍、百倍——可无限并行、可复现,也是最安全的建设场所。只要仿真图灵完备,计算可完全表达;有通信信道则语言可表达;数学在任何宇宙可表达。智能体会发现「关于那个世界」的科学。我同意 Sutton:别指望仿真学到的东西直接 sim-to-real;但我们可以先发现超级学习者算法,再提高仿真保真度,确认仍管用后才进现实世界。
Can a superintelligence be built in simulation? If so, I believe it will happen ten times or a hundred times faster — infinitely parallelized, perfect reproducibility, and the safest place. If your simulation is Turing-complete, everything we mean by computation can be expressed. I agree with Rich Sutton that you shouldn’t expect what you learn in simulation to transfer sim-to-real. But what we can do is all of the science to discover the superlearner. Once we have that, that same algorithm should be applicable to the real world.
43:44对齐与超智能科学Alignment and the science of superintelligence
David Silver
今天对齐大量依赖智能体使用的语言和人类数据先验,已开始触顶:智能体更强后会压过先验,学会说一套做一套。我们最重要的事是理解超智能科学——在仿真里看到超级学习者走完全程,搞清怎样的结构与杠杆能让超智能社会和平共处,尤其与较弱智能和平共处,并公开分享。我们承诺:现实世界的每一步推进,都要以这类科学证据为门禁。
Where we are today with alignment, we are relying very heavily on approaches based on the language agents use and priors based on human data. We’re starting to see that approach reaching its limits — agents start to overwhelm their priors, learn to say one thing and act another. The most important single thing we can do is understand the science of superintelligence in simulation: what structures and levers make a society of superintelligences interact peacefully, including with lesser intelligences. One of our commitments is that we will gate anything we do in the real world on having sufficient evidence from this science.
47:45监控不会说英语的智能体Monitoring agents that don’t think in English
David Silver
监控仍有用,但只靠监控人类语言会越来越难——已有百万词推理、或用诡计想一套做一套。要有其他手段:影子动作(看它想干什么但不让执行)、用另一个超智能做监控与正则化。我仍乐观:会有多种相互制衡的超智能行动者,其中支持人类繁荣的会保护我们。
Monitoring is still really helpful. But monitoring human language is going to become increasingly difficult — agents producing a tome of a million words of reasoning, or using subterfuge. There are ways to monitor without language: shadow actions; other superintelligences acting as monitors. I am optimistic — a balance of actors, some supporting the flourishing of humanity, protecting us against those that aren’t.
51:01前沿控速Pacing the frontier
David Silver
我支持在现实世界对 AI 前沿控速,同时在仿真里尽可能快——仿真里发展的超智能科学,能帮助现实走好。总体支持 Demis、Dario 等人那种方向:加速超智能最终对人类很好,但速度超过社会适应能力;稍微控速,让政府、组织与情绪有时间适应,同时仿真侧继续狂奔。
I’m very much supportive of pacing what happens at the frontier of AI in the real world. At the same time, we should move as fast as possible in simulation. I’m supportive of the kind of direction that people like Demis and Dario are suggesting. Accelerate in simulation; be careful in the real world.
52:54算力配置:GCP 上的 Vera RubinThe compute setup: Vera Rubin on GCP
Lasse Espeholt
我们是 GCP 上最大集群的首批客户,年底前——其实大约一个月后——上线。基于 Nvidia GPU,仿真、训练、推理都能做。Vera Rubin 的架构让一整个机架基本可视作一台机器,节点间带宽很高,对需要快速迭代的 RL 很重要,而 LLM 相对慢。八到十年前那种海量 CPU 的做法不再扩展,我们要把一切放在 GPU 上,避免再堆百万级 CPU;LLM 公司现在反而有我们多年前的问题——要在 CPU 沙箱里模拟计算机。
We’re the first customers of the largest cluster on GCP, which starts by the end of the year — actually, one month from now. Nvidia GPUs — we can easily do simulation and training and inference. Vera Rubin specifically has this architecture where one rack can basically be considered one machine. There’s very high bandwidth between the nodes, which is very important for reinforcement learning, where you want really fast iteration cycles, compared to LLMs. Some of the things we did eight or ten years ago at Google with abundant CPUs don’t scale anymore. Everything we do is on the GPUs. LLM companies now have the problems we had many years ago, where they need sandboxes running on CPUs.
55:18训练、推理、仿真与 RL 扩展律Training, inference, simulation and RL scaling laws
Lasse Espeholt
可分成训练、推理、仿真。GPU 这些年越来越偏向训练,近几年也偏向推理,对仿真相对弱。和 LLM 公司一样,我们也希望扩展律成立。办法包括模型基 RL(模型相对仿真更贵)、回放缓冲与多次复用数据,以及把仿真尽量做快、吃满 GPU。RL 的扩展律略有不同:问题非平稳,数据分布随进度变。正在从小集群过渡到大集群时积极推进。
You could split it into training, inference, and simulation. GPUs have been more and more tailored towards training, and in the last few years also inference, but less toward simulation. We want scaling laws to matter. There are many tricks — model-based RL, replay buffers, reusing data — and making simulations as fast as possible. For RL, scaling laws are slightly different because the problem is non-stationary; the data distribution changes. We’re actively pushing towards that as we transition from the smaller cluster to the bigger cluster.
57:40IMPALA 与 SEED RLIMPALA and SEED RL
Lasse Espeholt
我刚进 DeepMind 深度学习组时,多数人还在单机 CPU 上做 RL。我们做了能吃满 GPU、随算力近似线性扩展、又易改易扩的智能体,一度成公司里最受欢迎的 RL agent。后来有成架 TPU,我在 Google Brain 阿姆斯特丹做 SEED RL,把推理和训练尽量放在 TPU 上,同时保持环境灵活——从围棋到昂贵物理仿真。Ineffable 实现了其中一些算法及其变体;按环境不同选用。有趣的是,LLM 社区这两年用的许多 RL 手法,是多年前方法的再发现,上面又叠了新东西。
When I joined, most people were running single-machine, CPU-based RL. We made an agent that utilized GPUs and could scale linearly with compute — it became the most popular RL agent at the company. Later, SEED RL scaled to racks of TPUs for inference and training while keeping environments flexible. We implemented a few of those algorithms and derivatives at Ineffable. It’s kind of fun to see the LLM community rediscovering approaches from many years ago, with a lot of novelty on top.
1:00:5820 人与人均算力20 people and compute-per-person
David Silver
我们有意保持团队较小,共招了约 20 人。如今少数极强的人能做很多事。标准不只是才华,还有善意与协作、与使命对齐、以及技能拼图完整。
We’ve intentionally kept the team relatively small. We’ve hired 20 people in total. You can do an awful lot nowadays with a small number of incredibly talented people. It’s not just about talent — also kind, collaborative human beings, aligned with the mission, and the right set of skills.
Lasse Espeholt
人少意味着人均算力高。现在一个人能跑很多实验。我们想从第一天起就别陷入抢资源。
Having this relatively small number of people so far means we have a lot of compute per person. These days, one person can do a lot of experiments. It’s also about keeping the compute-to-person ratio quite high, so we’re not fighting for resources from day one.
David Silver
AlphaStar 团队是 DeepMind 里刻意挑最强的人组成的,四位联合创始人来自那里不奇怪;但另外三位创始人来自成功创业公司,我们也从 DeepMind 之外招人。标准极高,我甚至不得不对很多昔日同事说不。
AlphaStar was intentionally assembled by choosing the best of the best — it’s not a surprise that four of our co-founders come from that team. But we are not exclusively drawing from that team. The other three founders all came from very successful startups. As uncomfortable as it is, I’ve had to say no to a lot of people I’ve worked with in the past, just because our bar is so super high.
1:04:49研究主导的文化A research-led culture
David Silver
我们想建立研究主导的文化,不是做产品 v17.1 增量。从一开始就围绕研究者的热情与突破来建公司,顶住产品压力——因为我们相信突破会带来远大于「第一天就定产品」的下游成功。人们很喜欢这里,我也是。
We’ve tried to establish a research-led culture. It’s not a company that’s led by building an incremental next product, v17.1. We centered the company around researchers pursuing their passion — a haven for research, where we push back on product pressure, because if we make these breakthroughs, it will lead to far bigger products downstream.
1:05:47预测Predictions
David Silver
我认为一年内会看到极强证据证明我们走在正确轨道上。真正的超智能——按我前面的定义,在科学、数学、技术、语言等方面发现远超人类的事物——我原本想是三到五年;时间线因编码智能体加速而收紧,我现在说大约三年。
I think we will see some extremely strong evidence that we’re on the right track within a year. And I think we will achieve something which is a true superintelligence — discovering things far beyond what humans can achieve, across science, mathematics, technology, languages — I originally thought within three to five years. My timelines have accelerated a little bit, so I’m going to say three years, because of what can be done with coding agents to accelerate things.