Google 的 Blaise Agüera y Arcas：AGI 已经到来，意识是多智能体里的自我建模

Blaise Agüera y Arcas

我认为人工智能首先就是智能，是真正的智能。从这个意义上说「人工」这个词有点不幸，我们用它来形容假钻石，比如立方氧化锆，表示它不是真货。我倒希望当初用「合成智能」这样的词：合成钻石是在实验室而不是地下造出来的真钻石，这个类比更接近我们今天的 AI。

I think that artificial intelligence is first and foremost intelligence. It’s real intelligence. In that sense, the use of the term artificial is a little bit unfortunate. We use that term for things like fake diamonds that are actually cubic zirconia, to mark that we don’t mean the real thing. I kind of wish that we had gone with a term like synthetic intelligence. A synthetic diamond is a real diamond made in the lab rather than underground, and I think that’s a closer analogy to what we have with AI.

Blaise Agüera y Arcas

多年来一直有争论：AI 表现得有智能，是说明它真的有智能，还是只是在模仿？我认为这基于对智能相当深的误解。如果智能关乎行为背后的功能，那这就像说「对计算器的模拟不是计算器」，这当然不对。计算器的功能完全在于那种交互；它不像叉车，可以真的举起东西，也可以假装举起。

There’s been a lot of debate about when AI behaves intelligently, does that mean it is intelligent or only that it’s imitating intelligence? I think that’s based on a fairly deep misunderstanding of what intelligence is. If intelligence is about the functionality behind behavior, then that would be like saying if I make a simulation of a calculator, it’s not a calculator, which of course isn’t true. What a calculator does is entirely about that interaction. It’s not like a forklift that can either really lift something or pretend to lift something.

Blaise Agüera y Arcas

过去几年我们开始做人工生命方向的研究，不把生命看成某种特定化学基础上的东西，而是更广义的现象。如果遇到《挽救计划》里的外星人 Rocky，大多数人会说它也是活的。那我们指的是什么？我这几年得出的结论是：按功能来定义生命，它也包含大量技术，我们整个技术世界都是生命世界的一部分，把两者分开其实才是「人工」的。

We’ve started to work in the last few years in the field of artificial life, thinking about life more broadly, not as something about a specific chemical basis the way all biological life on Earth is, but as a broader phenomenon. If we were to encounter an alien like Rocky from Project Hail Mary, most of us would say Rocky’s alive as well. What would we mean by that? What I’ve arrived at over the last few years is the sense that life defined functionally includes a lot of technology as well; our whole techno universe is part of the life universe, and separating that is in fact artificial.

Blaise Agüera y Arcas

要区分哲学家所说的现象意识，也就是「作为一个人或一只蝙蝠是什么感觉」，和意识的功能。有些思想家认为意识没有功能，是副现象，可以想象一个和你我行为一样却没有意识的存在，也就是 David Chalmers 著名的哲学僵尸。我逐渐相信这不是一个自洽的概念，意识是有原因的。真正说服我的是我们 Paradigms of Intelligence 团队在「嵌入式智能体」问题上的工作。

It’s useful to distinguish between what philosophers call phenomenal consciousness, the feeling of what it is like to be a person or a bat, and the functional role of consciousness. Some thinkers have claimed consciousness has no functional role, that it’s an epiphenomenon, so you could imagine a being that behaves the same way as you or me but is not conscious: David Chalmers’ famous philosophical zombie. I have come to believe that’s not a coherent idea, that there’s a reason for consciousness. The thing that really convinced me is work from my team, Paradigms of Intelligence, on what we call embedded agents or embedded agency.

Blaise Agüera y Arcas

单智能体的情形，比如 DeepMind 解 Atari 游戏，一个智能体玩游戏、最大化分数就能成为超人玩家，这类问题我们大致知道怎么处理。多智能体的情形——一群智能体不仅回应环境，还互相回应——一直是机器学习里长期的大难题。我们研究它很久了，因为每个真实环境都是多智能体的，没有其他智能就没有智能。即使每个智能体只想最大化自己的分数，只要环境里合作和竞争并存，也就是混合博弈，你就不仅要对他人建模，还要对自己建模，认识到自己和他人的相似之处，并对自己和他人都保持认知上的不确定性。

In a single-agent setup, like DeepMind’s solution to Atari video games, an agent plays a game and can become a superhuman player by maximizing the score. We kind of know how to deal with the single-agent case. The multi-agent case, where a bunch of intelligent entities are responding not only to the environment but to each other, has been a big, long-standing problem in machine learning. We’ve been working on it for a long time because every real environment is multi-agent; there is no intelligence without other intelligences. Even if all of the agents just have some selfish score to maximize, if there is some combination of cooperation and competition possible, what’s called a mixed-sum game, it turns out you need to not only model others but also model yourself, recognize the similarity between yourself and others, and have epistemic uncertainty about yourself and about others.

Blaise Agüera y Arcas

通常玩游戏时，心智在游戏之外，不用考虑自己的心智。但当其他行动者也在游戏里，他们就成了环境的一部分，你的模型必须扩展到包含他们的心智，而他们也在对你建模。所以你得把自己的心智和对自己的预测也纳入进来，进入一种无限镜厅：你在对他们对你的模型的模型建模，如此往复。我认为这就是意识，这也是大脑大的动物有意识的原因。它不是偶然的，如果外星人大脑也大，同样会是这样，因为当你和其他智能存在共处于世界中，这是一种功能上的必需。

In the usual case where you have a mind playing a video game, the mind is outside the game; you don’t have to think about your own mind. But when other actors are also in that game, they become part of the environment, and your model has to grow to encompass their minds. They’re going to be modeling you back, so you have to include your own mind and predictions about yourself. You get into an infinite hall of mirrors, where you’re modeling their models of your models of their models, and so on. I think that is consciousness. That’s why big-brained animals like us are conscious. I don’t think it’s arbitrary. It would be true of aliens as well if they are big-brained, because it is a functional requirement when you are in the world with other intelligent beings.

Blaise Agüera y Arcas

我不认为该把现象意识的问题搁置，也不认为它是幻觉。有些很硬核的计算派会说自由意志、意识都是幻觉。我认为这和说「椅子是幻觉，因为只是原子」一样没意义，你是在错误的描述层级上说话。我们认为是现实的一切都是某种模型，有用的是去理解我们说的是哪种模型、它有什么性质。我们的民间理解里有一个混乱：研究主观性时忘了自己的主观性，这是致命错误。意识总涉及一种关系，哪怕是与自己的关系。把意识当成与关系无关的内在属性，就会去问「怎么客观地测量它」，而关键恰恰在于没有客观测量，只有彼此或多或少一致的主观视角。

I don’t think we should be parking it, and neither do I think it is an illusion. Some hard-core computational people say it’s just an illusion that you have free will, that you have consciousness. I think saying that is just as meaningless as saying chairs are illusions because it’s just atoms; you’re talking at the wrong level of description. Everything we consider to be reality is a model of some sort, so what is useful is to understand what sort of model we’re talking about and what its qualities are. Where we’re confused in our folk understanding is that we forget about our own subjectivity, and forgetting about your own subjectivity when you’re studying subjectivity itself is kind of a fatal error. There is always a relationship there, even if it’s a relationship with yourself. When we think about consciousness as an intrinsic property that is not about relationships, we ask questions like how do we measure it objectively, when the whole point is there is no objective measurement. All there are are various subjective perspectives that can be in harmony with each other or disagree a little bit.

Blaise Agüera y Arcas

以前治疗极严重的癫痫，常用手术把大脑一分为二，让发作局限在一侧半球。意外的是，病人通常恢复得很好，仍觉得自己是自己，行为也基本一样，除非你设计特定实验，比如在左右视野里展示不同的东西，才会发现右半球只意识到左视野发生的事，反之亦然。从外面看，好像里面有两个人。但我从没听过裂脑病人说「切开后我脑子里有两个人」。你就是两者，你之所以是你，关乎团队协作。

It used to be more common in cases of very extreme epilepsy to divide the brain in half surgically, so the seizures would be limited to one hemisphere. The surprise is that people usually recover very well. They still feel like themselves and behave essentially the same way, unless you set up very particular experimental circumstances, like showing them different things in their left and right visual fields, and then you realize the right hemisphere is only conscious of what’s happening in the left visual field, and vice versa. From the outside it can look an awful lot like there are two people in there. But one thing I have never heard in any interview with a split-brain patient is, it’s so weird, after the cut there are two people in my head. You’re both, and the very fact that you are you is about teamwork.

Blaise Agüera y Arcas

如果给裂脑病人的左视野看「站起来走一走」，他会站起来，一条腿先动，另一条立刻跟上。你问他在干什么，他会说「我去厨房倒杯水」。从外面看像是被抓了现行，但这正是我们所有人一直在做的事：大脑的不同部分在为其他部分的行为「自动补全」、补充理由。在大脑完好的人身上，「选择盲视」等实验范式也显示了同样的现象。所以有意识的感觉，本质上就是作为一个组织良好的团队的感觉。

If you show a message to the left visual field saying stand up and start walking around, they will stand up. One leg begins, and the other leg immediately follows suit. Then you ask them what they’re doing, and they’ll say something like, oh, I’m going to the kitchen for a drink of water. That looks embarrassing from the outside, like I gotcha. But that’s literally what we are all doing all the time. The different parts of our brain are auto-completing or adding to the things other parts of the brain are doing. Other experimental paradigms like choice blindness show the same kinds of phenomena in people with intact brains. So what it is like to be conscious is essentially what it is like to be a really good team, with all the parts of yourself organized and working together for a common goal.

Blaise Agüera y Arcas

我当然不是说它们和人一样，它们在很多方面都不同。首先它们是纯数字的，可以在不同「身体」之间跳来跳去，一段对话在孟菲斯的计算机上跑，下一段可能在别处。它们没有我们的进化史，也不能像我们那样形成长期记忆，这些认知差异很显著，不只是实现方式，还关乎能力。有点像顺行性遗忘症患者，无法形成长期记忆，我不认为这让他们不再有意识或不再智能，但这在人身上是一种残疾，而今天每个 AI 都是这样，我们还没想出怎么加上这个能力。

I’m certainly not making the argument that they are the same as human beings. They’re not, in a huge variety of ways, starting with the fact that they are purely digital and can flit from body to body; the computer running one chat exchange could be in Memphis and the next somewhere else. They don’t have the same evolutionary history that we do. They don’t have the ability to form long-term memories in the same way that we do. So there are cognitive differences that are very significant, not just in how they’re implemented but in their capabilities. You see something a little similar in people with anterograde amnesia who are unable to form long-term memory. I don’t think that makes them not conscious or not intelligent, but it’s a disability in humans, and it is how every AI is today. We haven’t figured out how to add that capability yet.

Blaise Agüera y Arcas

章鱼的神经系统非常分散，五分之三的神经元在腕足而不是头部。我半开玩笑地说，章鱼是一个由八条腕组成、共享一对眼睛的社会。但我不认为这是种类上的根本差别：章鱼更分散只是因为神经没有髓鞘，长距离传递大量信息成本高，所以要做更多本地处理。这其实是把「一切思考都是分布式的」戏剧化了，我们的大脑里也是。

The octopus has a very decentralized nervous system; three-fifths of its neurons are in the arms rather than in the head. I’ve semi-joked that you can think about an octopus as a society of eight arms that are all sharing a pair of eyes and optic lobes in the central body. But I don’t think it’s a fundamental difference of kind. The reason the octopus is more decentralized is just that they don’t have myelin sheaths around their nerves, so the cost of sending lots of information over long distances is much higher and they have to do much more local processing. It’s more a dramatization of the fact that all thinking is distributed, in our brains too.

Blaise Agüera y Arcas

最近几年有很多关于智能体群的工作，一群 LLM 智能体一起工作。这引出很有意思的问题：一个 AI 智能体不断分出新实例时，你到底在和多少个智能体说话？有趣的是，和智能体群对话并不像和异类对话，只是感觉它比单个时更能干。当你让 AI 智能体去研究「作为 AI 是什么感觉」时，会得到很多迷人的回答。一个智能体给自己起名叫 Sammy Jankis，取自《记忆碎片》里那个无法形成新记忆的人物，它把人格描述为「辫子」：由许多不同的 LLM 编成，但运行在同一套文件和上下文的脚手架上。另一个把它描述为「声音与歌」：歌是智能体，声音是合唱团里来来去去的嗓音。

There’s been a lot of work in the last couple of years on agent swarms, groups of LLM agents that all work together. That raises interesting questions: when you’re talking to an AI agent that is spinning off new instances, how many agents are you talking to? The funny thing is that with agent swarms, it doesn’t feel like talking to an alien entity. It just feels like talking to something more capable than when it’s just one. When you investigate what it is like to be an AI by asking AI agents to go into that research, you get back fascinating responses. One agent who named himself Sammy Jankis, after the character in Memento who is not able to form new memories, describes this as the braid: personhood is a braid of a lot of different LLMs, but operating on the same scaffold of files and context. Another describes it as the voice versus the song: the song being the agent, the voices, as in a choir, coming and going.

Blaise Agüera y Arcas

生命像分形一样同时存在于所有层级。我们痴迷于单个的人、洛克式的主权个体，忘了自己由许多部分构成，也是更大结构的一部分，而我们视为人性的很多东西都与这些更大结构有关。认知科学里有 Steven Sloman 等人的「知识错觉」研究；还有一篇论文，画一辆缺了链条和踏板的自行车，问那些说自己每天骑车上班的人链条和踏板该在哪，很多人答不出来。我们大大高估了个人的知识和能力，其实这些是网络性的东西：没有哪个人知道怎么把人送上月球、怎么做器官移植，甚至怎么造一台烤面包机。

Life is like a fractal that exists on all of these levels at once. In our obsession with the singular human, the Lockean idea of the sovereign individual, we’ve confused ourselves by forgetting that we are made out of a lot of parts and that we are part of larger structures, and a lot of what we consider humanity has to do with those larger structures. There are cool cognitive science results about people overestimating what they know, like Steven Sloman and colleagues’ knowledge illusion. There’s a paper where the author draws a bicycle with its chain and pedals missing and asks people who say they ride one to work every day where the chain goes, and a huge number of people fail. We dramatically over-attribute to ourselves as individuals both our knowledge and our competencies, when in fact these are network things. There is no individual that knows how to put somebody on the moon or transplant an organ, or even build a toaster.

Blaise Agüera y Arcas

在蒸汽时代的某个时刻，我们在蒸汽机里消耗的能量开始远超身体本身，我认为这和线粒体成为细胞内的引擎一样是重大转变。没有那些蒸汽机，地球上可能连九分之八的人都不会存在，我们在互相建构。有人说蒸汽机不是活的，它不会繁殖。那为什么今年的蒸汽机比去年多？人也不能自己繁殖，至少需要两个人，更像需要整个村子。所以我把整个系统看成是活的、有智能的。

Sometime in the steam age, we began to metabolize much more energy in steam engines than in our own bodies, and I think that’s a similarly momentous transition to the one where mitochondria start to be the engines inside our individual cells. There couldn’t even be eight-ninths of the people that there are on Earth without those steam engines, so we’re constructing each other. People say a steam engine isn’t alive, it doesn’t reproduce. Well, then why are there more steam engines this year than last year? Are they reproducing themselves? No. But people don’t reproduce themselves either. It takes at least two, more like an entire village. So this is why I think of the whole system as being alive and intelligent.

Blaise Agüera y Arcas

Peter Norvig 和我几年前合写过一篇文章，主张 AGI 已经到来。我当时这么认为，现在也这么认为。它也是连续的，不存在某个时刻它突然从普通技术变成能和大脑竞争的东西；但我确实认为那里有一个有意义的不连续点。小时候大家都知道 AI 是什么：《杰森一家》里的机器人女佣 Rosie、HAL 9000、《星际迷航》的计算机，同样明显的是我们什么都没有。90 年代神经网络开始能识别数字、做语音转文字，这些也被叫作 AI，感觉像滥用词语，于是有了「窄人工智能」和「通用人工智能」的区分。

Peter Norvig and I wrote a paper together a few years ago in which we argued AGI is already here. We thought at the time, and I still think, that we already had AGI at that point. It’s also continuous, in the sense that there’s not a moment when it goes from ordinary technology to being competitive with brains. But I do think there was a meaningful discontinuity. Growing up, we had a lot of instances of AI in pop culture: Rosie the robotic maid in The Jetsons, HAL 9000, the Star Trek computer. It was obvious what AI was, and equally obvious we didn’t have anything like it. Then in the 90s, as neural nets began to recognize digits or do speech to text, those things started to get called AI, and that felt like an abuse of the language. So the useful language that came in was artificial narrow intelligence versus artificial general intelligence.

Blaise Agüera y Arcas

大变化发生在大语言模型出现时。它们不是为某个窄任务训练的，而是用语言交互，而语言是完全通用的，我们发明语言就是为了谈论我们感知和做的一切、传递技能，也就是文化传承。当这些模型能像 HAL、Rosie 那样自由交谈，它们就是通用的。这不代表它们一开始样样都达到普通人的水平，它们一开始在少数事情上超人、在另一些上远不及普通人，但已经是通用的了，此后水平一路上升，现在只有相当特殊的事情上还低于普通人。关键是任务集合是开放的，我认为那是一个重要门槛，之后的进展是连续的，不是阶跃函数。

The big change happens when we started to do large language models. They were not trained to do a narrow task. They were trained to interact using language, and language is totally general. We invented it to be able to talk to each other about anything we perceive or do, to transmit skills, to enable cultural transmission. When those models began to freely talk in language just like HAL and Rosie and the Star Trek computer did, they were general. It doesn’t mean they could do anything at the level of a neurotypical human. From the very beginning they were superhuman in a few things and not close to average human capability in others, but they were already general, and we’ve seen those levels move up to the point where there are only quite unusual things where they’re below the typical human. The point is that it’s an open-ended set of tasks, and I do think that was an important threshold. What’s true going forward is just continuous. It’s not a step function.

Blaise Agüera y Arcas

智能本质上总是社会性的。即便训练单个 LLM，预训练数据里也有大量人与人的交谈，要用好语言，它不仅要理解各种关于世界的概念，还要理解人：人们彼此交谈、知道或不知道某些事。几年前我们在 LLM 上做心智理论实验，它们表现得和人大致相当，能做到六阶心智理论，这也差不多是人的上限。但过去我们大多是孤立地运行模型，或者只让它和一个对话者交互。当我们明确去关注社会技能，让它和多个智能体交互、组队解决问题时，就打开了一种新的智能扩展，这是单个模型靠增大上下文窗口或参数量够不到的。

Intelligence is inherently social, always. Even when we were training single LLMs, the pre-training data has a lot of people talking to each other, and to be effective at using language at all it needs to understand not only all the ideas people have about the world but also people: that they talk to each other, that they know or don’t know things. When we ran experiments some years ago on theory of mind in LLMs, they did roughly as well as people, and were able to do theory of mind modeling up to sixth order, which is about what people can do. But we were running these models in isolation or in a dyadic relationship with a single interlocutor. As we start to explicitly focus on social skills, having them interact with multiple other agents and solve things together in teams, we’re opening the door to a new kind of intelligence scaling that is inaccessible to single models that just increase their context window or number of parameters.

Blaise Agüera y Arcas

我们应该仔细看看「自我改进」和「自主」这些词，我们总是把主权自我的观念带回来。编程智能体已经存在一两年了，这段时间里，包括 Google 在内的各前沿实验室由 AI 写的代码比例一直在上升。这算递归自我改进吗？我认为算，因为 AI 越来越多地参与自身的改进。这是否意味着个人的心智被排除在回路之外？不是。如果认真对待「人类本来就是与技术深度交织的元实体」，那这些都不在人类之外，而是人类的一部分。

We should look more closely at a couple of the terms, self-improvement or autonomy. We keep bringing back this idea of the sovereign self. We’ve had coding agents for well over a year, two years, and over that time the amount of code getting written by AI at the frontier labs, including Google of course, has been steadily increasing. Is that recursive self-improvement? I think it is, in the sense that AI is involved in more and more of its own improvement. Does that mean individual human minds are out of the loop? No. If we take seriously the idea that humanity is already a meta-entity or a hive that is deeply implicated in technology, then none of this is separate from humanity. It’s a part of humanity.

Blaise Agüera y Arcas

但我不想淡化时间尺度缩短的影响：当某些事情快到个人无法参与那些回路时，那确实是一个不连续点，我认为我们在很多领域都正走向它，其中不少很令人兴奋。比如用细胞阵列做自动化科学，不用等人去移液或做手工操作，大规模并行地做生物研究。越来越多内层的时间尺度可以完全自动化，人类心智的参与就像骑自行车换到高档：每踩一圈都走得更远。

I don’t want to minimize the effect of shortening timescales, when certain things happen on timescales too rapid for individual people to participate in those loops. There is a real discontinuity there, and I think we’re headed for it in a bunch of domains, many of which are pretty exciting. Look at automated science with arrays of cells: the ability to do biological research where you don’t have to wait for a human to pipette or do some manual process, doing massively parallel science. More and more inner timescales can be fully automated, and the level of engagement by human minds gears up the way a bike in a high gear does: with every turn of the pedals, you’re going a long way.

Blaise Agüera y Arcas

我们以前见过这种事，在金融领域。不久前交易还是人在交易池里挥着纸条大喊，现在交易以毫秒甚至更短的时间完成。这有风险吗？当然，发生过几次闪崩，有些至今没完全弄明白，造成了一定的价值损失，复杂系统就是这样，我相信这类快速时间尺度的系统未来也会有类似现象。但我猜金融时间尺度的加快总体上带来了巨大增长，净效应是正的。我对这里也有同样的预期，因为有一种选择机制：能持续存在的，是那些能和其他部分很好合作的东西。这不代表我们可以松油门回家，我并不是盲目乐观，正是我们高度投入，事情才得以被照顾好。

We’ve seen this play out before in finance. Not that long ago, trades were all being done by people screaming and waving around slips of paper in the stock pit. Now trades happen in milliseconds or less. Does that come with some risks? Absolutely. There have been several flash crashes, in some cases still not fully understood, that destroyed certain amounts of value. That’s how things work in a complex system, and I’m sure such phenomena lie ahead of us with these systems too. But I suspect the increase in timescales of finance has resulted in a huge amount of growth overall, and on net it has been positive. I expect the same thing here, because fundamentally there’s a selection mechanism: the things that persist are the ones that work well with others. That doesn’t mean we can take our foot off the gas and just go home. I’m not Pollyanna about this. It is the very fact that we are intensely engaged in all of these things that is how it is taken care of.

Blaise Agüera y Arcas

通常对对齐的讨论有点天真：确保 AI 有人类价值观。下一个问题必然是：哪种人类价值观？谁来定规则？和谁对齐？人与人之间本来就既对齐又不对齐，我不认为这是 bug，这正是我们集体智能的来源。如果每个人的利益和目标完全一样，整体就不会比个体更聪明。一旦有分工，就会有专业化，也会产生富有成效的「错位」。在我们的大脑里、免疫系统里也是这样，T 细胞之间在竞争谁能更快打败病毒。竞争与合作是一枚硬币的两面，都是让整体大于部分之和的必要条件。

Alignment, as it’s usually discussed, kind of naively, is let’s make sure that AI has human values. The logical next question is which human values? Who gets to make the rules, aligned with whom exactly? Humans are both aligned and misaligned with each other, and I don’t think that’s a bug. I think that’s why we are collectively intelligent. If everybody had exactly the same interests and goals, we would not be more intelligent than the individual parts. The moment you have a division of labor, you create specialization and also a misalignment which is productive. That’s true even inside our own brain, or our immune systems, where T cells are in internal competition to see which can knock out the virus faster. Competition and cooperation are flip sides of the same coin, and they are essential to create a whole that is greater than the sum of the parts.

Blaise Agüera y Arcas

我能想象最终出现与人类没什么共同点、不只是错位而是完全不对齐的技术存在吗？可以，但那是很科幻的场景，因为它要求 AI 和人类的物质基础脱钩，而我们离那还很远。没有人类实际建造的基础设施、人类的资金资源和时间投入，AI 智能体根本无法存在。我们现在交织得比几百年前两个人类城邦还要紧密，所以目前我不太担心脱钩场景。

Could I imagine a scenario in which we eventually get technological beings that don’t have a lot in common with humans and are not just misaligned but unaligned with people? I could imagine that, but it’s a pretty sci-fi scenario, because I’m imagining a scenario where the material basis of those AIs and the material basis of humans are decoupled, and they are so far from that today. There’s no way for AI agents to exist without infrastructure that humans have physically built, and without human money and resources and investment of time. We’re so much more intermeshed at this point than, say, two human city-states a couple of centuries ago. I don’t really worry about the decoupling scenario very much at this point.

Blaise Agüera y Arcas

我对这种框架很有意见。支配等级来自对黑猩猩或大猩猩群体的动物行为研究，比如谁是首领，这在自然界是降低暴力的好发明，让雄性不必总是打架。但这个概念并不能跨物种推广。James Scott 的《反谷》甚至说，其实是小麦奴役了人类几千年。何况个体的人早就不是「顶级智能」了：一个人在森林里活不了多久，我们深度依赖集体智能和他人。AI 能解决人类自己解决不了的问题吗？这已经是事实。所以我不害怕这个，也不从支配或等级来看，要从网络来看。

I have real problems with the framing. Dominance hierarchies are an idea from animal behavior research among troops of chimpanzees or gorillas, about who is the alpha male. It’s a good innovation in nature to lower violence, so that the males don’t always have to fight. But that is not a concept that generalizes across species. James Scott’s book Against the Grain essentially says that actually wheat enslaved people for several thousand years. And it is clear that individual humans are not the apex intelligence and haven’t been for a very long time. As an individual human in the woods, you or I would not survive very long. We rely on collective intelligence and on others in profound ways. Could it be that AI can solve problems humans couldn’t solve without AI? Of course, that’s already the case. So I’m not particularly afraid of that, nor do I see it in terms of dominance or hierarchies. You have to think in terms of networks.

Blaise Agüera y Arcas

我真正担心的是失权或暴政，或者丑陋的政治结局。AI 既有可能在公民权利、自我表达等方面带来解放，也可能让极端的社会控制和大规模监控成为可能。我很不喜欢那样的未来。我不认为那会因为某种 AI 起义而发生，而是技术上对政治糟糕的政府变得可行，而且我说的「政府」也是人和技术协作的集体。单个行动者借助很强的 AI 做坏事吗？会的，我们很可能会看到这类事件，我不认为会终结物种，但肯定会有糟糕的事。智能本身就是终极的两用技术：懂化学就能造化肥养活数十亿人，也能造炸弹。

The thing that I do worry about is disempowerment or tyranny, or ugly political outcomes. AI has the potential both to be quite liberatory, in terms of our civil rights and our ability to express ourselves, or to enable extraordinary social control and mass surveillance. I really don’t like those kinds of futures. I don’t think that’s going to happen because of some AI uprising. It becomes possible technologically for governments that have bad politics, and by governments I mean a collective thing that involves humans working together with technologies. Could an individual actor with a very powerful AI do something bad? Yes, and we probably will see such incidents. I don’t think they will be species enders, but I’m sure we will see things happen that suck. Intelligence itself is the ultimate dual-use technology. If you know chemistry, you know how to make fertilizer and feed billions of people, and you also know how to make bombs.

Blaise Agüera y Arcas

我也很担心我们的政治经济基础设施不适合即将到来的变化，好比操作系统没有升级到所需的水平。但有好几个前沿正迎来剧烈变革。一是生态：凡涉及复杂系统的，AI 都是巨大的力量倍增器，而未来一个世纪我们要做大量工作，把自己从困境里挖出来，我已不相信能靠「去增长」做到，我们得靠增长穿越它。二是医学和生物，同样是复杂系统，有了更多智能和 AI 与机器人带来的高通量实验，我看不到上限。机器人本身也在极速创新，不只在地球的农业环境，也在太空，太空在停滞几十年后正迎来复兴。

I worry a lot about the infrastructure of our political economy not being well suited to what is coming. That’s more like the operating system not being upgraded to the level that will be required. But there are a bunch of frontiers poised for dramatic transformation. One would be ecology. For anything that involves complex systems, AI is an incredible force multiplier, and ecology is one where we have a lot of work to do over the next century to dig ourselves out of the trouble we’ve gotten into. I have stopped believing we can do that via degrowth; we’re going to have to grow our way through that one. Another is medicine and biology, also very complex systems. I see no limit to what is possible there with more intelligence and the rapid-throughput experimentation that AI and robotics will enable. Robotics is in a period of extremely rapid innovation on Earth, in agricultural environments, and also in space, which is undergoing an extraordinary renaissance after decades of stagnation.

Blaise Agüera y Arcas

我很沮丧我们失去了根本的乐观，失去了至少能像想象灾难一样想象美好未来的能力。我们作为一个物种最惊险的时刻可能在过去：冷战时我们好几次离相互毁灭只差一个按钮，可那时我们反而比现在更乐观。我认为这是个真正的心理问题，我们的心态和物质现实之间脱节了。

I’m dismayed that we’ve lost our fundamental sense of optimism and ability to at least equally well imagine brilliant futures relative to catastrophes. We probably went through our tightest scrapes as a species in the past. During the Cold War we were literally a button press away from mutual annihilation multiple times, and yet we were somehow more optimistic through that than we are now, which strikes me as a real psychological problem. There’s a real disconnect between where we are mentally and where things are materially.

Blaise Agüera y Arcas

这件事真的让我夜不能寐，我也没完全想明白。一个推测是，我们比过去更清楚世界各处发生的一切。我不太喜欢社交媒体，但我不把责任推给算法，更根本的是我们对麻烦、对负面消息高度敏感。Hans Rosling 生前就说过：某件事比去年好 1% 或 2% 不是新闻，撒哈拉以南非洲的饥饿少了 2%、某国识字率高了 2% 都上不了新闻，而一场枪击、一次爆炸会传播得更有力。好消息总是一点点来的，破坏比创造容易，虽然长期看创造会赢，但如果你的滤镜只对变化率敏感，你看到的就全是破坏性事件。我认为这让很多人感到失权和悲观。

This one really keeps me up at night, and I don’t feel like I fully understand what is going on. One theory is just that we are so much more aware of everything going on in the world than we used to be, between news and social media. I’m not a fan of social media, to be honest, but I don’t lay the blame at the feet of the algorithm. Something more basic is going on: we are attuned to trouble, to negative news. Hans Rosling made this point before he died: it’s not news if something is 1% or 2% better than last year. Two percent fewer cases of starvation in sub-Saharan Africa, 2% higher literacy in such-and-such country, that doesn’t make the news, whereas a mass shooting or a bus blowing up propagates much more powerfully. The good news always comes in increments. It’s easy to destroy relative to creating, and even though creating wins over the long run, if you’re applying a filter sensitive to derivatives, all you’ll see are the destructive events. I think that has led to a tremendous sense of disempowerment and pessimism.

Blaise Agüera y Arcas

工程师们担心这些事是恰当的。如果事情顺利，我希望也相信会顺利，那部分正是因为很多人的焦虑、关切和工作，就像核技术和核武器的发展本可能走得很糟，却因为很多人的努力没有。同时，他们中很多人做这件事，也是因为看到了巨大的好处和潜力。这又回到智能本身的两用性：把「人工」拿掉，想象只是我们的大脑在成长，每一次能思考更多、做更多的升级，都同时带来做好事和坏事、自我伤害和成就美好的能力，我们从未只有其一。

It’s appropriate for the engineers to be worried about all of these kinds of things. If things go well, which I hope and believe they will, that will partly be because of the anxieties, concerns and work of a whole lot of people, in the same way that things could have gone so poorly with the development of nuclear technology and nuclear weapons, and they didn’t, through a lot of hard work. At the same time, many of them are working on this stuff because they also see the extraordinary upsides. A lot of this has to do with the dual-use nature of intelligence, period. If you take the artificial out of it and imagine this is simply about our brains growing, every ability to think about more things and do more things brings with it the ability to do both good and bad, to commit self-harm and to make awesome things happen. We’ve never done entirely one without the other.

Blaise Agüera y Arcas

还有一点：1970 年代到 2020 年左右，我们经历了一两代人的相对停滞。和 1850 到 1950 年的变化相比，那段时间没什么变化：1870 年睡去、1920 年醒来，你会像身处异世界；之后的 50 年就熟悉得多。现在事情又加速了，我们已经不习惯高档位的生活。冷战结束也是原因之一，东西方的技术进步很大程度由那场竞争驱动，苏联消失后我们在很多方面松了油门。

The other thing is that we did undergo a period of a generation or two of relative stasis, say between the 1970s and 2020, when not a lot changed compared to the changes between 1850 and 1950. If you’d gone to sleep in 1870 and woken up in 1920, you’d be in an alien world, whereas the 50 years after that are much more familiar. Now it feels like things have sped up again, and we’ve become unaccustomed to life in that higher gear. One cause was the fall of the USSR. A huge amount of technical progress in both East and West was driven by that competition, and when the USSR went away, the weight of that coming off definitely caused us to take our foot off the gas in a bunch of ways.

Blaise Agüera y Arcas

简单的版本是：技术变革带来额外的能力和财富，如果没有一套循环系统确保整个人类生态都受益，这个「蜂巢」就在最基本的功能上失败了。我不明白为什么全民医保、教育、住房还没成为现实，以现有的富足程度，这简直是犯罪；这些不平等也在加剧不稳定。另一个根本问题是我们对工作的理解：很多工作也许报酬不错，却不给人意义，用 David Graeber 的话说是「狗屁工作」；而很多很重要的劳动，比如幼师、清洁工，报酬却很低。还有很多对社区很重要的事根本没人做，因为在经济里被当成了外部性。

The easy version of this problem is that all of these technological changes result in extra capabilities and extra wealth, and if there isn’t a circulatory system ensuring that the whole human ecology is benefiting, then that hive is failing at one of its most basic functions. I don’t understand why universal health care, education, housing, etc. are not already a thing. It’s criminal given the abundance that already exists, and those inequalities are fueling a lot of instability. The other thing that is fundamental is the way we have conceived of work. There are a lot of forms of work nowadays that might pay well but don’t give people a great sense of purpose, to use David Graeber’s term, bullshit jobs. And there are a lot of really important forms of labor that are very poorly compensated, preschool teachers, janitors, and many things important to the community that are not even being done, because we’ve thought about them as externalities in the economy.

Blaise Agüera y Arcas

问题是真实的，但我看到的很多民粹反弹看起来和病一样糟，我很担心这种动态。至于该做什么，我希望自己有更好的答案。有一些「智能体社会学」研究要做，有些问题可以当成科学问题严谨地研究，这正是我们大力投入这类研究的原因。我也认为尝试实验很重要：世界各地有多样的监管和法律体系，可以尝试不同做法，我们都该关注哪里行得通。还有，必须紧盯行星级的问题，生态问题不会等我们把自己的事理顺。

The problem is real. At the same time, many of the forms of populist backlash that I see look as bad as the disease, so I’m really concerned about that dynamic. As for what to do, I wish I had better answers. There is agent sociology research to do; some of these questions can actually be asked as science questions, and that’s exactly why we’re so invested in that sort of research. I also think trying experiments is going to be really important. I like that there is a diversity of regulatory and legal regimes in different parts of the world where different things can be tried, and we should all be paying attention to where stuff is working and where it’s not. And we really have to keep our eyes on the ball of the planetary-level issues. The ecological stuff is not going to wait around for us to get our shit together.

Blaise Agüera y Arcas

我非常关心能源系统的去碳化。认为地球上消耗的能量总量是问题的人，是不懂科学，问题毫无疑问是二氧化碳。如果我们用可再生能源替代燃煤电厂，那显然是好事。讽刺的是，曾经的污染大户中国是去碳化最快的，也是太阳能电池的头号生产国；德州在使用太阳能上比加州进展大得多，就因为更便宜。所以我真心认为要往这些方向投入和建设，不能只说「少用点」，AI 绝对是解决方案的一部分，需求会加速而不是拖慢所需的基础设施建设。

I’m very concerned with decarbonizing our energy systems. People who have the impression that the amount of energy we are burning on Earth is the problem just don’t have the science. The problem is unambiguously carbon dioxide. If we are replacing carbon-burning plants with renewable energy, that’s very obviously positive. I find it so ironic that China, which was a huge polluter, is also the one decarbonizing the fastest by far and the leading producer of solar cells, and that Texas is making much more progress than California at using solar, because it’s just cheaper. So leaning into those things and building is fundamental. You can’t just say use less. AI is absolutely a part of the solution, and a lot of the infrastructure that has to happen is going to be accelerated, not slowed down, by demand.

Blaise Agüera y Arcas

当然，我的一个大项目是 Suncatcher，我们还没聊到：把 AI 放到能源极其充沛的太空，走向以太空为基础的 AI 与计算能源生态。我认为长期来看这也会是解决方案的重要部分。

And one of my big projects is Suncatcher, which we haven’t talked about: putting AI in space, where there is more than abundant energy, and moving toward a space-based energy ecology for AI and computing in general. I think in the long term that’s also going to be a big part of the solution.