Google AI 基础设施负责人 Amin Vahdat：算力的物理与经济学

Amin Vahdat

做硬件有一个规律：越是针对某个特定工作负载做专用化，灵活性就越差，但硬件会更快、更省电。所以这是一门艺术，要预判你设计所针对的东西，以及这个工作负载能持续多久。如果它一两个、两三个月后就会消失，哪怕那三个月里规模很大，你能「截住」它的窗口也非常窄。所以它必须有一定的持久性，而且你得能准确预估专用化能换来多少收益。

In hardware, there is this opportunity where the more you specialize to a particular workload, the less flexible it is, but the faster and the more power-efficient the hardware is going to be. So it is this art, and it’s this projection of what you’re designing to and how persistent that workload is. If it’s going to go away after a month or two months or three months, even if it’s big for those three months, you’ve got a really narrow window to intercept it. So it has to be somewhat durable, and you have to be able to project exactly what you can get for specializing to it.

Sonya Huang

我们正处在人类史上最大规模的资本开支扩建之中，而你就在它的中心。光是 Google 今年的资本开支预计就超过 2,000 亿美元，大部分用于建数据中心。你去年底被任命为 Google AI 基础设施的负责人，是这场史上最资本密集的扩建的领头人之一。

We are in the middle of the biggest CapEx buildout in human history, and you are in the middle of it. Google alone is expected to spend more than $200 billion on CapEx this year, most of which is going into building data centers. You were named the head of Google’s AI infra at the end of last year, so you are the man spearheading one of the most capital-intensive buildouts in human history.

Amin Vahdat

这一年对全行业、包括 Google 来说确实非同寻常。说实话，我们从没见过这样的情形，Google 内部肯定没有。正如你所说，从人类历史上的扩建规模和转型速度来看，这都令人难以置信。

It’s a huge year, for sure—across the industry, including at Google. I don’t think we’ve ever seen anything like this, honestly. Certainly not at Google. In the history of humanity, in terms of the buildout and the pace of transformation, it’s really unbelievable.

Amin Vahdat

AI 数据中心和非 AI 数据中心其实有很多相似之处，并没有根本区别：混凝土和外壳，电气场、机械场、冷却，一排排的供电，大量网络设施把海量算力连起来，还有相当多的存储。

An AI data center and a non-AI data center do have a lot of similarities, actually. They’re not radically different. It consists of concrete, an enclosure, electrical yards, mechanical yards, cooling. You have row after row of power being distributed. There’s a huge amount of network infrastructure, and a fair amount of storage infrastructure goes in there.

Amin Vahdat

最大的区别是专用化。过去建数据中心是一项 25–30 年的建筑投资，要规划它在这 25–30 年里怎么演进；里面的服务器、网络、存储、加速器的寿命可能只有 6 年，所以要为好几代硬件做规划。

The big difference that we’re seeing with AI infrastructure is specialization. In the past, building a data center really was a 25-, 30-year building investment. You might have servers, networking, storage, some accelerators going into it, but the lifetime of the hardware might be six years. So there are many generations that we have to plan for.

Amin Vahdat

AI 数据中心往往更「量身定做」：我们常常把建筑和里面的硬件一起协同设计。比如决定这栋楼不放太多存储——一个存储机架可能是 10–40 千瓦，而 CPU 或 GPU 机架如今轻松达到数百千瓦，业内正在讨论未来两年达到兆瓦级。一排放 30 个存储机架和一排只放一两个 AI 机架，是完全不同的设计，供电分配、网络需求都不一样。

An AI data center oftentimes is going to be more purpose-built. We are oftentimes co-designing the building with the hardware that might go into it. We might say we’re not going to put a lot of storage into this building, because a storage rack might have 10, 20, 30, 40 kilowatts of power, next to a CPU or GPU rack that easily hits hundreds of kilowatts today and might hit megawatts—people are talking about that in the next two years. Designing a building that takes 30 storage racks in a row versus one or two AI racks in that same row is a very, very different design, in terms of power distribution and networking.

Amin Vahdat

如果你想让它在 30 年里完全通用，很可能会建得过大、过度冗余。AI 数据中心更可能是定制的，与硬件协同设计，一直细到冷却和配电，协同优化的程度高得多。

If you’re trying to make it fully fungible over a 30-year period, you’ll probably make it too big and too overbuilt. An AI data center likely is going to be much more purpose-built, co-designed with the hardware, even to the point of cooling and power distribution. So a lot more co-optimization.

Sonya Huang

你们给我投资的一家公司 Ineffable Intelligence 交付了一个大型 Vera Rubin 集群，我看到了出货时的照片，太美了。

You all delivered a big Vera Rubin cluster to one of my portfolio companies, Ineffable Intelligence, and I saw the photographs of it as it went out. That is a thing of beauty.

Amin Vahdat

那其实只是几个机架，但布线、光纤分布真的很漂亮。我们发到社交媒体上，大家喜欢 Ineffable，但更喜欢那些光纤照片和它分形般的结构。那是我们有史以来最受欢迎的帖子之一。

This is just a couple of racks. The cabling, the fiber distribution for it, is really beautiful. We put this picture up on social media, and people love Ineffable, but people just loved seeing the pictures of the fiber and the sort of fractal nature of the fiber. One of the most popular posts that we’ve ever had, actually.

Sonya Huang

在这场大扩建中，你怎么衡量、怎么给自己问责？开录前我们聊到，你认为 FLOPS 是虚荣指标，更看重另一个指标。

How do you measure and hold yourself accountable in the middle of this big buildout? We were talking right before the show about how FLOPS is a vanity metric and you prefer an alternate metric.

Amin Vahdat

无论 FLOPS 还是其他以芯片为中心的指标，都是理论值——某种条件下这颗芯片最多能交付多少。我们最终在乎的是每个工作负载交付的性能。性能由单颗芯片的 FLOPS、HBM、SRAM 决定的情况其实相对少见；更多取决于 2、4、8、16、1,000、10,000 颗乃至更多芯片如何组合，以及给它们喂数据的 CPU、把它们连在一起的网络。

Whether it’s FLOPS or your other favorite chip-centric metric, these are in theory—under some conditions, this is the maximum amount of FLOPS a chip can deliver. What we really care about in the end is the performance delivered by workload. It happens relatively rarely that performance is determined by a single chip—its FLOPS, its HBM, its SRAM capacity. It might be how two, four, eight, 16, 1,000, 10,000 or more of these chips compose together, plus the CPUs that feed them the data and the network that connects them all.

Amin Vahdat

很多工作负载——训练、推理、智能体——都是同步的，成千上万个组件要在微秒、毫秒级协调。一个挂了，可能整个任务停下：要查出哪个停了、找到之前的检查点、恢复、重启，最坏情况从头再来。这些都是「功」，但不帮你得到答案。就像在纸上解题，做错了退回第一步，你仍在做功，这是吞吐量；但交付答案的 goodput 是你解完这道题花的总时间。

Many of these workloads, whether training or serving or agentic, are synchronous: 1,000, 10,000, 100,000 components coordinating at microsecond and millisecond granularity. One of them fails, and it might bring the whole thing to a stop. Which one stopped? What’s a checkpoint? How do we restore it and restart? Worst case, we restart from the beginning. All that work is work, but it’s not helping you get the answer. If you’re working a problem on paper and have to go back to step one, you’re still doing work—that’s throughput. But goodput is the total time it took you to solve the problem.

Amin Vahdat

所以我们的问责标准是交付的 goodput——不是理论基准吞吐量，而是真实工作负载在真实故障条件下的表现。而残酷的现实是，在 10 万加速器规模下——

So how do we hold ourselves accountable? It’s delivered goodput. Not theoretical benchmark throughput, but for a workload, under real failure conditions, what’s happening. And the unfortunate reality is that at 100,000-accelerator scale—

Sonya Huang

我正想说，那种规模下总有东西在坏吧？

I was going to say, at that scale something’s failing all the time, right?

Amin Vahdat

一直在坏。每颗芯片都处在制造工艺的最前沿，而且往往是由 2、4、8 甚至更多 chiplet 组成的封装，旁边还有 HBM、网络互联，也许还有共封装光学。这不是批评，只是能坏的东西太多。你得准实时地检测、准实时地恢复，遥测问题极其庞大，就像在干草堆里持续找针，对有些任务要持续数小时、数天甚至数周。

All the time. Each one of these chips is at the very bleeding edge of what’s possible to manufacture. And they’re packages made up of two, four, eight, maybe more chiplets, with HBM off to the side, network connectivity, maybe co-packaged optics. Not a criticism—lots of things can fail. You have to detect that in near real time and recover in near real time. The telemetry problem is massive. It’s like finding a needle in the haystack continuously, for some jobs across hours and days and even weeks.

Sonya Huang

goodput 是 Google 的术语还是行业术语？10 万加速器的规模，大概多久坏一次？

Is “goodput” a Google term, or an industry term? And at 100,000-accelerator scale, how often does something fail?

Amin Vahdat

是 Google 的术语，但越来越多业内人开始用。在那种规模下肯定一天多次，视具体配置，也许每小时都会有东西坏好几次。

It’s a Google term, but more and more people across the industry are starting to pick it up. At that scale, it is definitely going to be multiple times a day, and perhaps, depending on the exact configuration, multiple times an hour something is going to fail.

Amin Vahdat

至于最常见的故障原因——问题就在这：如果有一个常见原因，我们早就查出来修好了。它是一条持续发现的长尾。每推出新产品总会有东西打到我们，可能是网络、高速互联、硬件，也有很多是软件。芯片也许能达到某个 FLOPS，但只要有编译器 bug、运行时 bug、模型问题、操作系统问题，端到端性能都会受影响。

If there were a common failure reason, we would have figured it out and fixed it. It is a long tail of constant discovery. Whenever we introduce a new product, something is going to hit us—it might be network-related, how we connect things at super high speed, the hardware. But a lot of the issues could be software. If you have a compiler bug, a runtime bug, a model issue, an operating system issue, it’s going to impact the end-to-end system performance. You might have perfect hardware and still have software issues that hurt you.

Amin Vahdat

参考架构是有的。Nvidia 是一家了不起的全系统公司，不只是半导体公司，他们给出非常强的参考栈。大多数客户会用它，但很多也会做针对自身场景的专用化。TPU 这边也一样。

There’s a reference stack. Nvidia is an incredible whole-systems company—not just a semiconductor company—and they give you a really strong reference stack. Most of our customers leverage it, but many also specialize for their particular use case. Similarly, for the TPU side, we have a reference stack, and many will specialize it as well.

Sonya Huang

你跟团队说过，Google 大约每六个月就得把服务容量翻一番，对吗？

You’ve told your teams that Google has to roughly double the serving capacity every six months or so. Is that right?

Amin Vahdat

准确地说，是「有效可用容量」，从服务角度看就是生成 token 的能力。不一定是每六个月 FLOPS 翻倍——那只是一条路——而是把硬件生成 token 的能力翻倍，其中来自软件的部分与硬件相当甚至更多：模型优化、运行时优化，往往是几十上百个单项优化一次次落地。

To be clear, this is in terms of effective available capacity—from a serving perspective, token generation capability. It’s not necessarily that you have to double the number of FLOPS every six months; that’s one path. You have to double the capability of that hardware to generate tokens, and as much or more of that is going to come from software as from hardware—a model optimization, a runtime optimization. It’s probably dozens or hundreds of individual optimizations landing again and again.

Sonya Huang

以「每瓦智能」来衡量，容量增长中有多少来自芯片本身，多少来自模型和其他软件？

Measured as intelligence per watt, how much capacity increase has come from the silicon itself versus the models versus other software?

Amin Vahdat

我没有精确拆分，但在我们的经验里，就每瓦智能而言，大部分收益来自模型侧的改进。每瓦智能是个很好的指标；对我们来说是每瓦 goodput，分母必须是瓦。模型和软件系统能确保硬件被有效利用。而硬件本身也很惊人：如今每年 2 倍以上的性能提升完全可能，它是上面所有层都能指望的一个乘数。

I don’t have the exact breakdown, but in our experience, most of the benefits come from model-side improvements in terms of intelligence per watt. For us, it’s goodput per watt—the watts need to be the denominator. The model side and the software system make sure the hardware is used effectively. And the hardware is pretty stunning: 2x or more year-over-year performance improvement is absolutely possible. It’s a multiplier everyone above it can count on, lifting everyone up year over year.

Amin Vahdat

项目在 2013 年启动时确实是逆势之举。当时的主流共识是：不要为单一工作负载造专用加速器，因为摩尔定律还在，每 18 或 24 个月性能翻倍，你还能沿用 C++、Java、Python 等标准编程模型。

When the program started in 2013, it was really a contrarian call. The 2013 conventional wisdom from all the smartest people was: you don’t build a custom accelerator for a single workload, because Moore’s Law is there—doubling every 18 or 24 months—and you get to leverage standard programming models, your C++, Java, Python.

Amin Vahdat

也就是「芯片的苦涩教训」：专用化永远赢不了。但当时我们有一两个应用会极大受益，否则需要难以想象的通用 CPU 来支撑。所以那是一次押注，公司内部也有不少人怀疑能否成功，结果大获成功。

The bitter lesson of chips is that specialization never wins. But in this case, we had one application, or a small number, that would benefit tremendously and would otherwise require an unimaginable amount of general-purpose CPU. So in 2013 it was a bet, and quite a few people even within the company were not sure it would work out. It turned out to be a massively successful bet.

Amin Vahdat

第一代芯片只做推理，主要是语言翻译和语音识别；第二代把同样的思路用于训练。大约在第二代出来时，Transformer 诞生了，这彻底改变了 TPU 项目。我们也发现推荐系统在 TPU 上跑得很好，广告等场景随之而来。之后随着生成式 AI 兴起，系统越做越大、越做越可扩展。

The first chip was all about inference—language translation primarily, and voice recognition. The second chip was a training chip. Around the time the second chip came out, transformers were invented, and that completely shifted the TPU program. We also learned recommender systems run really well on TPUs, so ads and related use cases came along. Then, as the GenAI moment took hold, ever larger, ever more scalable systems.

Amin Vahdat

每一代我们都在想要不要进一步专用化。大约两年多前我们面对的问题是：2026 年该做一款芯片还是两款？最后今年发布了两款：推理用的 TPU 8i 和训练用的 TPU 8t。几年前未必如此，但到 2026 年推理和服务真正起飞了，我们认为它可能占芯片生命周期内 30%、40%、50%、60% 的市场，做一款推理明显更快的芯片就很合理。如果推理只占 2% 或 5%，哪怕专用芯片快 2 倍，也不如一款统一的通用芯片。

The question we were facing two years ago or a bit more was: in 2026, should we have two chips or one? This led to the release of two chips this year: TPU 8i for inference, TPU 8t for training. It wasn’t necessarily the case a couple of years ago, but by ‘26 we saw inference and serving really taking off. A chip significantly faster for serving—which we thought might be 30, 40, 50, 60% of the market in its lifetime—started making a lot of sense. If we projected inference at 2% or 5% of the market, even a 2x faster specialized chip might not make sense, because you’d want the uniformity of one general-purpose chip.

Amin Vahdat

关键还有一点：8i 和 8t 都能做对方的工作。如果 8i 完全不能训练、8t 完全不能推理，我们就得在硬件 6 年寿命里提前精确预测每种各需多少。现在两者各自擅长本职，但哪边有富余容量都能顶上。专用化过头，会把自己的灵活性和可替代性锁死。

For our 8i and 8t chips, both chips can do the other workload. This is also key. If 8i had zero performance for training and 8t zero for inference, we would have to predict ahead of time, over the six-year lifetime of the hardware, exactly how much we need of each. Here both chips are better at what they’re specialized for, but if you had leftover capacity in one place, each can do the other’s job. You might get so specialized that you limit yourself from being flexible, fungible.

Sonya Huang

问个有挑衅性的问题：既然现代 AI 大多基于 Transformer，为什么不把 Transformer 架构直接烧进芯片？

To ask a provocative question: so much of the modern AI market is transformer-based. Why not just burn the transformer architecture into the chip?

Amin Vahdat

Transformer 归根结底是向量与矩阵乘法、softmax 等一组线性代数原语，这些我们和别家基本都已烧进硬件了。再往下是你的模型架构：多少层、层间怎么走、每个维度的矩阵和向量是什么形状。你可以进一步专用化到你的模型本身，这是下一层的专用化，有不少公司在想这件事，我觉得也是非常有意思的方向。

Transformers, in the end, are about vector and matrix multiply operations and a softmax—a range of linear algebra primitives. We have those, and others do as well, roughly baked into the hardware. Then there’s your model architecture: how many layers, how you go through and across them, what exact shapes of matrices and vectors. You could specialize to not just the transformer but your model. That’s the next level of specialization. A number of companies are thinking about that; it’s a very interesting direction.

Amin Vahdat

TPU 和 GPU 各有更适合的问题。GPU 比 TPU 更通用，这很清楚，它们是了不起的产品。Google 和 Google Cloud 卖大量 GPU，内部也用 GPU。两者有重叠，客户会评估自己的工作负载；我们希望给客户选择。

There’s for sure a set of problems better suited for one or the other. GPUs are more general-purpose than TPUs; they’re incredible products. At Google and Google Cloud, we sell a lot of GPUs and use GPUs internally. There is overlap between the two, and our customers evaluate their workload and options. What we like to do at Google is give our customers choice.

Amin Vahdat

协同设计有巨大的优化空间。反过来，你也可以设计抽象层，做到任何硬件、任何软件、任何网络拓扑都能跑，系统完全自适应。好处是你能去任何地方，新容量一夜之间可用你就能用上，没有锁定；坏处是一旦对所有人的软硬件栈完全通用，你多半会在桌上留下大量性能。

There are huge optimization opportunities with co-design. Alternatively, you could design abstraction layers that say: I can run on any hardware, any software, any network topology, and my system is fully adaptive. You can move anywhere; new capacity becomes available overnight and you’re up and running. You have no lock-in. The downside is you probably leave a lot of performance on the table if you become fully flexible to anybody’s hardware, software, network, storage, compute stack.

Amin Vahdat

支持协同设计的理由是：追求完全通用时，每两层之间都有很大的阻抗失配，每层可能差 10%、20% 乃至 2 倍。这些优化空间层层相乘，就是端到端的巨大机会，无论用每瓦智能还是每瓦产出衡量，一直延伸到供电和软件优化。

The case for co-designing is that between each layer there’s a big impedance mismatch if you try to get full generality—10%, 20%, 2x across each layer. Multiply those optimization opportunities through, and you’re left with a big end-to-end opportunity in intelligence per watt or output per watt, all the way down to power delivery and software optimizations.

Sonya Huang

据我理解，OpenAI 主要在同构算力栈上构建，Anthropic 则更异构。协同设计是不是它们据传模型架构差异很大的原因之一？

My understanding is OpenAI was primarily building on a homogeneous compute stack, and Anthropic on a more heterogeneous one. Is co-design part of why they converged on what is rumored to be very different model architectures?

Amin Vahdat

我不想揣测 OpenAI 和 Anthropic 在做什么。这是一种可能，但在不了解细节的情况下，我想架构不同可能有很多原因。

I don’t want to speculate on what OpenAI and Anthropic are doing. It’s one possibility, but without knowing the details, I would imagine there could be many reasons why they wind up with different architectures.

Amin Vahdat

在 Google 最有趣、最有成就感的事之一，就是能和 DeepMind 团队肩并肩地协同设计硬件和模型。还有第三个维度是消费级服务和 Cloud，但和 DeepMind 之间是真正深入的合作。

One of the most fun and gratifying parts of being at Google is the opportunity to work shoulder to shoulder with the DeepMind team on co-design of our hardware and models. There’s a third element with consumer services and Cloud, but with DeepMind it really is a deep partnership.

Amin Vahdat

举个过去的例子：他们想出某种模型优化，而我们有一款芯片还没做完。他们说「如果硬件支持这个，端到端训练或推理会显著更快、更高效，要改正在执行的硬件定义需要什么？」于是双方工程师和研究员会在一个房间里密集工作几天、一周、两周。更常见的结论是：「你要的我们做不到，但能做另一件事，你的模型架构往那边改一改，你拿到 98%，我们拿到 90%。」或者我们说：「这么大的收益，值得把流片推迟一两周。」

They have come up with model optimizations—to transformers or particular math—while we have a chip in progress, not quite done. They say, “If we had hardware support for this, our end-to-end training or serving might get significantly faster. What would it take to change the hardware definition we’re in execution on?” Then our engineers and their researchers get together intensely in the same room for a few days, a week, two weeks. More likely: “We couldn’t quite do what you wanted, but we can do this other thing, and maybe you change your model architecture in this direction. That gives you 98%, that gives us 90%.” Or we say, “We’re going to delay the tape-out by a week or two, because for this level of benefit it’s totally worth it.”

Amin Vahdat

任何时刻我们都有好几代芯片在进行：已量产的；已从晶圆厂回来正在调试的；即将流片的；处于设计阶段的；还有处于概念阶段的。从推出到退役，是一条五六个阶段、跨越多年的流水线。

We have many generations of chips in progress at any point in time: the chips in production; the chips back from the manufacturer that we’re debugging; the chips in implementation about to tape out; the chips in design; and the chips in concept. So it’s a five-, six-stage, many-year pipeline from introduction to end of life.

Amin Vahdat

对量产芯片，我们一起最大化每瓦交付的智能或 goodput，因为模型里发生什么、硬件里发生什么、中间每一层发生什么我们都清楚。对即将流片的芯片，我们能在途中修改芯片架构——如果跨公司合作，这从困难到几乎不可能。对设计中的芯片，我们会问 DeepMind 两三年后模型架构往哪走，把我们能做的帕累托前沿和模型方向的帕累托前沿摆在一起，还有很强的仿真设施预测工作负载如何映射到不同硬件。我每周会和 Koray 或 Demis 交流好几次。

For in-production chips, we work together maximizing delivered goodput per watt—we know exactly what’s happening in the model, the hardware, and everything in between. For chips about to tape out, we can make changes to the chip architecture in flight, which would be somewhere between hard and impossible across company boundaries. For chips in design, we ask our DeepMind colleagues where model architectures are going in two or three years: here’s the Pareto of what we can do, here’s where models are going. We have deep simulation infrastructure that predicts how workloads map to hardware architectures. Same building, same rooms, deep daily interaction. I talk to Koray or Demis multiple times a week.

Sonya Huang

终有一天他们的模型会帮着设计芯片吧。

And eventually their models will help with chip design.

Amin Vahdat

我们已经在用 Gemini 为未来的 Gemini 设计硬件了。

We’re using Gemini to design hardware for future Geminis as well.

Amin Vahdat

硬件我们要提前两到五年规划，TPU 8i、8t 刚发布，可以想见 9、10 乃至更多代正在概念或执行中。而做模型架构的人默认不会想那么远。但 Google Research、DeepMind、Transformer 的诞生都发生在相互重叠的房间里，整整一代研究员习惯了能影响硬件，也知道硬件是多年周期。为一颗即将流片的芯片带来 0.5% 或 1% 的小改动，他们多半不会来找我们，因为叫停流片是大事；但真正的好机会，我们一定会一起想办法塞进去。两年前的点子如今已在量产、让 Google 所有工作负载跑得更快，这种感觉很好。

We are planning hardware two, three, four, five years in advance. We just announced TPU 8i and 8t, but you can imagine 9, 10 and maybe others are in concept or execution, years out. With model architecture, by default you’re not thinking years out. But Google Research and DeepMind and transformers being invented—all of this happened in overlapping rooms. A whole generation of researchers has grown accustomed to influencing the hardware and knowing it operates on multi-year cycles. If they have a small tweak worth 1% or 0.5% for a chip about to tape out, they’re probably not going to come to us—stopping a tape-out is a big deal. But if they’ve got a really good opportunity, we’ll absolutely work together to get it in. The idea I had two years ago is in production now, helping all the workloads at Google go faster. That’s a good feeling.

Sonya Huang

但预测五年后哪些工作负载最常见、会有哪些算法突破，看起来是不可能的任务。

It seems like an impossible task, though, to predict in five years what workloads will be most common and what algorithmic breakthroughs will have occurred.

Amin Vahdat

看起来确实如此。但惊人的是——我们正在把它详细写成文章——在中等粒度上，TPU 架构从 TPU v1 至今基本没变。就像 CPU 的指令集，有 load、store、加减、分支，上面的软件却变了无数次。TPU 也有一组基础原语：为超大矩阵乘单元专门设计的数值格式；处理向量运算和 scatter-gather 的 SparseCore；通过 ICI 网络读写远程内存的 remote load-store。定义它的大概就五六样东西，它们延续了很多代模型和神经网络结构。

I hear you that it would seem impossible. But the stunning thing—and we’re working on writing this up in great detail—is that the TPU architecture, at a medium level of detail, hasn’t really changed since TPU v1. Look at the instruction set of a CPU: loads, stores, adds, subtracts, branches—and look at what the software on top has done. Same with TPUs. We’ve extended it, but the primitives are there: specializing the numerics to very large matrix multiply units; a SparseCore that manages vector and scatter-gather operations; remote load-store, reading and writing remote memory through our ICI network. Five or six things really define it, and they’ve extended to many generations of models and neural network structures.

Sonya Huang

过去一年最大的工作负载变化之一——我觉得从今年年初开始——是长程智能体的兴起。这对数据中心意味着什么？

One of the biggest changes in workloads over the last year—I think it really started at the beginning of this calendar year—was the rise of the long-horizon agent. What does that mean in terms of data center needs?

Amin Vahdat

有两大方面。第一，不再是人与加速器的交互。你在浏览器或手机上输入提示词，回复来了你得读、得想，再追问，交互间隔是好几秒。长程智能体里没有人来天然限速：一拿到回复就解析、推理一下、决定下一个请求，交互间隔从几秒、几十秒缩到也许几毫秒。

There are two huge aspects. One is it’s no longer human-to-accelerator interaction. When you type a prompt, the response comes back, you read it, think about it, maybe follow up—multiple seconds of interaction time. In long-horizon agents, there’s no human in the loop to rate-limit requests to the model. As soon as I get a response, I parse it, reason about it, figure out my next request. So what went from seconds, maybe tens of seconds, is now going into perhaps milliseconds.

Amin Vahdat

第二，这些推理和解析大多发生在 CPU 上，CPU 还得去收集下一次提示所需的状态和上下文——可能在本地 DRAM、别的 CPU 的 DRAM、SSD 或远处的 HDD。大量编排随之而来。所以设计在显著变化：加速计算需求在涨，而 CPU、网络、存储这些传统数据中心需求也在暴涨。

Two is all that reasoning and parsing is probably going to happen on a CPU, which then has to gather other state before the next prompt—context from local DRAM, someone else’s DRAM, SSD, or HDD somewhere else. A huge amount of orchestration has to take place. So the design is changing significantly: demand for accelerated compute is going up, but demand for CPU, networking and storage—traditional data center CPU—is also going through the roof.

Amin Vahdat

如果在 GPU、TPU 机架旁边放大量 CPU 机架，就没法完全按 TPU 机架的密度和网络需求去专用化——TPU 机架比 CPU 机架更密、需要更多网络，楼的设计就得变。另一种选择是保持统一：GPU/TPU 放一栋楼，CPU 放隔壁，硬盘也许在另一边的楼。但楼与楼之间需要大规模网络，一出楼，可靠性、成本的复杂度都显著上升，延迟也会到数百微秒甚至更多。

If we start putting lots of CPU racks next to the GPUs and TPUs, we can’t fully specialize to the density and network requirements of a TPU rack—a TPU rack is denser and needs more networking than a CPU rack—so our building design changes. Another option is to maintain uniformity: GPUs or TPUs in one building, CPUs in the building next door, maybe hard drives in another. Now you need significant networking between buildings. Once you leave a building, complexity goes up significantly in reliability and cost, and latency might go to hundreds of microseconds, potentially more, with queuing.

Amin Vahdat

大约十五六年前，Google 率先在数据中心内部引入波分复用（一根光纤承载多路信号），机架间通信全都用它。同时我们引入了光路交换（OCS）：与传统电分组交换不同，它完全在光域里传输和转发数据。

Probably 15, 16 years ago, Google was among the first to bring wavelength-division multiplexing—multiple signals on a single fiber—within the data center, and we leverage that for all communication between racks. At the same time, we introduced optical circuit switching. In contrast with traditional electrical packet switching, it transmits and moves data entirely in the optical domain.

Amin Vahdat

分组交换要在电域读包头、查表决定从哪个端口转发，每秒处理数十亿乃至万亿个包。光路交换则不碰这些比特，只决定把某个输入端口的光送到哪个输出端口。我们最早用 MEMS 开关，用微型电机控制三维空间中的镜面；一台设备可能有 128 或 256 个端口，可编程地把每根入射光纤映射到输出端口，光照在镜面上被反射到正确的出口。

In a packet switch, you look at a packet header in the electrical domain, look up the destination in a table, and forward it—billions, perhaps trillions, of packets per second. Optical circuit switching says: I’m not touching these bits electrically; I just decide, for an input port, which output port to send the light to. We started with MEMS switches—micro-electrical motors controlling mirrors in 3D. A box might have 128 or 256 ports; we programmatically map every input fiber to an output port by rotating mirrors, and the light literally shines on these mirrors and reflects to the right output.

Amin Vahdat

最初有两个原因：一是在机架组之间建立局部性，比如为搜索服务的计算集群和存储集群通信很多，就用镜面在它们之间纯光学地搭捷径；二是不挪动任何光纤就能扩张或收缩网络脊柱。

Initially it was twofold. One, to create locality between groups of racks: if a compute cluster and a storage cluster for, say, search talk a lot, we configure the mirrors to create purely optical shortcuts between them. Two, to expand and contract the network spine without moving any fiber.

Amin Vahdat

到了 TPU，TPU 采用把所有芯片直接互连的环面（torus）拓扑。我们随时备有一个空闲机架，某个 TPU 机架故障时，把光重定向到新机架，不用动任何光纤，几毫秒内完成。

Fast-forward to TPUs. A TPU has a torus topology that connects all the TPUs to one another directly. If a TPU rack fails, we can replace it with another without moving any fiber. We have a spare rack available at all times, and when a rack fails, we redirect the light to the new rack. That can be done in milliseconds.

Sonya Huang

那为什么还要光纤，不全用自由空间光？

Why do you have the fiber at all, then?

Amin Vahdat

因为衰减损耗，带宽会大幅下降；而且在一栋巨大的建筑里不借助光纤做三维对准，从困难到几乎不可能。我们讨论过，但目前主要还是光纤，到了光路交换机里光才照到那些小芯片上。数据中心里网络的能力和需求都在爆发。

Attenuation loss, and bandwidth would drop dramatically. And across a very large building, aiming everything in 3D without fiber would be challenging to possibly impossible. We’ve talked about it. But it’s mostly in fiber, and when it hits the optical circuit switch, the light shines down on these tiny chips. Networking is exploding in terms of its capabilities and need in the data center.

Amin Vahdat

常有人问最大的约束是什么。现实是没有单一最大的约束，它们都很难，而且不断变化。但如果非要从根本上回答，电力是最根本的约束。其他的我们似乎都知道怎么解决，只是时间问题；电力是一个长期的硬约束。核能也许、丰富的清洁能源会解决很多问题，但何时能规模化还未知。

The reality is there is no single biggest constraint—they’re all super hard and they shift continuously. But if I had to answer fundamentally, power is the single most fundamental constraint. Everything else, it seems like we know how to solve over some period of time. Power is a binding, long-term issue. Nuclear perhaps—abundant clean energy would solve a lot of problems when it happens, and when it happens at scale is still unknown.

Amin Vahdat

Google 首选的模式始终是并网——相当于很客气地给当地电力公司打电话，当然要提前很多年打招呼。GW 级不可能「明天就要，什么时候开始计费？」，而是一起规划。我们也很认真地确保：为我们扩容的成本——新建或升级输电线路、变电站等——由我们承担，否则按计费方式，理论上别人的电价可能因此上涨。

Our preferred model at Google is always to be utility-connected, grid-connected. You call, very politely, your favorite utility, and you give them many, many years of notice. At gigawatt scale, it’s not “I need a gigawatt tomorrow, when can you start billing me?” It’s something we plan together. We also ensure that the costs of putting that infrastructure in place—transmission lines built or upgraded, additional base stations—we pay for, because otherwise, by the act of building capacity for us, other people’s rates could go up, in theory.

Amin Vahdat

比如我们 2028 年需要 1GW（日期是我编的），电力公司 2029 年才能给到 1GW，2028 年只能给 700MW。那 300MW 怎么办？一种答案是等；另一种是自己发一部分，比如太阳能加电池备份，或其他来源。也可以保留部分本地发电，即使电网全部到位后，在一年中最热、居民用电最旺的那两周把电回馈电网。

Let’s say we need a gigawatt in, I’ll make up a date, 2028, and the utility can get us a gigawatt in 2029, but 700 megawatts in 2028. How do we cover those 300 megawatts? One answer is just wait. Another is to generate some subset ourselves—solar cells, batteries as backup, other sources. We might maintain some local generation even once the utility is fully online, and provide power back to the grid—say the hottest two weeks of the year with huge residential demand.

Amin Vahdat

为什么不自己垂直整合？主要是灵活性和双方受益，也就是统计复用、大数定律。如果要 1GW 达到 99.99% 以上的可靠性，就得 1+1 冗余，建 2GW，那很贵，还要尽量是数据中心旁边的清洁能源。和电网合作，在更大的基数上做统计复用，我们、电网、居民都受益。极少数情况下我们会做表后（behind the meter）供电，但也是以一年后并网为前提。

The main reason is flexibility and uplift on both sides—statistical multiplexing, the law of large numbers. If we need a gigawatt with 99.99%-plus reliability, we have to build two gigawatts—one-plus-one redundancy. That gets expensive, and it should ideally be clean energy right next to our data center. By leveraging statistical multiplexing over a much larger base with the utility, everybody wins: we win, the grid wins, residences win. In rare cases we do it behind the meter, but even then assuming we’ll connect to the grid, maybe a year later.

Amin Vahdat

数据中心多大合适，是门艺术，也是大量争论的来源。十到十五年前 Google 就争论过要不要把一切放进一个 GW 级数据中心。一大顾虑是单点故障；另一方面，对今天的训练负载来说越大越好，网络上希望集中在尽量小的距离内。但如今不可能在世界任何地方把 Google 的全部需求建在一处。所以规模因地而异：网络边缘或境内部署可能几十 MW，与 ISP 合作的可能就是一个机架，训练集群接近 1GW，其他站点几百 MW。

How big to make a data center is an art and a source of significant debate. 10, 15 years ago there was a big debate at Google: should we put everything into one data center, a gigawatt? One concern is single point of failure. On the other hand, for training workloads today, bigger is better—you want things concentrated in as small a distance as possible. But there’s no way anywhere in the world we’ll get the total requirements of Google built in one place. At the edge or in-country, that might be tens of megawatts; with ISPs, literally a rack; a training cluster closer to a gigawatt; other sites hundreds of megawatts.

Sonya Huang

我猜最新最大的集群用来训练前沿模型，旧的回收做推理？你们也在建专门的推理集群吗？

My guess is the biggest, newest clusters train the latest frontier model, and you recycle the older stuff for inference. Are you also standing up inference-specific clusters?

Amin Vahdat

这个框架很合理，但推理需求大到不能只靠退下来的训练集群。而且某一年的训练集群会集中在少数大型站点，彼此网络距离要小，所以多半在同一大洲甚至同一区域，其他大洲的服务容量就不够，只能在全球各地建专用推理集群。训练集群几年后会用于服务，但我们也得另建服务集群。

It’s a very reasonable framework, but the demand for inference is such that we can’t just rely on older training clusters. In a particular year we might centralize training into a small number of large sites, keeping network distance small—probably the same continent, or the same portion of a continent. Then you’re left with not enough serving capacity on other continents, so we build specialized inference elsewhere across the world. Yes, training clusters will be used for serving in some number of years, but we also have to build serving clusters.

Amin Vahdat

服务集群不一定更便宜，因为服务更需要把计算、网络、存储放在一起，没法像训练那样做大块、密集、统一的部署。而且服务不希望在一处放太多，要从全球各地提供，还有不同的模型端点和模型变体要按局部性分布到世界各地。所以它们更小，垂直整合程度也更低。

Not necessarily cheaper, because for serving there’s more need to co-locate compute, networking and storage, so we can’t specialize the way we do with big, dense, uniform training deployments. For serving, you don’t want too much in one place; you want to serve from all over the planet, with individual model endpoints and model variations spread out accounting for locality. So they will be smaller and less vertically integrated.

Amin Vahdat

我公开说过——没想到会引起这么大反响——我们七八年前的 TPU 仍是 100% 利用率。我们最终会替换它们：折旧期大约 6 年，折旧完、加上新一代能效更高，替换升级就合理了。

I’ve been on record saying—I was surprised how much pickup this had—that our seven- and eight-year-old TPUs are still at 100% utilization. We do replace them in the end. The depreciation lifetime is approximately six years, so once they’re fully depreciated, given the power efficiency of newer generations, it makes sense to replace and upgrade them.

Amin Vahdat

但我们替换的其实是系统，以 pod 为单位。一个 TPU pod 可能有 9,600 颗芯片、大约 152 个机架。拆掉旧 pod，换成 TPU 12、13 或 14 的 pod，新 pod 的占地未必刚好塞进旧 pod 的空位，这没法提前规划，因为不知道那么多代之后的 TPU 是什么样，所以如何退役再尽快换上新 TPU，也是门艺术，而且是苦活。

We replace the chips, but it’s really the systems—we think in pods. A TPU pod might be 9,600 chips and 140-something, 152 racks. We pull out that pod and replace it with a TPU 12 or 13 or 14 pod, whose footprint might not perfectly fit the old pod’s vacancy. We can’t plan that, because we don’t know what that many generations of TPUs will be. So figuring out how to decom and replace as quickly as possible is a bit of an art, and hard work.

Amin Vahdat

关于开放标准：我们支持垂直整合，也让你能榨出想要的全部性能，但不强迫锁定、不搞围墙花园。比如 Google 开发了 JAX，我们很喜欢、内部大量使用；但很多客户喜欢 PyTorch。我们可以说「想用 TPU 就得用 JAX」，也可以说「喜欢 JAX 就用 JAX；喜欢 PyTorch，我们有 TorchTPU，未修改的模型就能跑」。

While we support vertical integration and make it possible to extract as much performance as you want, it’s really important to not force lock-in or a walled garden. At Google we developed JAX; we like it a lot and use it extensively internally. Many of our customers like PyTorch. We could say, “If you want TPUs, you’ve got to use JAX.” Or: if you like JAX, use it; if you like PyTorch, we have TorchTPU, where your unmodified models can run.

Amin Vahdat

为什么 IP 协议赢了？七八十年代有很多竞争协议，IP 赢在它是开放标准，是沙漏的细腰：上面能跑任何软件，下面能接任何硬件。谁带着 IP 插上路由器端口就成了互联网的一部分，这让互联网爆发并遍及全球。我们相信开放的接入点，最好周边也开源；这也是我们全面支持 CPU、GPU 和其他加速器的原因。

Why did the Internet Protocol win? There were many competing protocols in the ’70s and early ’80s. IP won because it was an open standard, the narrow waist of the hourglass: anything on top in software, anything underneath in hardware. Anyone who brought IP could plug into a router port and become part of the internet. That allowed the internet to explode across the world. We really believe in open standards for plug-in points, ideally open source around it as well. That’s also why we fully support CPUs, GPUs, other accelerators.

Sonya Huang

AI 怎样改变了你团队的日常？

How has day-to-day life for your team changed with AI?

Amin Vahdat

最容易说的是软件工程，外界报道很多，我们用它做开发、测试发布，甚至辅助设计。但硬件侧的变化外界关注较少：我今天刚看过数字，按 token 用量——不是最好的指标，但也是指标——我们的硬件工程师用 AI 的量和软件工程师一样多。生产力提升了，从设计启动到流片的时间在缩短，芯片点亮调试（bring-up）的时间也在缩短。

The easiest answer is software engineering, which has been documented externally quite a bit—we use it for development, test rollouts, even helping with design. But on the hardware side there’s also been significant change. I was looking at the numbers earlier today: by token counts—not the best metric, but still a metric—our hardware engineers are using AI as much as the software engineers. Productivity has gone up. Time from design kickoff to tape-out is shrinking. Time for bring-up is shrinking.

Amin Vahdat

意料之外的变化还有数据中心设计：是建 1GW 的楼还是 100MW、200MW 的园区，过去是非常细致、靠电子表格和人工的流程，如今很多 AI 参与进来，大大简化规划。还不算推理模型在做决定，它不取代人的判断，而是把必要信息汇集到一处，摆在做决定的人面前。

Less expected: how we do data center design has changed significantly. Whether to do a gigawatt building or a 100- or 200-megawatt campus used to be a very detailed, spreadsheet-driven, human-driven process. There’s a lot of AI involved now that streamlines planning. I wouldn’t say reasoning models quite yet. It’s not replacing human judgment, but it brings the necessary information together and puts it in front of the humans making the decisions.

Amin Vahdat

这是个令人兴奋的方向，我们确实在做，并且毫不玩笑地称之为一个 moonshot。回到前面说的，能源是根本性的关键挑战。在太空中，由于没有大气衰减，可用太阳能多约 40%，即 1.4 倍；在太阳同步轨道上，太阳能板有 98%–100% 的日照覆盖，而地面只有 28%、30%，也许 35%。1.4 倍乘以每天日照时长的 3–4 倍，还基本省掉电池，而且零碳。

It’s an exciting direction. We are pursuing it, and we’ve referred to it, with no tongue in cheek, as a moonshot. In terms of fundamental constraints, energy is a key challenge. In space you have something like 40% more power because of lack of atmospheric attenuation—1.4x more solar capacity. And in a sun-synchronous orbit you have 98 to 100% sunlight coverage on your solar cells, versus 28, 30, maybe 35% on land. Take the 1.4x and the 3 to 4x in hours per day, remove batteries more or less from the equation—and of course carbon-free.

Amin Vahdat

挑战也非常多。直觉上太空散热更容易，其实更难；可靠性方面，东西会坏，在太空修理更难，不是不可能。冗余会是你的朋友。你之前问的自由空间光会成为现实：我们不太可能在组件间拉光纤，而是用激光对准接收器并实时校准。没有根本性的拦路虎。

Lots and lots of challenges. Naively you might think cooling in space is easier; it’s actually harder. Reliability—things fail, and repairs become harder, not impossible. Redundancy will probably be your friend. Your earlier question on free-space optics is now going to become reality: we’re probably not going to string fiber between these components, so we’ll have lasers pointing at receivers, calibrating in real time. There are no fundamental showstoppers.

Sonya Huang

最后一个问题：十年后最前沿的超级计算机会是什么样？

Last question: 10 years from now, what will the most frontier supercomputer look like?

Amin Vahdat

十年正好在边缘上，也许有人在想 2036 年的计算机，但不确定性的锥面太宽了。看趋势，集成度会非常高。我们给 Ineffable 装的 VR200 机架里有漂亮的光纤；我猜将来集成度更高、光纤更少——不会是零——可能只有一小束光纤从机架里出来。从模块化制造看，2036 年的机架很可能集中制造，内置 72、144、288，甚至 576、1,152 颗 GPU 或 TPU。一个机架多少功率？我们不确定怎么做到，但可以想象单机架数兆瓦。接上水、电、光纤，推进来插上就能跑。

10 years is right at that edge—probably some people are thinking about what the 2036 computer looks like, but the cone of uncertainty is way too wide. If you look at the trends, the level of integration is going to be tremendous. We saw the beautiful fiber in the VR200 racks for Ineffable. My guess is more integrated, less fiber—I won’t say no fiber in 2036—maybe a small fiber bundle coming out of the rack. These racks are likely going to be manufactured centrally, with 72 or 144 or 288, or probably more—576, 1,152—GPUs or TPUs deeply integrated. Multiple megawatts in a single rack. You bring water, power and fiber, wheel that rack in, plug those three things in, and you’re off to the races.

Sonya Huang

所以不会是太空里一个巨大的外星球体？

So it’s not going to be a big alien orb off in space, then?

Amin Vahdat

到 2036 年，也不是没可能直接发射到太空，再由某个空间站的机械臂取下来插进对应的模块。想想就很有意思。我们有幸亲历并定义这一切。

By 2036, I wouldn’t say it’s not possible for it to be launched directly into space and then taken by a robotic arm at some space station and plugged into the right module. Fun to think about. We get to live through this, and we get to define it.