Anastasis Germanidis：Runway 押注世界模型、机器人与神经操作系统

Anastasis

接口世界模型的终局，其实是一套完全神经化的操作系统。Andrej Karpathy 很早就写过这个方向。

Effectively the endgame of something like interface world models is you have a fully neural operating system. Andrej Karpathy has written about that quite a while back.

Anastasis

今天的 LLM 几乎什么都能聊、任务也很通，但你还是透过一套僵硬的界面跟它交互。迟早界面本身也会变成可学习的一环——你交付的不只是语言模型，还有渲染与像素，那同样是可学习的组件。

With an LLM of today you can talk about anything and solve many tasks, but you interact through a very rigid interface. It’s a matter of time before the interface itself becomes learnable—you’re delivering the language model and also the render and the pixels, and that’s also a learnable component.

Swyx

（片头插播订阅 CTA 后）好，我们和 Runway 的 Anastasis 在录音室，我和 Vibhu。欢迎。

Okay, we’re here with Anastasis from Runway, with me and Vibhu in the studio. Welcome.

Anastasis

很高兴来。

Good to be here.

Swyx

恭喜 Runway 一路走到现在，办公室开到世界各地。创业时想到会这样吗？

Congrats on all your success and progress with Runway. You’re opening offices all over the world. Did you envision this when you first started out?

Anastasis

不完全是。我们一开始就觉得：2016、2017 年那些早期生成模型，分辨率和质量会可预期地涨——问题是何时而非是否。终有一天多数内容会被生成；Runway 最初的论点是：因此必须重想创作工具怎么做。把研究做深之后，又发现这些模型的用途远不止创作。

Not quite. Even when we started, it was more a matter of when, not if—we were seeing the early generative models of 2016–2017 and extrapolating that resolution and quality would increase predictably. There’d be a point where most content would be generated, and the initial thesis of Runway was we’d need to rethink how creative tools are made. As we built out the research, it became clear they were useful far beyond that as well.

Swyx

你怎么从 Zocdoc、计算机视觉一路走到和 Chris 等人创办 Runway？

I’m curious how you go from a background in Zocdoc and computer vision into Runway. Take us back to those early conversations with Chris and the founding team.

Anastasis

我一直在两边切：一边做交互艺术，一边在创业公司当 ML / 后端工程师。我对编码、计算尤其是仿真很着迷，也带回早期作品。比如 2015 年一个项目：在画廊里用软件给观众语音指令、分配身份（建筑师、30 岁、喜欢运动），再配对，用模板加马尔可夫链生成闲聊——当时语言模型还不行。一边是生成模型，一边是「用极简交互模型能学到人类什么」。

I was always splitting two worlds—my own interactive art practice, and working as an ML / backend engineer at startups. I’ve always been interested in coding, computation, and especially simulation. One 2015 project gave voice instructions in a gallery, assigned identities—you’re an architect, 30, you like sports—matched people, and simulated small talk with templates and Markov-chain text; language models weren’t there yet. I was fascinated by early generative ML and by what simple models of interaction teach us about humans.

Anastasis

和联合创始人 Chris 最早的项目之一叫 Uncanny Valley / Uncanny Road：用 NVIDIA 2016–2017 左右的 pix2pixHD，把语义图变成「接近」照片级输出——大概是最早能到约 1K 的图像生成模型之一，全训在自动驾驶路上数据：行人、标志、车、红绿灯。人们却做出百万行人、巨型人体等超现实图——同一套「无聊」路上模型，换工具给艺术家，会跑出分布外、却有艺术力的东西。这在某种程度上就是 Runway 的缩影。

One of the first projects with co-founder Chris was Uncanny Valley / Uncanny Road—pix2pixHD from NVIDIA around 2016–2017, taking a semantic map to a photorealistic-ish output. Very early, not high-fidelity, but maybe the first image model that could generate at 1K, trained on self-driving road datasets: pedestrians, signs, bikes, stoplights. People made surreal imagery—a million pedestrians, gigantic humans. You could take a boring road dataset model, repurpose it out of distribution, and make something artistically compelling. That’s a summary of Runway’s thesis in some ways.

Vibhu

你们做了七年，怎么从驾驶仿真数据走到覆盖整条生成媒体栈？

You’ve been doing this for seven years at Runway. How do we go from driving simulator data to covering the whole stack of generative media?

Anastasis

有趣的是我们几乎绕回原点：现在又把模型用到真实世界场景，不只创作工具。第一版 Runway 是让艺术家轻松用上当时的开源模型（如 pix2pix）——对非 ML 工程师太难了。很快意识到必须建研究组织，大概第一年就建了：当时的图像/视频生成还撑不起可量产的创作工作流，必须推研究前沿。前四年研究几乎在后台；直到 2022 年 latent diffusion、DALL·E 2 那种阶跃。

Interestingly we’re almost full circle—applying models beyond creative tools into real-world scenarios. The first version of Runway made open-source models of the day like pix2pix usable for artists; those models were too hard if you weren’t an ML engineer. Very quickly we realized we needed a research org—year one. Image and video models of the time weren’t productionizable in creative workflows, so we had to push the frontier. For maybe the first four years research happened in the background until 2022—latent diffusion, DALL·E 2—that step-function change.

Anastasis

2018 到 2022 之间我们很早就做图像/视频分割——特效大量依赖把主体抠出来，rotoscope 极痛苦。早期主力产品叫 Green Screen，很长一段时间是人们用 Runway 的主因，进过《瞬息全宇宙》等片。在 latent diffusion 和 Gen-1、Gen-2 之前，Runway 本质上是后期工具。

Between 2018 and 2022 one early line was image and video segmentation—most VFX is separating subjects; rotoscoping is extremely manual. A lot of early Runway was Green Screen; for a long time that was the main thing people used Runway for. It ended up in Everything Everywhere All at Once and other high-visibility films. Until latent diffusion and Gen-1 / Gen-2, Runway was essentially a post-production tool.

Anastasis

到 2022 年中很清楚：我们在很小算力上做研究，但缩放律同样适用于图像/视频。我们下了大注——Series B 阶段签了约一千张 A100 的集群，略显非理性，但坚信大规模训视频模型能出好模型。目标是：视频的 latent diffusion / Stable Diffusion 时刻长什么样？当时最好的大概是 CogVideo，约 256×256、质量一般。我们想做文生视频，但发现更强条件更容易——从视频到视频起步：重风格化已有视频，比从零生成容易。于是 2023 年 1 月先发 Gen-1。

By mid-2022 it was clear scaling laws would apply to image and video gen like language. We made a big bet—as a Series B startup we signed a deal for a cluster of a thousand A100s, slightly irrational, but we believed training a video model at large scale would yield a great model. The goal around fall 2022: what does the latent diffusion / Stable Diffusion moment look like for video? The best then was CogVideo—256×256, not high quality. We wanted text-to-video, but stronger conditioning is easier: start video-to-video—restylizing existing video vs generating from scratch. We released Gen-1 in January 2023.

Anastasis

Gen-1 是深度条件视频模型：先把输入视频变成深度图，再用 latent diffusion 生成像素。有人用书或纸箱搭城、手机拍再转成接近照片级；也有人把无贴图 3D、Blender / Unity 场景拍下来再重风格化。我仍觉得视频到视频很强——用真实视频当地基再转到不同风格。

Gen-1 was a depth-conditioned video model—convert the input to a depth map, then generate pixels with a latent diffusion model. People made cities of books or boxes, shot on a phone, translated to photorealistic output; power VFX editors took Blender or Unity captures and restylized them. I still think video-to-video is powerful—ground-truth video as inspiration, then translate to different styles.

Swyx

想补一段 Stable Diffusion / Stability AI 的往事。你写过长文；现在回看怎么说？

I did want to cover the Stable Diffusion controversy, or what happened with Stability AI. Looking back with some years behind it—what’s the retrospective?

Anastasis

故事很长。大致是：2021 年底有 latent diffusion 论文；当时在 Runway 的 Patrick Esser 与 CompVis 的 Robin Rombach 等人合作。之后目标是继续迭代、加数据、加任务。Stable Diffusion 本质上是同一架构、更多算力，加上 classifier-free guidance 等技巧，训在更好的数据（如 LAION 的 aesthetic 子集）上；那次大训练跑在 Stability 的集群上、由他们出资。对我们来说它本是开源研究项目，应继续发版本——于是有了 Stable Diffusion 1.5。中间有过约一天的沟通误会，几小时内就解决了。现在都过去了：Stability 走自己的路，Runway 走自己的路。

It’s a very long story. Essentially the latent diffusion paper came at the end of 2021; Patrick Esser, then at Runway, built it in collaboration with Robin Rombach and others at CompVis. The goal was to keep iterating—more compute, data, tasks. Stable Diffusion was the same underlying architecture with more compute and tricks like classifier-free guidance, trained on better data like the aesthetic subset of LAION; that big run happened on Stability’s cluster, financed by them. For us it was meant to be an open-source research project, so we continued releasing versions—that led to Stable Diffusion 1.5. There was maybe a day of miscommunication, resolved within hours. It’s all in the past; Stability took its path, Runway took ours.

Anastasis

我很怀念 2018–2022 那个小圈子——大家都确信这会成大事，做个项目就容易病毒式传播。早期爆款包括 neural style transfer、Deep Dream；Jim Kogan（早期顾问）拍自己坐纽约地铁过 Williamsburg Bridge，再用梵高之类风格化——当时觉得太酷了。距今不到十年，变化有多快。

I reminisce fondly about 2018–2022—a small community convinced this would be big; projects went viral immediately. Early viral creative AI included neural style transfer and Deep Dream. Jim Kogan, an early Runway advisor, showed himself on the New York subway over the Williamsburg Bridge stylized like Van Gogh—so cool at the time. That was maybe less than ten years ago—an indication of how quickly things have gone.

Anastasis

我们几乎从零搭视频扩散的整套 serving。Gen-2 大概是市场上第一款文生视频。很早清楚：纯文生视频不够——人要更多控制。于是很快在模型上做控制：相机轨迹、首帧等。2023 年大量是对基座做后训练：Motion Brush 画箭头规定运动方向；相机控制描述镜头怎么动。因为长期和电影人合作，反馈立刻变成投入——可控性很早就成主题。

We had to build the entire model-serving infrastructure for video diffusion—Gen-2 was the first text-to-video model out in the market, I think. It was clear early that text-to-video wouldn’t be the answer; people wanted more control. We invested in control quickly—camera trajectory, initial frame. 2023 was about conditioning: Motion Brush—draw arrows to dictate where things move; camera control—describe how the camera should move. Working with filmmakers, we got that feedback immediately—controllability became a big theme early.

Anastasis

有趣的是 Gen-2 怎么从 Gen-1 来的：Gen-1 尚未全面可用我们就宣布了 Gen-2，只隔约两个月。Gen-1 是深度到视频；直接做文/图到视频当时做不好。团队在讨论要不要再花六个月加算力训更大文生视频。我周末想了个主意：先训「文本 → 深度图」模型，再接 Gen-1 把深度变成 RGB——Gen-2 就是那条两阶段管线。知道内情能看出结构有时略怪，但够快上线。有意思的是现在又有人回到两阶段：比如 Reve 先用 planner 出 bounding box 再进 DiT；Ideogram 同一天也有类似创新——小圈子，人会交流。如今量产视频模型底下几乎都有复杂的 prompt completion。

Something fun: Gen-2 came out of Gen-1—we announced Gen-2 two months after Gen-1, before Gen-1 was even generally available. Gen-1 was depth-to-video; we couldn’t get text or image-to-video to work directly. We discussed spending six months on text-to-video at larger scale. I had a weekend idea: take a model from text to depth maps, then use Gen-1 to convert depth to RGB. Gen-2 was that. With that knowledge you can tell the structure looks a bit off sometimes, but it worked and shipped fast. People are coming back to two-stage approaches—Reve’s planner generated bounding boxes before the diffusion transformer; Ideogram the same day. Small community; people talk. Every production video model now uses a complex prompt-completion pipeline under the hood.

Anastasis

DALL·E 3 论文先公开了合成、极细 caption 很好用；Sora 也建在这上。2023 我们给 Gen-2 加相机控制、Motion Brush。相机控制第一次让人感觉不是在「做短片」，而是在世界里导航——那是世界模型研究的种子。看待这些模型有两种方式：内容机器，或为了把视频预测好、必须越来越强地仿真世界。若视频也有缩放律，加算力就会更好仿真物理、人类动作与动力学。我们大概在 2023 年末拉起专注世界模型的研究组。

The DALL·E 3 paper was the first public description that synthetic, detailed captions work really well; Sora built on that. In 2023 we released camera control and Motion Brush on Gen-2. Camera control was the first time it felt like you were navigating inside a world, not just creating a short video—maybe the seed of our world-model ideas. One way to see these models is content machines; the other is that to predict video well you need to simulate the world with increasing capacity. If scaling laws apply to video like LMs, as we scale compute they’ll simulate physics and human dynamics increasingly well. That was late 2023—we spun up a research group focused on world models.

Anastasis

与其说 pivot，不如说能力变强后应用面扩展。看 Gen-2 输出，当时并不显然它能缩放到通用世界仿真器：运动有限、分辨率低、人体解剖错误……但心态是：这是视频版 GPT-2——句子还不连贯，视频也还不连贯，缩放没有理由不灵。Runway 一直是外推趋势：2014–15 年 32×32 脸，到 2018 年已能生成约 1K 街景——世界模型也是早期信号指向更大的东西。

It’s in a way an expansion of applications as models become more capable. Looking at Gen-2 outputs, it wasn’t obvious this would scale to a general simulator of the world—limited movement, low resolution, anatomy mistakes. The idea was: that’s GPT-2—barely coherent sentences; Gen-2 barely coherent video—but if you scale it, there’s no reason it shouldn’t work. Runway’s mindset is extrapolating the trend: from 32×32 faces when the first GANs came out around 2014–15, to street images at 1K by 2018. Same with world models—very early signs of something much bigger.

Swyx

你们六月那篇视频随笔写：「人类心智不再是 AI 的中心。我们的世界才是。」过去五年 LLM 很像在拟合人类偏好与语言；那块大致解决了，焦点转向准确建模世界。

I like how you phrased it in June—“The human mind is no longer the center of AI. Our world is.” The past five years of LLM-based AI tried to emulate human preferences and speech; that’s mostly solved. Now the focus is modeling the world accurately.

Anastasis

DeepMind 早期使命是「先解决智能，再解决其他一切」；但从「其他一切」出发也可能有价值。世界复杂到很难只靠人类文字描述来学——LLM 学的是人类写过的、2020 年代我们对世界的理解，大量低层动力学根本没写进文本。系鞋带很难用文字说清，演示却很明显。我们低估潜意识技能的复杂度（某种 Moravec 悖论），把太多权重放在「好谈」的东西上。若直接在观测数据上训、而不是在「人们如何描述世界」上训，会学到否则学不到的东西。

DeepMind’s initial mission was solve intelligence then use it to solve everything else—but starting from everything else can be valuable. There’s so much complexity in the world that’s hard to learn from human descriptions alone. LMs learn from everything humans have written—our understanding as of the 2020s—and so much low-level dynamics isn’t captured in text. Describing how you tie your shoes is hard; demonstrating is obvious. We underestimate complexity in things we do subconsciously—a Moravec-like paradox—and overemphasize what’s easy to talk about. Train on observational data instead of how people describe the world, and you’d learn something you wouldn’t otherwise know.

Swyx

现有范式够用吗？还需要 JEPA 一类另一层吗？

You think the present architectural paradigm is fine? You don’t need another layer, like JEPA, as another famous New York AI leader would say?

Anastasis

我们很务实：有证据表明别的路线更好，我们不执着。目前没看到「视频预测本身不缩放」的迹象；机器人里不少有希望的工作也从视频预测起步再接到动作。把时间花在新架构上，不如改进数据与缩放现行路线。Yann LeCun 几天前还在推：理解世界动力学 ≠ 生成可爱视频。我们的回答是：它们是同一件事——当然模型可以「作弊」切镜头躲开难物理，别被精选样例骗了，还有很多工作。但类似语言：从不连贯句子到能对话、再到能自主干活——主因是规模。Physics-IQ 等基准测固体力学、流体、光学，分数随模型与算力规模可预期提升。

We’re a very pragmatic research lab. If evidence shows another approach works better, we have no qualms. We’ve seen no indication that video prediction itself doesn’t scale; some of the most promising robotics work starts from video prediction then adapts to action. Little evidence your time is better spent on a novel architecture vs improving data and scaling the current approach. Yann LeCun tweeted that understanding world dynamics is very different from generating cute videos. Our answer: they’re the same thing. Models can cheat with cuts that hide hard physics—don’t be tricked by cherry-picked demos. But like language—barely coherent sentences to conversation to autonomous work—the main thing is scale. Physics-IQ measures solid mechanics, fluid dynamics, optics; scores improve predictably with model and compute scale.

Anastasis

那些基准大致是：拍一批代表物理现象的视频，取首帧丢进图生视频，看 rollout——天花板挂着的球会怎么落地。测的是模型的「直觉物理」，和人想象瓶子掉落差不多。

Those benchmarks capture videos of physical phenomena; take the first frame, pass through an image-to-video model, generate a rollout—a ball hanging from the ceiling, predict how it falls. It measures the same intuitive physics understanding we have imagining dropping a bottle.

Anastasis

答案的一部分当然是规模——Gen-3 把这课上得很痛。Gen-3 是 2024 年、Sora 发布几个月后推出的。我们要在几个月内学完语言模型界两三年的课。Sora 一大变化是用扩散 Transformer 替代卷积；DiT 论文 2023 年显示图像扩散也有缩放律。我们花了 2023 年大半个秋季搭分布式训练、模型并行，缩放图像/视频 DiT 有很多失败。2024 年 2 月 Sora 出来，质量远超 Gen-2；推特上「Runway 完了、追不上」。OpenAI 当时如日中天。有几个小时的存在主义危机。但公司做了约八年，这种时刻逼你快速适应：三个月冲到比 Sora 更好的模型——模型规模与训练算力约 10×，从零学会模型并行——夏天推出 Gen-3。那是转折点：研究组织迅速变大，认真追求通用世界模型。

Part of the answer is definitely scale—we learned that in a big way with Gen-3, released in 2024 a few months after Sora. We had to learn in months what the LM world learned in two to three years. One of Sora’s biggest changes was diffusion transformers instead of convnets; the DiT paper in 2023 showed scaling laws for image diffusion. We spent most of fall 2023 building distributed training and model parallelism—lots of false starts scaling image and video DiTs. February 2024, Sora comes out—results very much superior to Gen-2. Chatter on Twitter: Runway’s done, no way they’ll catch up. OpenAI was on top of their game. There was an existential crisis for a few hours. But we’ve been around eight years—dinosaur in AI—and had to adapt. Anyone there says their favorite time was that three-month push to a model better than Sora: we scaled model size and compute ~10×, figured out model parallelism from zero expertise, came out with Gen-3 that summer. Big turning point—research org grew fast; we pursued the general world model in earnest after Gen-3.

Anastasis

Gen-3 后几个月我们发了 Turbo——大概是量产里第一款步数蒸馏模型，才能以更大规模式服务（首版 Gen-3 很贵）。我不会说蒸馏没留下来：很多步数蒸馏模型仍在生产用，和未蒸馏比仍有质量差距，但视频相对语言大概有两到三年偏移，更好的蒸馏只是时间问题。我们有实时角色模型，大概是最大规模的实时视频部署之一，就是步数蒸馏、约 24 FPS 的自回归说话头像。世界模型方向一大块是：从一次生成整段视频的双向扩散，走到一次生成一帧或几帧的因果自回归，再步数蒸馏到实时。我会很惊讶若两年后我们主要用的不是实时模型——体验更好、更便宜服务，质量差距会收。

A few months after Gen-3 we released Turbo—I think the first step-distilled model in production—so we could serve at larger scale; the first Gen-3 was expensive. I wouldn’t say distillation didn’t stick; many step-distilled models are in production. There’s still a quality gap vs non-distilled, but we’re two to three years offset from language models—better distillation is a matter of time. We have a real-time characters model—largest deployment of real-time video models I think—step-distilled, 24 FPS, talking-avatar, step-distilled autoregressive. A big component of world models is going from bidirectional diffusion that generates a whole video at once to causal autoregressive—one or a few frames at a time—then step distillation for real-time. I’ll be very surprised if two years from now we don’t primarily use real-time models—better UX, cheaper to serve, quality gap will close.

Anastasis

扩散有两条蒸馏轴：模型变小（像 LLM），或步数变少——50 步压到 4 步，常有可接受退化。很多场景我们直接对前沿模型做步数蒸馏就能实时。

Nice thing about diffusion: two axes of distillation—smaller model like LLMs, or fewer diffusion steps—fifty steps to four, often comparable outputs. In many use cases we take the frontier model, step-distill it, and make it real-time.

Anastasis

最近的一项研究更新：把通用世界模型当软件的通用界面——接口世界模型。它替换应用前端，直接渲染界面像素，并预测点击等交互后下一步。没有 HTML、CSS、React——全是实时视频模型输出，点击直接当输入；也支持拖拽、滚动。你可以用自然语言描述按钮行为：按这个发生什么、按那个发生什么。还能加音频。哲学是端到端：何必生成生成像素的代码，直接生成像素。适合原型、探索、教育可视化；也能做出传统很难做的镜头转场。和用 Claude 从 Figma 参考图复刻同一交互比，更慢之外，很多交互也很难单靠 LLM 抓住。个性化也更容易——按访问者改 prompt、放大字号做无障碍。早期，实时视频相对渲染 HTML 算力贵很多，但潜力很大。

One research update: using general world models as a universal interface to software—an interface world model. It replaces the front end, renders interface pixels directly, and predicts what happens next from a click or other interaction. No HTML, CSS, React—directly the output of our real-time video model, taking clicks as input; also drags and scrolling. You can describe in the prompt what each element should do—if I press this button, expect this. Add audio too. End-to-end philosophy on front ends: why generate the code that generates the pixels? Just generate the pixels. Powerful for prototyping, exploration, education; camera transitions hard to build otherwise. Prompting Claude from a Figma reference for the same interaction is slower and many interactions are hard to capture with LLMs alone. Personalization may be easier—different prompts per visitor, larger text for accessibility. Early days; real-time video vs HTML is much more compute, but we see a lot of potential.

Anastasis

终局就是全神经操作系统。应用概念来自「每个工具要有独立代码库」；若视频模型边走边生成界面，可吃 LLM 上下文、把传统分属不同 App 的功能合在一起——端到端解决软件。对训练 computer-use 智能体也强：合成数据生成器，或在线 RL 环境，飞出多样界面。世界模型对人、也对智能体训练——机器人同理。

The endgame is a fully neural operating system. The concept of applications comes from needing separate codebases for each tool; with a video model generating the interface as you go, it can take LLM context and combine functionalities that traditionally lived in different apps—solving software end-to-end. Also powerful for training computer-use agents: synthetic data, or a live RL environment with diverse interfaces on the fly. World models for humans and for training agents—same with robotics.

Anastasis

长上下文很关键。角色模型、接口世界模型还较容易维持长会话；开放世界任意动作时，可生成时长掉得更快，误差累积是自回归硬伤——把生成帧再喂回去，小错会滚雪球。LLM 也有过，现在能出很长输出，算「解决了」但仍是挑战。角色模型自回归能到约 30 分钟；更开放的 GWM Worlds 大概几分钟量级。Genie 一类大概最多约一分钟。理想是永远生成不降解——只是时间问题。

Long context is very important. For characters or the interface world model it’s easier to maintain long sessions; in more open-ended worlds with arbitrary actions, duration limits hit faster. Biggest challenge with autoregressive models is error accumulation—feeding generative frames back in, small errors accumulate. LLMs had this; we can generate really long outputs now—solved-ish, still a challenge. With characters we can generate up to ~30 minutes autoregressively; GWM Worlds, more open-ended exploration, is on the order of a few minutes. Genie has like max one minute. Ideally you generate forever without degradation—a matter of time.

Anastasis

去年发了 Gen-4.5 作为最新基座。世界模型路径很大程度是：把双向扩散变成自回归、并接受动作——从「看的视频」变成「可逐步控制的仿真」，可探索反事实。GWM-1 建在 Gen-4.5 上，做完自回归与步数蒸馏。缩放视频时我们「意外」做出对机器人很强的模型：去年中开始有机器人实验室要视频模型做合成数据、要求后训练适配机器人。后来清楚：不只合成数据，还能当仿真器——在线测动作模型，闭环 rollout。关键是真实世界相关：世界模型里的动作序列，在真机上应得到相似结果。我们用常见的 RoboArena 类基准场景与 embodiment，在世界模型里跑同一动作模型，看到与真机表现高度相关——评估策略可以大量在仿真里扩，而不必事事上真机。

Last year we released Gen-4.5, our latest base model. A lot of our world-model approach is taking a bidirectional diffusion model, making it autoregressive, and accepting actions—from a video you watch to a simulation you step through, exploring counterfactuals. GWM-1 was built on Gen-4.5 with autoregressive and step distillation. As we scaled video models we accidentally created a state-of-the-art model for robotics. Mid last year robotics labs asked to use video models for synthetic data and post-train for robotics. Increasingly clear they’d be useful as simulators too—test action models online in closed loop. Biggest thing for a simulator: real-world correlation—same actions in the world model and reality yield similar outcomes. We used RoboArena-style benchmark data, same scenarios and embodiments in our world model, and saw very good correlation with real-world policy performance—so you can scale evaluation in sim instead of physical hardware.

Anastasis

传统仿真对刚体还行（Isaac Sim、MuJoCo），但布料、湿滑表面等复杂操作很难、很耗时去为每个任务建数字孪生。世界模型只要首帧就能 rollout。我们的论点：遥操作、人类操作器、GoPro 第一人称都有价值，但世上最充裕的是第三人称视频。人很多技能是看别人做学会的；视频预训练本质上就是海量第三人称任务镜头。预训练后再用少得多的真机数据适配机器人——第三人称数据量级仍远超第一人称。最充裕的数据源最终会赢。

Traditional sims work well for rigid bodies—Isaac Sim, MuJoCo—but cloth, slippery surfaces, complex manipulation are hard and time-consuming to twin per task. With a world model you provide the first frame and roll out the policy. Companies use teleop, human-operator (“creeper”) data, egocentric GoPro—all great—but the most plentiful video is third-person. Humans learn a lot by watching others; video pretraining is largely third-person footage of tasks. After that you adapt with way fewer hours of robot data. Egocentric is still orders of magnitude less than third-person out there. The most plentiful data source ultimately wins.

Anastasis

GWM Robotics 从 Gen-4.5 同一视频扩散骨干出发，在很少小时的机器人数据上微调——大约数百小时量级；而从零预训练机器人模型可能要十万到百万小时。模型借用预训练里学到的物理、人类动力学与任务先验，泛化到训练未见的环境与任务。目前我们按合作伙伴的具体 embodiment（单臂、双臂、人形）分别后训练；长远期望变体统一——一只世界模型同时仿真操作、导航（游戏世界模型多是导航）和人类行为（角色模型），用同一个实时视频模型「活在世界里」。

GWM Robotics starts from Gen-4.5—same video diffusion backbone—then fine-tunes on a very small number of hours of robotic data, on the order of hundreds of hours, vs pretraining robotics models up to a hundred thousand or millions of hours. The model leverages physics, human dynamics, and tasks from pretraining, and you want generalization beyond trained tasks. Currently we post-train for specific embodiments for partners—single-arm, bimanual, humanoid. Over time variants unify: one world model simulating manipulation, navigation, and human behavior—characters—ideally one real-time video model of being in the world.

Anastasis

世界模型已用于自动驾驶研究，多是评估；我们更聚焦操作，也做过一些 AV 世界模型。有了好世界模型，加动作头就能预测动作、变成策略。首帧是机械臂场景，提示「生成手臂拿起物体」——若视频够准，也应能生成手臂该走的 3D 姿态。现在流行词是 world action models：从视频模型加动作头变成策略。

World models are being applied to self-driving research mainly for evaluation; our focus has been robotic manipulation—we’ve done some AV world-model work too. Once you have a great world model, add an action head and it predicts actions—a policy. Prompt from a starting frame with a robotic arm to generate the arm picking up an object; if the video is accurate enough it should also generate the 3D poses. Popular term now: world action models—start from a video model, add an action head, it becomes a policy.

Anastasis

训视频模型还会再往上缩很多。我喜欢的思想实验叫「清醒梦测试」——世界模型版图灵测试：戴 VR，多数头显有透视模式。总有一天交互式实时视频好到：你在同一房间走动、碰物体，最后有人问你——刚才是透视，还是生成画面？若分不清，说明模型够好了。我们还远。关键是动力学与反事实：让模型生成「进球成功」往往比「射失」好，因为训练分布偏成功；交互模型却要两种结果都同样真实。机器人评估或未来在线 RL 都需要把失败仿真好——视频模型与世界模型的大鸿沟就是反事实生成。

Training video models will scale way further. A thought experiment I like—the lucid dream test, almost a Turing test for world models: VR headset with pass-through. There’ll be a point where interactive real-time video is good enough that you walk around a room interacting with objects, and someone asks whether you were in pass-through or generated footage—and you can’t tell. We’re not close yet. A lot is simulating dynamics and counterfactuals well. Ask for a person scoring a goal vs failing—models do better at scoring because of training bias. Interactive models need equally realistic outcomes for either action. For robotics eval or future online RL you need to simulate failure well. The big gap between video models and world models is counterfactual generation.

Anastasis

harness 在各场景都很重要。我们的视频智能体本质上是擅长调用图像/视频工具的 LLM，帮你从 brief 到分镜到成片，还能看广告表现数据再迭代。生产视频模型都有 prompt completion，会越来越复杂。长远看会统一进 omni 模型：端到端既做自回归文本（推理、规划场景），又接扩散头出像素。

Harnesses are really important. Our video agent is essentially an LLM effective at tool use of image and video models—helps create a project end-to-end: brief, storyboard, video, and analyze ad performance to generate more. Production video models use prompt completion; that’ll get more complex. Eventually unification into omni models—trained end-to-end for autoregressive text prediction and diffusion: reason and plan the scene, then a diffusion head generates pixels.

Anastasis

趋势总是：先在 harness 里做，再内化进模型。思维链从超长系统提示，变成模型自己出推理轨迹。早期视频是单镜头，要编排器拼多镜；现在有多镜头生成——模型要学剪辑直觉、节奏，LLM 当剪辑师还不灵。该由 harness 干的，最终会端到端学进模型。

Generally something is first done by a harness then becomes part of the model—chain-of-thought from detailed system prompts to the model generating its own reasoning trace. Early video models were single-shot; you needed an orchestrator for multi-shot. Now multi-shot generation—the model needs editing instincts, pacing; LLMs aren’t great video editors yet. What you need the harness for eventually gets injected and learned end-to-end.

Anastasis

我们有很大的创意团队深度参与训练全流程：怎么 caption 才能抓住摄影、美学、镜头方向，推理时才能激发出来；也做「什么算可用视频」的评估。创作者与研究员并肩，是 Runway 很特别的一点。在纽约也有好处：媒体、广告、艺术圈交汇，又不完全泡在湾区同一套叙事里；人才池也够。

We have a large creative team actively involved in training every step of the way—how you caption video so cinematography, aesthetics, camera direction are detailed enough to elicit at inference; they evaluate what constitutes a usable video. That mix of creatives and researchers side by side is special to how we’ve operated. Being in New York: intersection of media, advertising, the art scene; also benefit from being outsiders, not the same Bay Area hive mind. Huge talent pool—hasn’t been a problem.

Anastasis

现在大概是史上开放岗位最多的时候：研究、尤其机器人方向的软硬件与研究；也在大幅扩 go-to-market——企业采用视频模型很猛。和 NVIDIA 共创了 Cosmos Coalition，推动世界模型开源：开源权重、物理等基准、基础设施，让刚入门的研究者更容易贡献。视频排行榜上前十、前二十多数是中国模型，西方只有少数——社区需要更广泛投入才有竞争力的模型。蒸馏别人不是最好的长期策略；我们自信不必靠蒸馏也能训出好模型。

This is probably the most open roles we’ve ever had—growing research significantly; robotics across software, hardware, and research; scaling go-to-market with very active enterprise adoption. We co-founded the Cosmos Coalition with NVIDIA to bring world-model research open—open-weight releases, benchmarks measuring physics, infrastructure—so developers and researchers starting out can contribute. On video leaderboards, the majority of the top ten or twenty are Chinese models; only a handful from the US or the West. Important we invest more broadly as a community for competitive models. Distilling from them isn’t the best long-term view—we’re quite capable of training great models without distillation.

Anastasis

创意人其实不太死磕 arena 分——图像视频用眼睛立刻能感到美学、伪影、物理问题，比语言侧复杂数学/代码题好判。实践中就是同一 prompt 多模型比；Runway 上也能跑第三方模型。

Creatives and marketers rely less on arena scores—with image and video you can immediately tell with your eyes what feels good aesthetically, artifacts, physics issues—easier than complex math and coding evals for LMs. In practice people try the same prompt across models; on Runway you can use ours and third-party models too.

Swyx

你站在艺术家/创作者与 AI 的张力上。社区很多人恨 AI；Runway 用户当它是另一支笔。情绪怎么变的？

You’re at the tension between artists and creatives and AI. A lot of people in that community hate AI; people in the Runway community don’t mind—just another brush. How have you seen the sentiment change?

Anastasis

我们的视角：就是另一支笔、另一台相机，技术与艺术长线共演。近几个月有明显转向——戛纳等场合有导演公开支持 AI；我们电影节有 Ron Howard；也有 Martin Scorsese 采用 AI 等故事。模型在去神秘化。一个热观点：早期反弹偏热，部分因为「一文生整片」的叙事——好像甩一句描述就出完整电影。今天的模型吃大量参考、高度可控；自由度一多，人的反应就不一样——像工具，而非帮你写完整部电影的魔法实体。最多可到约五十个参考时，对话质感就更像分镜工具。

Our perspective: just another brush, another camera—latest in a long line of tools; art and technology evolve together. Pretty significant shift over the past few months—directors speaking in favor at Cannes; Ron Howard at our film festival; Martin Scorsese adopting AI models. Models are becoming demystified. Hot take: early heat partly from the text-to-video idea—one text description, full video back, misconception you could generate a feature film. Models today take many references, are very controllable; when a tool affords many degrees of freedom, people respond differently—more like a tool than a magical entity that figures out your entire film. With up to fifty references on some latest models, the conversation feels more like a storyboard tool.

Anastasis

需求阶段：先追质量；质量大幅提升后，加上可控——参考、分镜、多种输入；接下来越来越在意延迟。一提示几乎瞬间出十个结果，探索速度完全不同。Photoshop 那种即时感被生成模型弄丢了——等两分钟才出片；实时模型会把那部分魔法带回来。

We’ve gone through stages: first people looked for quality—that still matters, plus controllability with references, storyboards, inputs. Increasingly people will care about latency. Generate ten outputs almost instantly from one prompt—explore way faster. Photoshop was instant; we lost some of that waiting two minutes for a video. We’ll bring some of that back with real-time models.

Anastasis

9 月 30 日我们在旧金山办 Summit，聚焦 Physical AI 与实时视频生成，有 NVIDIA、Physical Intelligence、DeepMind 等嘉宾，希望面板够技术、有实质分歧。机器人界现在的辩论包括：VLA vs 世界动作模型；最好的数据是继续砸遥操作，还是杠杆大规模视频；世界模型是直接预测像素、JEPA，还是更偏 3D。我非常坚定：直接预测像素、缩放视频生成是正确路径——但仍是健康辩论期。硬件也没「解决」：传感器、执行器、电压、发热，软件叙事抽象不掉。

On September thirtieth we’re doing a summit in SF focused on physical AI and real-time video generation—panelists from NVIDIA, Physical Intelligence, DeepMind, and others; we try to make panels really technical with substantive disagreements. Debates in robotics: VLAs versus world action models; best data—further scaling teleop vs large-scale video; world models—predict pixels directly vs JEPA vs more 3D-based. I feel very strongly that predicting pixels directly and scaling video generation is the right approach, but there’s healthy debate. The physical side isn’t solved either—sensors, actuators, voltage, heat—you have to deal with those too.

Anastasis

也不排斥更多模态：最简是同时生成视频与音频。我写过「极致版世界模型」——纳入宇宙中越来越多模态、在不同观测尺度上训练。跨模态有迁移：很早的例子是把 Stable Diffusion 微调在频谱图上的 Riffusion，变成还不错的音乐生成器。还有 The Well 这类物理/生物数值仿真数据集，跨极大时空尺度；我们把视频模型当 RGB 帧微调上去，比从零训快得多就有合理表现。数据稀缺时，把分散科学数据拢进单一模型、借迁移，可能打开新问题——RGB 是起点，不是终点。Omni 还有大量开放研究，但我觉得可解；眼下先把机器人做好。

I’m not opposed to more modalities—simplest case, generate video and audio together. I’ve written about the maximalist world model: incorporate more modalities from the universe, train across scales of observation. There’s transfer—Riffusion fine-tuned Stable Diffusion on spectrograms into a capable music generator. The Well is a dataset of physics and biology numerical simulations across very different scales; we’ve fine-tuned our video model treating them as RGB frames and get reasonable performance much quicker than from scratch. That helps data scarcity. Bring disparate scientific sources under one model and leverage transfer—RGB is a starting point. Lots of open research for omni models; I expect those to be solvable. We need to solve robotics first.

Swyx

非常感谢你的时间。也恭喜——哪天也许你们会办 AI 视频的「奥斯卡」。

You’ve been very generous with your time. Congrats—someday you’ll be hosting the Oscars of AI video.

Anastasis

AI 视频的奥斯卡，就叫奥斯卡。

The Oscars of AI video will be called the Oscars.

Swyx

好。谢谢。

All right. Thank you.

Anastasis

谢谢。

Thank you.