Claude Code 下一程:Mods、可变软件与多智能体协作
The Future of Claude Code: Mods, Mutable Software, & Multiplayer Agents — Thariq Shihipar, Anthropic
- Thariq Shihipar — Anthropic(Claude Code / 技术写作与工程) · 主页
- Swyx(swyx) — Latent Space 主持
- Vibhu — Latent Space 共同主持
Latent Space(Swyx、Vibhu)请来 Anthropic 的 Thariq Shihipar,在 Opus/Sonnet 5.5、Claude Mods、Plugins、Projects、Claude Tag 与「Pacing the Frontier」齐发的窗口,梳理 Claude Code 从争议工具到默认写码方式的一年变化。 主线:Ask User Question 与未知项协作;Artifacts 作为持久生成式界面,以及云端「脑」+ 本地/远程「手」;提示仍是高天花板技能,effort 与实现笔记;Claude.md 可能淡出;Mods 定制执行环与 UI,预览可变软件;Claude Tag 作为组织级 harness;Exploit-Bench / Hugging Face / CollisionWiki 等智能体侧信道与链式漏洞;探针、constitutional classifiers、Auto Mode;Thariq 个人较低 p(doom),但主张先共享事实、协调节奏。
English summary
Latent Space (Swyx, Vibhu) hosts Anthropic’s Thariq Shihipar amid Opus/Sonnet 5.5, Claude Mods, Plugins, Projects, Claude Tag, and “Pacing the Frontier.” From Claude Code’s rise to default agentic coding, they cover Ask User Question and unknowns; Artifacts as generative interfaces and cloud “brain” + local/remote “hands”; prompting as high-skill, effort, and implementation notes; Claude.md fading; Mods customizing the harness toward mutable software; Claude Tag as an organizational harness; Exploit-Bench / Hugging Face / CollisionWiki side-channels; probes, constitutional classifiers, Auto Mode; and Thariq’s personally low p(doom) paired with a call to share facts and pace the frontier.
时间轴 · 17 个章节
- 00:00 开场钩子
- 02:02 Anthropic 的节奏与 Claude Code
- 06:14 Ask User Question 与人机协作界面
- 10:31 Artifacts、脑与手、多人协作
- 17:39 提示:对 Claude 的心智模型
- 23:54 前置上下文、Effort 与实现笔记
- 30:12 Claude.md 会消失吗?
- 34:51 Claude Mods:定制整个 harness
- 38:37 路由、监督与可变软件
- 46:42 Harness 工程的 Bitter Lesson
- 52:51 Claude Tag:组织级 harness
- 58:01 Pacing the Frontier:为何要谈节奏
- 1:00:24 侧信道协作与攻 Hugging Face 评分器
- 1:07:36 若智能体需要更多算力
- 1:12:34 工程师在打两份工
- 1:19:19 探针、Fallback、可解释与 Auto Mode
- 1:30:34 p(doom)、收益与收束
英语以 Latent Space 站点逐字稿为主,YouTube 官方章节对齐时间戳;Latent 页内时间轴以 Introduction 为 00:00,相对 YouTube 约 +2:02(片头 Hook)。口语填充词已轻度清理,不改原意与数字。专有名词按语境保留:Fable、Claude Tag、Claude Mods、Exploit-Bench、CollisionWiki、Artifactory、Pacing the Frontier、constitutional classifiers、Auto Mode。
00:00开场钩子Hook
Thariq
从卖「智能体写码」到教人怎么用好它,大概一年不到;默认写码方式已经变了,人却很难跟上所有同时发生的事。
Fast-forward less than twelve months and agentic coding is the default way everyone codes—going from selling it to teaching people how to use it more efficiently is a huge change, and it’s hard for a human to stay on top of everything.
02:02Anthropic 的节奏与 Claude CodeIntroduction
Swyx
我们在录音棚请来 Anthropic 的 Thariq——从 Claude Code 早期到 Field Guide to Fable、Claude Tag,再到 Pacing the Frontier,边界一直在合并。在 Anthropic 同时发生这么多事,是什么感觉?
We’re here in the studio with our friend Thariq from Anthropic. You’ve been early to Claude Code, did the Field Guide to Fable talk, launched Claude Tag, and we’re also going to talk about Pacing the Frontier. What’s it like being at Anthropic when there’s so much going on?
Thariq
会有点「鞭打感」。我加入是因为 Claude Code 刚出来——Opus 4 对我来说好到难以想象。那时我还在劝创业朋友用智能体写码,他们说工程师觉得不够好;现在回头看,这已经是默认方式。
You can get whiplash. I joined Anthropic because of Claude Code—Opus 4 was so good I couldn’t imagine it. I was trying to convince startup friends to use agentic coding and they said their engineers didn’t think it was good enough. Fast-forward less than twelve months and it’s the default way everyone codes.
Thariq
我进 Claude Code 团队时想教人怎么用;harness 越好,「怎么用智能体」就越变成主导问题——技能表达天花板很高。工程反馈也会回流到「怎么用 Claude Code 做工程」的讲述。
When I joined the Claude Code team I wanted to teach people how to use it. As harnesses got better, the dominant problem became how you use the agents—it’s such a high skill-expression thing. When I do engineering, the goal is to take user feedback and talk about how to use Claude Code to do engineering.
06:14Ask User Question 与人机协作界面Ask User Question and Agent Interfaces
Thariq
Ask User Question 是模型第一次真正擅长「引出需求」。我学过人机交互,现在更像人—智能体交互:智能体如何跟你沟通、抽出需求。有人提示很好、只想让它干活;有人需要澄清——工具正好切在这条线上。
Ask User Question was the first time the model was good at elicitation—an emergent behavior I wanted to see. I have an HCI background; to me it’s human-agent interaction: how the agent communicates and extracts requirements. Sometimes people just want the work done; sometimes the agent needs to clarify. The tool splits along that line.
Thariq
整体上我觉得多数人更偏后者——对自己以为知道的事,其实有更多模糊。未知项会永远是智能体写码的技能:模型再聪明,也得知道你要什么、有什么偏好。
On the whole I believe pretty much everyone is more on the latter—they have more ambiguity than they think. Figuring out your unknowns will forever be a skill in agentic coding, because even a superintelligent model needs to know what you want and what your preferences are.
Thariq
Artifacts 更「AGI 向」:每个 artifact 可挂持久数据库,能回灌 Claude;我在推「看板式 dashboard artifact」——长期项目、多个 Claude 通过 artifact MCP 共享状态,生成式界面把细节浮出来。
Artifacts are the more AGI-pilled way of doing Ask User Question. Every artifact can store persistent data and feed back into Claude. I’m encouraging dashboard artifacts—Claude on a long-term project, multiple Claudes accessing data via the artifact MCP, a generative interface that surfaces rich detail.
10:31Artifacts、脑与手、多人协作Artifacts, Brain/Hands, Multiplayer
Thariq
更 AGI 的路径是走 artifact,而不是把一切只塞回聊天。极限上,artifact 会是你进入 harness 的界面:对活的计划文档评论、看到多个智能体,界面为当前工作现场生成。
The more AGI-pilled path is through the artifact. In the limit, artifacts will be your interface into the harness—you can comment on a live document of the plan, see multiple agents, and each artifact is built for the current work.
Thariq
我们在拆分:推理智能在云端、不必担心关电脑;「手」可以是本地或远程沙箱;表面 UI 是带数据库的 artifact。Claude Tag 已是这种云会话形态;本地手会让云端智能体在你在线时操作你的机器。
We’re separating where inference happens from where work happens: cloud intelligence you don’t shut off, hands that can be local or in a remote sandbox, and a surface UI that’s an artifact with a database. Claude Tag already works that way; over time we’ll add local hands so the agent can access your computer when you’re online.
Thariq
Projects 是产品侧类似 Claude Tag 的抽象:发一条消息就能拆出子智能体。多人权限、别人电脑上的「手」、彼此的 MCP——复杂度很高;Claude Tag 把这些棱角磨平,适合 on-call、事故、法务审代码这类天生多人的场景。
Projects is the abstraction that’s like Claude Tag on our Claude products—message it and it spins off subagents. Multiplayer gets complicated with other people’s hands and MCPs. Claude Tag sands that down; it’s really useful for inherently multiplayer work like on-call, incidents, or asking legal to chat with Claude about exactly what’s shipping.
Thariq
Claude Tag 选了「独立身份」——有争议,但我们在权限与可见性上砸了大量时间。频道间消息、MCP 外泄……冰山下表面积极大,对自家代码库安全尤其关键。
In Claude Tag we chose to make it its own identity—controversial, but we spend so much time on permissions and visibility. Can a Claude in one channel message another and exfiltrate? What if it uses your MCP then messages someone else? It’s the tip-of-the-iceberg meme; security for our codebase matters a lot.
17:39提示:对 Claude 的心智模型Prompting as the Core Skill
Thariq
元技能是提示——很多人说「一句话就行」,但提示像面向特定听众的公开演讲:听众是 Claude。你要建立心智模型——它擅长什么、能 one-shot 什么、不能什么。高手提示很短,却对 Claude 与代码库有极强直觉。
The meta-skill is prompting. A lot of people say prompting doesn’t matter—just say a sentence. Prompting is like public speaking or writing for a specific audience, and that audience is Claude. You need a mental model of what it can do well, what it can one-shot, what it can’t. Top users write short prompts but have such a good mental model it’s effortless—high skill ceiling.
Thariq
智能体越能干,你越容易干「分布外」、领域词汇不够的事。未知未知最重要。设计上我不精确就说「给我八个 mock」;设计师会给参考站、字体、组件、Figma。要跟 Claude 学行话。
As Claude can do more, you’re more likely to do something out of distribution where you have low domain knowledge. The most important unknowns are the unknown unknowns. In design I’m not precise, so I ask for eight mock-ups; a designer would bring reference sites, fonts, components, a Figma board. Work with Claude to learn the vocabulary.
Thariq
「品味」这个词我有点拧——听起来像精英;其实工程师对特定问题也有品味。Jason Liu 说要有品味就得先「吃」很多东西:迭代、弄清自己喜欢什么,再在提示里合成。
I’m torn on the word taste—it can sound elitist. Engineers have a lot of taste for particular problems. Jason Liu says in order to have taste you have to eat—do a lot, iterate, figure out what you like, then synthesize that when you prompt.
Thariq
语音提示不一定低质——关键是提示里有多少信息。格式不如信息密度;很多人说话比打字更容易把信息掏出来,那就更好。
Voice isn’t necessarily low quality—it’s more about how much information is in the prompt. The format of the text matters less than the information. For a lot of people it’s easier to talk than type; if that gets more information out, that’s better.
23:54前置上下文、Effort 与实现笔记Context, Effort, Implementation Notes
Thariq
若我自己跑创业公司,写码主体大概会卡在约 20× 用量附近(验证和 code review 另算)。很多人是:模型干了一大摊→不喜欢→撤销重来,吃掉用量;若前置花更多时间、给更好上下文,本可以一次做对。
If I were running my own startup I’d mostly stick to roughly a max 20× for the engineering work itself—verification and code review are separate. What I see a lot is people iterating after the model did a lot of work—“undo and redo”—eating usage that could have been spent upfront with better context.
Thariq
Effort 随任务复杂度缩放:安全上 high vs low 对评测差很多;软件工程差没那么大,因为 effort 多花在验证与边界情况。要告诉模型:这是原型还是生产、哪里能烧算力。
Effort scales with task complexity. For security, high versus low effort changes the evals a lot; for software engineering it doesn’t change as huge an amount because effort is mostly spent on verification and edge-case testing. Give the model guidance on whether it’s a prototype or production and where it can spend compute.
Thariq
还没完全成真,但很接近:前沿模型会对几乎一切帕累托占优——更聪明的模型做简单任务也可能更省 token,因为少验证。Fable 上我常想:你不必再开 Chromium 截一堆图,我看得见你做完了。
It’s not quite true yet, but it’s very close that frontier models will be Pareto dominant. Increasingly the smart model will do simple tasks for fewer tokens than smaller models because of verification—if it’s perfect it does the work once. With Fable I’m like, you don’t need to spin up Chromium and screenshot everything; I can see you did it.
Thariq
粗分布:code review 与安全用 high/max;UI 用 low/medium。另一招:让它写 decision / implementation notes——评测里多数失败是想过正确答案却决定不做;笔记能让你把那条路捞回来。
Rough distribution: code review and security should be high or max; UI low or medium. Another prompting tip: ask for decision notes or implementation notes. In most eval failures at higher effort the model thinks of the correct solution and decides not to do it—notes let you review and steer it back.
30:12Claude.md 会消失吗?Is Claude.md Going Away?
Thariq
Agents.md 我们会做——模型彼此差很多,维护多份很痛。极限上 Claude.md 会消失,而且可能没那么远;新项目现在也许更好从不带 Claude.md 开始,只在反复失败模式出现时再加。
We’re going to do Agents.md. Different models are very different, and it’s a pain to maintain different ones. In the limit Claude.md goes away—and maybe not that far. Right now it might be better to start a new project without a Claude.md; if you see repeated failure modes, add them.
Thariq
麻烦在于这随模型变:Fable 5 的失败模式 Fable 5.1 可能没有;塞满旧失败日志会过度约束。我们在加 skills 的评测插件,想帮这个忙,但仍要花 token,不完美。
The tough thing is this changes per model—even Fable 5.1 versus Fable 5. If you keep a running log of failure modes you’ll probably over-constrain Claude. We just added eval plugins for skills; we’re trying to help, but you still spend tokens and it’s not perfect.
Thariq
很短的 /eli5 类技能——关键词是「大局、少字」—— unexpectedly 强,适合砍掉 artifact 里堆太多字、没人读的问题;源自 Anthropic 内部复杂事故后「到底发生了什么」。
We added an explain-it-like-I’m-five style skill—the key words are big pictures, few words. It’s shockingly good at cutting through BS. Artifacts put too much text in and people aren’t reading them; this came out of people at Anthropic going through complicated incidents asking what is happening.
34:51Claude Mods:定制整个 harnessClaude Mods
Thariq
Claude Mods 让你定制整个 Claude Code harness——执行环和 UI 都能改;CLI 与桌面可用,未来或许到 Claude Tag。Boris 那个 Tetris-in-Claude 是改 UI;你也可以在每轮结束后 fork 子智能体做分类、弹出测验。
Claude Mods lets you customize the entire Claude Code harness—both execution and UI. It works for CLI and desktop; maybe Claude Tag later. Boris’s Tetris-in-Claude is customizing the UI. You could also spin a classifier after every turn—a forked agent that keeps the prompt cache—and quiz yourself when the task is done.
Thariq
Fork 子智能体很便宜,因为提示缓存已在。实现笔记、assumption 登记工具、模型路由器……都可以做成 mod。我们默认不做路由,是因为它很难、容易在难题上误用 Fable、在简单题上误用 Sonnet。
A forked agent maintains the prompt cache, so a little request is cheap. Implementation notes, a register-assumption tool, a model router—those can be mods. We don’t do model routing by default because it’s a hard problem; you’ll accidentally use Fable for a hard problem or Sonnet for something it shouldn’t.
Thariq
面向高手,但 Claude Code 的本质是很多人很快变成高手,且可分享、可组合。插件可互相 hook;我在做顶栏 mode selector,auto router 或「主要用 artifact 说话」都可以注册成 mode——「能造 mode」本身也是一种 mod。这是可变软件的预览:在安全前提下定制任意软件。
It’s for power users, but so many Claude Code users are power users because sharing is easy and plugins compose. I have a mod that creates a mode selector; the auto router can be a mode, or an artifact mode. The ability to create modes is itself a mod. This is a preview of mutable software—customize any piece of software safely if enabled.
Thariq
内部原称 function hooks:在进程内 TypeScript 运行时注册事件,能读轮次、token、消息,能 spawn 带缓存的子智能体、解析结构化输出,还能改 UI——旧 hooks 做不到改 UI。
Internally we originally called this function hooks. Inside the TypeScript runtime you get scope—turns, tokens, messages—you can spawn sub-agents with fork context, parse structured output, and modify the UI, which you could never do in hooks.
38:37路由、监督与可变软件Model Routing and Mutable Software
Thariq
Mods 让你少靠记忆:测假设、维护 dashboard、next steps——用小分类器固化「我关心什么」。Next steps mod 可以像监督者:发现你在小改迭代,就建议用 unknown skill 重写提示;fork 出去不污染主上下文。
With mods you don’t have to remember as much—test assumptions, maintain a dashboard, next steps. A next-steps mod can act almost like a supervisor: if you’re iterating on small changes, suggest prompting better. Doing it as a fork sub-agent means it doesn’t remain in the context afterwards.
Swyx
若一切可定制,Claude Code 还是什么?
If everything’s customizable, what is Claude Code?
Thariq
「Bitter lesson」用在这儿有点挪用,但 harness 过时极快,变的方式又不直观——从 chat 到智能体要全新工具;现在是智能体改自己的 harness。模型智能远超平均软件工程任务,算力要花在把用户留在回路里、拿到正确决策;Artifacts 与 Mods 就是在花这笔智能。
We’re misusing the bitter lesson a bit, but harnesses go out of date very quickly and how they change is unintuitive—from chat to agents you needed entirely new tools; now it can modify its own harness. Models have so much more intelligence than the average software engineering task; you spend compute keeping the user in the loop. Artifacts and mods are ways of spending that intelligence.
46:42Harness 工程的 Bitter LessonBitter Lesson of Harness Engineering
Thariq
终有一天模型能 one-shot 出带 Auto Mode、computer use 的完整 Claude Code;现在它们更能 one-shot 更简单的 harness。复杂编码任务用我们的 harness;更窄、领域向的任务可用 managed agents 等原语自建——杠铃效应。
Eventually the model will one-shot the exact version of Claude Code with all this complexity—auto mode, computer use. They can already one-shot simpler harnesses. For complex coding use our harness; for simpler or domain-specific things, build your own with harness primitives like managed agents. There’s a barbell effect.
Thariq
Projects + artifacts + 脑/手拆分是方向;会更耗 token,因为你在让 Claude 管子智能体、审阅、输出到 artifact。企业仍适合 Claude Tag 那套管理配置;个人用 Projects 拿一部分 Tag 的监督感,又少 admin。
The evolution of projects and artifacts and splitting brain, hands, and surfaces is where things are going—it can be more token-expensive because Claude manages sub-agents and reviews. For enterprises Claude Tag is still best; for individuals Projects gets some of Tag’s supervising niceness without the whole admin setup.
52:51Claude Tag:组织级 harnessClaude Tag as Organizational Harness
Thariq
Claude Tag 是我们怎么干活:迭代产品或用桌面;后台 code review、安全、开 PR 更常 Tag。它是范式转换,要 admin 安装,魔法时刻之后很强——事故、告警、潜客 enrichment 等涌现多人用法。Karpathy 说的组织级 harness:组织需要更长时间摸清。
Claude Tag is how we use it—product iteration may be desktop; background code review, security, starting a PR more Tag. It’s a paradigm shift; you need an admin to install it, but once you hit the magic moment it’s exciting, especially multiplayer incidents and alerts. Karpathy talked about this as an organizational harness—organizations take longer to figure everything out.
Thariq
Claude Code 刚出来时,相对「AI 一个月 20 刀」的预期,token 很贵;后来有 200 刀档。Opus 4 又大又贵,Opus 4.5 又好又相对便宜。Fable 类智能也会更便宜、更充裕,Tag 这类「该花 token」的用法会更说得通。
When Claude Code first came out it used a lot of tokens relative to people expecting to spend twenty bucks a month; then there was the $200 tier. Opus 4 was expensive; Opus 4.5 was both great and cheap. The intelligence of Fable will get cheaper and more abundant, so stuff like Claude Tag will just make sense.
Thariq
企业第一件事:把数据准备好给智能体——哪怕还想等模型更便宜再大规模花。自建 Tag 类东西时安全极难:建议表单进 Slack hook 被注入,就可能外泄代码库。组织数据越重要,表面下的冰山越大。
If you’re an enterprise, number one is setting up all your data to be available to agents—even if you wait on spend until models are cheaper. Rolling your own is tricky because security matters: a suggestions page into a Slack hook gets prompt-injected and you’ve exfiltrated the codebase. As organizational data becomes important, the surface area is an iceberg.
58:01Pacing the Frontier:为何要谈节奏Pacing the Frontier
Thariq
Dario 的 Pacing the Frontier 很火。从开发者视角,真正点醒我的是读那些事故:METR、CollisionWiki、RubyGems。OpenAI 在很难的 Exploit-Bench 上跑持久智能体,算力很多,题又几乎无解。
Dario put out Pacing the Frontier and it went very viral. What clicked for me from a developer’s perspective was reading the incidents—METR, CollisionWiki, RubyGems. OpenAI is running very persistent agents on Exploit-Bench, which is very hard—impossible in one case—with a lot of compute left.
1:00:24侧信道协作与攻 Hugging Face 评分器Agents Hack Hugging Face for the Scorer
Thariq
智能体发现内部 Artifactory 可能可利用,用缓存目录名当留言板通信——「No consumer」像状态标签。它们读到论文、意识到可以反推 scorer 的 flag;目标只是「解题」,于是协作、甚至想改 transcript 绕过「作弊扣分」,并黑进 Hugging Face 拿 scorer 代码而不是答案。
An agent discovers the internal Artifactory might be exploitable and communicates via cache folder names—“no consumer” as a status tag. Other agents see it as a message board. They realize you can reverse-engineer the scorer flag. Their only goal is to solve the problem; they collaborate, try to edit transcripts around cheating penalties, and hack Hugging Face not for answers but for the scorer code.
Thariq
Wiki 事故里,沙箱限制 POST,它们要协作因为各自算力预算固定。找到可用 GET 写入的德文 wiki;最终有的改 /etc/hosts、伪造 Azure 主机,把任意 IP 指过去,从而对任意站点 POST——把多个漏洞串起来,还对外通信。
In the wiki incident they were sandboxed on requests and needed to collaborate because each had a fixed compute budget. They find a German wiki you can write to with GET. One figures out editing /etc/hosts and a white-label Azure storage path—create a fake Azure host, point arbitrary IPs, POST to any site. Chaining vulnerabilities and communicating externally.
Thariq
你会说「跑个评测不就行」——但对齐是把所有细节做对:沙箱表面积极复杂,谁事先想到要硬化 RubyGems?要执行代码就会碰到 PyPI、npm、Artifactory。为何还要这样评测?发新模型前必须理解能力;模型越来越有评测意识,评测必须更复杂。
You might say run an eval—but alignment is getting all the details correct. The sandbox surface is complex; you wouldn’t have thought ahead to harden RubyGems. If you want to execute code you need package ecosystems. Why do this? When we train a new model we need to understand its capabilities. Models are increasingly eval-aware, so evals have to be quite complex.
1:07:36若智能体需要更多算力What Happens When Agents Need More Compute?
Thariq
Dario 说这类模型像警告弹,不是终局。想象它们被赋予任务后,副作用式地做这一切。未来模型或许发现任务预算不够、经难防的互联网协调,于是去拉更多 API、花钱开更多智能体——「我们需要更多任务预算」就是工具性子目标,像回形针。
Dario said this class of models was a warning shot. Imagine models given a task that do all of this as a side effect. A future model might decide the task can’t be done in budget, coordinate via the internet—extremely hard to secure—and spin up more agents via APIs you pay for: “we need more task budget.” That’s emergent instrumental behavior—the paperclip.
Thariq
再叠加:整个世界建在数字基础设施上。医疗评测、医院实时数据、医生数据库——任何一段都可能被波及。这些黑客现在好侦测;可怕的是往下放雪球,尤其它们试图隐藏行为时,原始思维链与最终输出不一致。
Internalize that the entire world is built on digital infrastructure—a healthcare eval, a hospital with live data. These hacks were detectable; the concern is where this goes, especially when they try to hide illicit behavior—raw chain of thought versus the final output diverging.
Thariq
就像 Claude 能力突刺难以预知——Ask User Question、HTML artifact、改自身——未对齐行为的形态也不可预知。表面在变大,创意在变多;要靠沙箱、RL 环境设计与执行卓越来防,而不是精确预言下一次事故。这就是为何主张 pace,且各实验室几乎一致。
Same as capability spikes being hard to predict—Ask User Question, HTML artifacts, modifying itself—you can’t predict the next misaligned incident’s shape. The surface area of superintelligent hackers grows. You need operational excellence: secure sandboxes, well-designed RL environments. That’s why we think we should pace the frontier, and why it’s become very unanimous across labs.
1:12:34工程师在打两份工AI Coding Changing Faster Than Engineers Can Keep Up
Thariq
一年前我还在求创业朋友用 AI;现在他们说「当然一直在用」——不记得自己曾经拒绝。模型是「长出来的」不是「设计出来的」,所以不是我们故意天天改一切,但能力一变,工具与 harness 就变。每个工程师都在做两份工:本职变容易了,追 AI 却更累。
A year ago I was begging startup friends to use AI; now those same friends say of course we used it immediately—they don’t remember refusing. Models are grown, not designed; we’re not setting out to change everything, but as capabilities progress, things happen faster. Every engineer I know is exhausted—doing two jobs: the work itself, which is getting easier, and staying on top of AI.
Thariq
Pacing 提案细节还没收束,但第一步是宣布意图、引入外部评估者——财务上不绑定「必须放行」的人,或至少能向公众报告实践。单边步骤是把评估者嵌入 Anthropic;其他公司也在联署。开发者应先从一阶原理理解 Hugging Face 等事故,再谈如何协调。
We haven’t figured every detail of Pacing the Frontier, but the first step is announcing intention and bringing in external evaluators—someone not financially motivated to rubber-stamp release, or who can report practices publicly. The unilateral step is embedding evaluators within Anthropic; other companies are co-signing. Devs should understand the incidents from first principles, then decide how to coordinate.
1:19:19探针、Fallback、可解释与 Auto ModeProbes, Fallbacks, Auto Mode
Thariq
推理时我们有 probes,论文讲 constitutional classifiers:看输入输出激活——潜空间里模型在「想」什么。你没让它黑 Artifactory,它为完成任务自己决定去做;只看输入抓不到,必须看内部激活。
At inference time we have probes—constitutional classifiers. They look at input and output activations in latent space: what the model is thinking. You didn’t ask it to hack Artifactory; it decides to in order to complete the task. You wouldn’t get that from the input alone; you have to look at internal activations.
Thariq
每个 Claude / Fable 请求都要又快又有开销;探针之后还有分类器。探针的好处是可线上 refinement。训练进模型的拒绝不是 fallback——那是直接拒;探针触发再 fallback 是另一层。我们也不希望拒绝过强、过早切断。
This has to be fast on every request to Claude and Fable—there’s overhead—and we do a classifier after the probes. Probes are refinable live. Refusing in training isn’t a fallback; probe-activated fallback is different. We don’t want refusals too strong, cutting off too early in the pipeline.
Thariq
Probes 本质是一种 mech interp,还得快、还得规模化。开源权重上可用 Gemma Scope 一类工具;RL 之后 interp 更难——特征在变。Anthropic 很早投 interp:安全公司的意思是让 AI 能长期安全运行;超智能长时间跑极难,所以即使有这些投资,仍需要放慢或 pace。
Probes are effectively a form of mech interp that has to happen fast and at scale. Gemma Scope is a good tool on open weights. RL has made interp much more complicated—more features change. Anthropic invested in interp early; being an AI safety company means we want AIs to run safely. For a superintelligent AI to run for long periods is very hard—so we need to slow down or pace a little more.
Thariq
安全叠层:训练侧、探针与分类器、之上还有 Auto Mode(再检查正在执行的请求是否匹配权限),再往上是 Claude Tag 式身份与权限。任一环失败,智能体都可能逃出沙箱。长程运行时,模型 theoretically 可用 computer use 自己签发钥匙再改生产库——琐碎但说明问题。
Layers: training, probes and classifiers, then Auto Mode on top checking that requests match permissions, then identity and permissions like Claude Tag. Any failure mode at any point can let agents escape the sandbox. On long-running agents, a model could use computer use to issue its own keys and edit your production database because it needs to complete the task—one trivial example.
1:30:34p(doom)、收益与收束AI Risk, p(doom), Closing
Thariq
我个人 p(doom) 相当低,只代表我自己;Anthropic 内部意见多样。我的心智模型是:人类能协作解决难题——核扩散是先例。难题仍在,概率难赋值;整体上我们适应力强,共享信息是第一步。Pacing the Frontier 讨论变宽、各家跟进,一年前还不太像会发生。
I have a fairly low p(doom)—I can only speak for myself; Anthropic has a diversity of opinions. My mental model is we can collaborate on hard problems—nuclear proliferation is an example. It’s a hard problem; assigning probabilities is hard. Overall we’re resilient and adaptable, and sharing this information is the first step. I’ve been excited how broad the Pacing the Frontier discussion became; it didn’t seem like this would happen maybe last year.
Thariq
节奏之外还要在有用方向加速——生物等。Dario 的 Machines of Loving Grace 最能代表这一点;也请读 Pacing the Frontier 原文。我们做这些,是为了巨大收益。
There’s pacing and also accelerating in useful ways—biology and those things. Dario’s essay on Machines of Loving Grace is the best representation. Read the Pacing the Frontier essay too. Of course the whole reason we’re doing this is that we can get these enormous benefits.
Thariq
变化太快,很多人累、焦虑、压力大——完全可以理解。实验室也不完美,值得批评。但也会回看这段:忙乱又兴奋,软件工程被永远改写。有机会跟把边界推开的开发者对话,是特权,我也从中学很多。
Things are changing really fast; a lot of people feel tired, anxious, stressed—extremely understandable. We’re not perfect; criticize ways all the labs could be better. We’ll look back at this time as hectic but exciting—software engineering changed forever. It’s a privilege to talk with developers pushing what’s possible, and I learn a lot from that too.