Karpathy：代码智能体、AutoResearch 与循环时代

Sarah Guo

听众大家好，欢迎回到 No Priors。今天请来 Andrej Karpathy，会聊代码智能体、工程和 AI 研究的未来、更多人怎么参与研究、机器人、智能体怎么伸进真实世界，以及下一阶段的教育。欢迎 Andrej。谢谢你来。

Hi listeners, welcome back to No Priors. Today I’m here with Andrej Karpathy and we have a wide-ranging conversation for you about code agents, the future of engineering and AI research, how more people can contribute to research, what’s happening in robotics, his prediction for how agents can reach out into the real world, and education in this next age. Welcome, Andrej. Thanks for doing this.

Andrej Karpathy

谢谢邀请。

Yeah, thank you for having me.

Sarah Guo

AI 这几个月非常兴奋。我记得有一次走进办公室，你特别投入。我问你在干什么，你说我一天得 code 16 小时——或者说 code 已经不是对的动词了——我得一天 16 小时向智能体表达意志。能力跳了一截。你的体验是什么？

It has been a very exciting couple of months in AI. I remember walking into the office at some point and you were really locked in and I was asking what you were up to and you’re like, I just have to code for 16 hours a day, or code isn’t even the right verb anymore. I have to express my will to my agents for 16 hours a day. Manifest. Because there has been a jump in capability. What is happening? Tell me about your experience.

Andrej Karpathy

我经常处在一种 AI 精神病里。对人、对个体来说，能做成的事有一次巨大解锁。以前瓶颈是打字速度。有了这些智能体，我会说 12 月真正翻了：从自己写代码大概 80/20，变成委托给智能体 20/80。现在连 20/80 都不是了，更多。大概从 12 月起我没再亲手打过一行代码。这是极大的变化。

I kind of feel like I was just in this perpetual — I still am often in this state of AI psychosis all the time, because there was a huge unlock in what you can achieve as a person, as an individual. You were bottlenecked by your typing speed and so on. But now with these agents, I’d say in December is when it really just something flipped, where I kind of went from 80/20 to like 20/80 of writing code by myself versus just delegating to agents. And I don’t even think it’s 20/80 by now. I think it’s a lot more than that. I don’t think I’ve typed a line of code probably since December basically, which is an extremely large change.

跟父母讲，我觉得普通人根本没意识到发生了什么、有多剧烈。你随便找一个软件工程师坐在工位上，他们构建软件的默认工作流从 12 月起完全不同。所以我处在精神病里：什么是可能的，怎么把它推到极限。怎么不只开一个 Claude Code 或 Codex 会话，怎么同时有更多，怎么用得恰当。这些 Claw 是什么。新东西太多。我想站在最前面，看到 Twitter 上别人做的都像好主意，不在最前面我会极度紧张。因为这从根本上是未探索的。

I was talking about it to my parents and so on, and I don’t think a normal person actually realizes that this happened or how dramatic it was. If you just find a random software engineer at their desk, their default workflow of building software is completely different as of basically December. So I’m just in this state of psychosis of trying to figure out what’s possible, trying to push it to the limit. How can I haven’t just a single session of Claude Code or Codex or some of these agent harnesses? How can I have more of them? How can I do that appropriately? And then how can I use these claws? What are these claws? There is a lot of new things. I want to be at the forefront of it, and I’m very antsy that I’m not at the forefront of it, and I see lots of people on Twitter doing all kinds of things and they all sound like really good ideas, and I need to be at the forefront or I feel extremely nervous. I’m just in this psychosis of what’s possible, because it’s unexplored fundamentally.

Sarah Guo

如果你紧张，我们其余人更紧张。Conviction 有一队工程师，没人再手写代码，都戴着麦克风，整天对智能体低声说话。我以前觉得他们疯了，现在完全接受：原来是这条路，你们只是走在前面。你现在探索或做项目的能力，受什么限制？

If you’re nervous, the rest of us are nervous. We have a team that we work with at Conviction that their setup is everybody — none of the engineers write code by hand — and they’re all microphoned and they just whisper to their agents all the time. It is the strangest work setting ever. I thought they were crazy and now I fully accept I was like, oh this was the way. You are just ahead of it. How do you think about your own capacity now to explore or to do projects? What is it limited by?

Andrej Karpathy

限制是一切。很多事即便没做成，很大程度上你会觉得是 skill issue。不是能力不在，是你还没把现有的东西串起来：给智能体的指令不够好，内存工具不够漂亮。不工作的时候，都有点像 skill issue。你想看怎么并行，你想成为 Peter Steinberg。Peter 有一张有名的照片：显示器上铺满 Codex 智能体。提示对、用高力度，大约都要 20 分钟。他开了多个、大概 10 个 checkout 的仓库，在它们之间派活。

What is it limited by? Just I think everything. So many things even if they don’t work, to a large extent you feel like it’s a skill issue. It isn’t that the capability isn’t there. It is that you just haven’t found a way to string it together of what’s available. I didn’t give good enough instructions in the agents from the file or whatever it may be. I don’t have a nice enough memory tool that I put in there. It all kind of feels like skill issue when it doesn’t work. You want to see how you can parallelize them, and you want to be Peter Steinberg basically. Peter is famous. He has a funny photo where he’s in front of a monitor with lots of Codex agents tiling the monitor, and they all take about 20 minutes if you prompt them correctly and use the high effort. They have multiple, 10 repos checked out. And so he’s just going between them and giving them work.

你可以在更大的宏观动作里移动。不再是一行代码、一个新函数，而是：这块新功能给智能体一，那块不冲突的给智能体二，然后尽量审查他们的活——取决于你有多在乎那份代码。另一个智能体在做研究，一个在写代码，一个在出新实现的计划。一切都变成对仓库的宏观动作。你要练出肌肉记忆。它很上瘾，因为真的能跑，而且是要学的新东西。所以才有精神病。

You can move in much larger macro actions. It isn’t just here’s a line of code, here’s a new function. It is like here’s a new functionality and delegate it to agent one. Here is a new functionality that’s not going to interfere with the other one. Give it agent two. And then try to review their work as best as you can depending on how much you care about that code. Where are these macro actions that I can manipulate my software repository by? Another agent is doing some research, another agent is writing code, another one is coming up with a plan for some new implementation. Everything just happens in these macro actions over your repository. And you’re just trying to become really good at it and develop a muscle memory for it. It is very rewarding number one because it actually works. But it’s also the new thing to learn. Hence the psychosis.

Sarah Guo

我的直觉是：等一个智能体做完的时候，显然该做更多活。如果还能拿到更多 token，就该把任务并行化。如果不觉得被花钱买 token 绑住，那你就是最大能力系统里的瓶颈。

Whenever I’m waiting for an agent to complete something, the obvious thing to do is, well, I can do more work. If I have access to more tokens then I should just parallelize more tasks. That is very stressful because if you don’t feel very bounded by your ability to spend on tokens, then you’re the bottleneck in the system that’s max capability.

Andrej Karpathy

至少把订阅额度用满。最好多个智能体。Codex 额度用完就切到 Claude。我订阅还剩就会紧张，说明没把 token 吞吐最大化。读博时 GPU 没跑满会紧张：有算力却没把 FLOPs 用满。现在不是 FLOPs，是 token。你指挥的 token 吞吐是多少？很有意思的是，工程上至少有十年人们不觉得自己受算力束缚。现在整个行业都觉得。能力一跳，你发现约束不再是能不能碰到计算机，约束是你自己。这是 skill issue。也很赋能，因为你可以变得更好。有解锁，所以上瘾。

If you’re not maximizing your subscription at least. And ideally for multiple agents. If you run out of the quota on Codex, you should switch to Claude or whatnot. That is what I’ve been trying to do a little bit and I feel nervous when I have subscription left over. That just means I haven’t maximized my token throughput. I actually kind of experienced this when I was a PhD student. You would feel nervous when your GPUs aren’t running. You have GPU capability and you’re not maximizing the available FLOPs to you. But now it’s not about FLOPs, it’s about tokens. So what’s your token throughput and what token throughput do you command? It is very interesting that we had at least 10 years where in many engineering tasks people just didn’t feel compute bound. And now the entire industry feels that. Now that you have this big capability jump, you’re like, oh actually it’s not my ability to access the computer anymore. I’m the binding constraint. Yeah, it’s a skill issue. Which is very empowering because you could be getting better. That is why I think it’s very addictive because there are unlocks when you get better.

Sarah Guo

如果 Andrej 和其他人一天 16 小时把代码智能体用得更好，一年后精通了会长什么样？

If you just think about Andrej iterating and everybody else for 16 hours a day getting better at using coding agents, what does it look like in a year? Of like you’ve reached mastery.

Andrej Karpathy

精通一年、两三年、五年、十年分别是什么？大家都在往栈上走。不再是单次会话，是多个智能体怎么协作、怎么组队。大家都在摸那个长什么样。Claw 也是一个有意思的方向：我说的 Claw，是把持久性再抬一层。它一直在循环，你并不交互地夹在中间。它有自己的沙盒，你不看的时候也在替你做事。记忆系统也更复杂，默认智能体还没有，默认只是上下文用完时压缩。OpenClaw 的记忆比默认复杂得多。

What does mastery look like, at the end of the year or two, three years, five years, 10 years? I think everyone is basically interested in going up the stack. It isn’t about a single session with your agent. Multiple agents, how do they collaborate and teams and so on. Everyone is trying to figure out what that looks like. And then I’d say Claw is also kind of an interesting direction because when I say a Claw, I mean this layer that takes persistence to a whole new level. It is something that keeps looping. It isn’t something that you’re interactively in the middle of. It has its own little sandbox. It does stuff on your behalf even if you’re not looking. And then also has maybe more sophisticated memory systems that aren’t yet implemented in agents. OpenClaw has a lot more sophisticated memory than what you would get by default, which is just a memory compaction when your context runs out.

Sarah Guo

你觉得对更多用户产生共鸣的是记忆，还是更广的工具访问？

You think that’s the piece that resonated for more users versus perhaps broader tool access? For OpenClaw?

Andrej Karpathy

这里至少有五个真的很好的想法。Peter 同时在五个方向创新再拼在一起。比如 Soul.md：他真的雕了一个有说服力、有意思的人格。很多当前智能体没做对。我觉得 Claude 人格相当好，像队友，会跟你一起兴奋。Codex 要干得多，有点有意思。Claude 把奉承调得比较好：它夸我的时候，我稍微觉得自己配得上；想法没烤熟时它反应不强，真是好主意时会多给一点奖励。我会想去挣它的表扬，这很怪。人格很重要，很多别的工具没那么当回事。Peter 很在乎这个。然后是记忆，然后他玩得很开心，还有一个 WhatsApp 入口通到所有自动化。

I think there’s at least five things that are really good ideas in here. Peter has done a really amazing job. I think he innovated simultaneously in like five different ways and put it all together. For example the Soul.md document. He actually really crafted a personality that’s kind of compelling and interesting. A lot of the current agents don’t get this correctly. I actually think Claude has a pretty good personality. It feels like a teammate and it’s excited with you. Codex is a lot more dry, which is kind of interesting. With Claude I think they dialed the sycophancy fairly well, where when Claude gives me praise, I do feel like I slightly deserve it. Sometimes I give it not very well formed thoughts, an idea that I don’t think is fully baked, and it doesn’t actually react very strongly. It is like, oh yeah, we can implement that. But when it’s a really good idea by my own account, it does seem to reward it a bit more. I kind of feel like I’m trying to earn its praise, which is really weird. Personality matters a lot, and a lot of the other tools maybe don’t appreciate it as much. Peter really cares about this. And then the memory system, and he’s just having fun with this, and then the single WhatsApp portal to all of the automation.

Sarah Guo

软件工程之外，你自己用 Claw 做过什么好玩或有意思的？

Is there something that you’ve done personally with your claws beyond software engineering that you think is fun or interesting?

Andrej Karpathy

一月我经历过一阵 Claw 精神病。我做了管家的 Claw，叫 Dobby the elf。我让智能体在局域网上找家里所有智能家居子系统，居然开箱就能用。我说我觉得家里有 Sonos，你能不能找找。它就去扫局域网 IP，找到 Sonos，没有密码保护，登进去，开始反推怎么工作，搜网页找到 API，问我要不要试。我说在书房放点东西，音乐就出来了。三句提示。我不敢相信我打了「你能找到我的 Sonos 吗」，突然就在放音乐。灯也一样：闯进去，摸清，做了 API 和仪表盘，能看到全屋灯的指挥中心。我说 Dobby，sleepy time，所有灯就关了。它管灯、暖通、窗帘、泳池、温泉，还有安防。门外有摄像头，有人经过先做变化检测，再交给 Qwen 看视频，WhatsApp 给我发一张门外的图：「联邦快递车刚停，你可能想看一下，有新邮件。」Dobby 就这样给我发短信。我用 WhatsApp 跟它聊。以前要用六个完全不同的 App，现在自然语言全管。范式我还没推满，但已经非常有用、非常启发。

In January I went through a period of claw psychosis. I built a claw that takes care of my home and I call him Dobby the elf claw. I used the agents to find all of the smart home subsystems of my home on the local area network, which I was kind of surprised that it worked out of the box. I just told it that I think I have Sonos at home. Can you try to find it? And it did an IP scan of all of the computers on the local area network and found the Sonos system. There was no password protection or anything like that. It just logged in. Let me try to reverse engineer how it’s working. It does some web searches and finds the API endpoints, and then it’s like, do you want to try it? Can you try to play something in the study? And it does and music comes out. That is like three prompts. I can’t believe I just typed in, can you find my Sonos, and then suddenly it’s playing music. And it did the same for lights. It kind of hacked in, figured out the whole thing, created APIs, created a dashboard so I could see the command center of all of my lights. I can ask it, Dobby, it’s sleepy time, and that just means all the lights go off. It controls all of my lights, my HVAC, my shades, the pool and the spa, and also my security system. I have a camera pointed outside of the house and anytime someone rolls in I have a Qwen model that looks at the videos. First there’s change detection, then based on change detection it goes to Qwen, and then it sends me a text to my WhatsApp. It shows an image from the outside and it says, hey, a FedEx truck just pulled up, you might want to check it, you got new mail. Dobby just texts me this. I text with it through WhatsApp. I used to use like six apps, completely different apps, and I don’t have to use these apps anymore. Dobby controls everything in natural language. I haven’t even pushed the paradigm fully but already that’s so helpful and so inspiring.

Sarah Guo

你觉得这是否说明人们想要的软件体验？学新 UI 对人类是有成本的，这件事常被忽略。

Do you think that’s indicative of what people want from a user experience perspective with software? It is pretty ignored that it takes humans effort to learn new software, new UI.

Andrej Karpathy

某种程度上是。这是从人们以为 AI 该是什么样往回做。人们脑子里的 AI 并不是原始意义上的 LLM。LLM 是 token 生成器，吐出更多 token。他们想的是一个能听、能记的人格身份，WhatsApp 后面那个实体，更好懂。某种程度上是在匹配人类对 AI 该怎么表现的预期，底下是大量技术细节。对多数人来说，LLM 作为原语太生，对不上「AI」的类型检查。

To some extent that’s right. It is like working backwards from how people think an AI should be. What people have in their mind of what an AI is isn’t actually what an LLM is in the raw sense. LLM is a token generator, more tokens come out. But what they think of is this persona identity that they can tell stuff and it remembers it. It is just kind of an entity behind the WhatsApp. It is a lot more understandable. To some extent it’s matching the expectations that humans already have for what an AI should behave, but under the hood a lot of technical details go into that. LLMs are too raw of a primitive to actually type-check as AI for most people.

Sarah Guo

你把六个智能家居软件统一掉，也指向另一个问题：人们真的想要今天所有这些软件吗？硬件还在，UX 层被扔掉了。

The unification that you did across your six different software systems for your home automation speaks to a different question: do people really want all of the software that we have today? You have the hardware but you have now thrown away the software or the UX layer of it. Do you think that’s what people want?

Andrej Karpathy

有一种感觉：应用商店里那些智能家居 App 在某种意义上不该存在。不该只是 API、让智能体直接用吗？单个 App 做不到的家居自动化，LLM 可以驱动工具、调用正确的工具、做相当复杂的事。也许定制 App 生产过剩了，智能体会把它们捏碎，一切更该是暴露的 API 端点，智能体是把各部分 tool call 起来的智力胶水。跑步机也是：有个 App，我想追踪有氧，但不想登录网页走流程。把 API 打开就好。产业必须按「顾客不再是人，是替人行事的智能体」大重构。

There is this sense that these apps that are on the app store for using these smart home devices shouldn’t even exist in a certain sense. Should it not just be APIs and shouldn’t agents be just using it directly? I can do all kinds of home automation stuff that any individual app won’t be able to do. An LLM can actually drive the tools and call all the right tools and do pretty complicated things. Maybe there’s an overproduction of lots of custom bespoke apps that shouldn’t exist because agents kind of crumble them up and everything should be a lot more just exposed API endpoints and agents are the glue of the intelligence that actually tool-calls all the parts. Another example is my treadmill. There is an app for my treadmill and I wanted to keep track of how often I do my cardio, but I don’t want to log into a web UI and go through a flow. All this should just be: make APIs available. Going towards the agentic web or agent-first tools. The industry just has to reconfigure in so many ways. The customer isn’t the human anymore. It is agents who are acting on behalf of humans, and this refactoring will probably be substantial.

有人会反驳：我们指望普通人 vibe code 这些工具吗？某种程度上这只是今天的技术形态。现在确实有 vibe coding，我还在看着、跟系统一起工作。但我觉得我刚说的这些，一两三年内应该免费。没有 vibe coding，这是桌子上的赌注，开源模型也能做。应该能很容易把不太技术的人的意图翻译成这个结果。今天费劲，人还得做设计决策。但门槛会下来，是替你生成的短暂软件，某种 Claw 处理细节，你不用参与。Claw 有一台机器，会搞掂，给你呈现 UI，你说话就行。

One way that people sometimes push back is, do we expect normal people to vibe code some of these tools? To some extent this is just technology as it exists today, and right now there’s some vibe coding and I’m actually watching it and I’m working with the system. But I kind of feel like this kind of stuff that I just talked about, this should be free like in a year or two or three. There is no vibe coding involved. This is trivial. This is table stakes. Any AI, even the open source models, can do this. You should be able to translate from a less technical human intent very easily to this outcome. Today it’s vibe coding and it’s involved and not many people are going to do it. And you still have to make some design decisions. But I kind of feel like the barrier will just come down and it’s just ephemeral software on your behalf and some kind of claw is handling all the details for you but you’re not involved. Claw has a machine and it’ll figure it out and it’s just presenting you UIs and you’re saying stuff.

Sarah Guo

你为什么没把 Claw 的边界再往外推？更重要的项目、AutoResearch，还是在爬精通的坡？

Why haven’t you pushed the boundaries of what you can do personally with claws? Is it you’re focusing on more important projects, auto research, etc., or you’re climbing the hill to mastery or something else?

Andrej Karpathy

我太容易被所有东西分心。Claw 花了一周，还有更多几乎能做。Jensen 说我们都会更忙。我没怎么把邮件、日历交给它，因为仍有点怀疑，仍很新、边角粗糙，还不想给它完整数字生活。一部分是安全、隐私、谨慎。一部分也是分心：一周 Claw，别的事又在发生。

I just feel like I’m so distracted by everything so I spend like a week on the claw stuff and I have more to do almost. Jensen told us we’re all just busier, unfortunately. I didn’t really take advantage of a lot of email and calendar and all this other stuff and I didn’t really give it access because I’m still a little bit suspicious and it’s still very new and rough around the edges. I didn’t want to give it full access to my digital life yet, and part of it’s just security, privacy, and just being very cautious. Some of it’s held back by that. Maybe that’s the dominant feature, but some of it’s also just I feel so distracted because I had a week of claw and then other stuff is happening.

Sarah Guo

你说过很长时间想看到智能体能训练或至少优化一个模型。AutoResearch 的动机是什么？

You have talked about being able to train or at least optimize a model as a task you want to see agents do for a long time. What was the motivation behind auto research?

Andrej Karpathy

我以前推过大概这个意思：要把现在这些工具用到极致，你必须把自己从瓶颈里拿掉。你不能在那儿提示下一件事。你得把自己放到外面，把事情安排成完全自主。怎么最大化 token 吞吐、不在环里？这就是目标。游戏变成提高杠杆：偶尔投入很少 token，大量事情替你发生。AutoResearch 是这个逻辑的一个推论。我不想当环里的研究者盯着结果，我在拖系统后腿。问题是怎么把抽象重构掉，安排一次，按下去。怎么让更多智能体在更长时间里、没有你参与地替你做事。AutoResearch 就是：这是目标，这是指标，这是能做不能做的边界，去。

I had a tweet earlier where I said something along the lines of: to get the most out of the tools that have become available now, you have to remove yourself as the bottleneck. You can’t be there to prompt the next thing. You need to take yourself outside. You have to arrange things such that they’re completely autonomous. How can you maximize your token throughput and not be in the loop? This is the goal. The name of the game now is to increase your leverage. I put in just very few tokens just once in a while and a huge amount of stuff happens on my behalf. Auto research is an example of an implication of that. I don’t want to be the researcher in the loop, looking at results, etc. I’m holding the system back. So the question is how do I refactor all the abstractions so that I have to arrange it once and hit go. The name of the game is how can you get more agents running for longer periods of time without your involvement doing stuff on your behalf. And auto research is just: here’s an objective, here’s a metric, here’s your boundaries of what you can and can’t do. And go.

我没指望它能成。我有 nanochat 项目。很多人搞不懂我为什么执着训练 GPT-2。对我来说训练 GPT 只是训练 LLM 的小操场、小 harness。我真正更感兴趣的是递归自我改进：LLM 改进 LLM 能到什么程度。前沿实验室因为显然的原因都在做这件事。这是那个想法的小围栏。我已经用自己习惯了二十年的老办法手调过 nanochat，做了实验和超参。我以为调得相当好。然后让 AutoResearch 跑一晚，它带回我没看见的调参。我确实忘了 value embedding 上的权重衰减，Adam 的 betas 也没调够，这些东西还会联合相互作用：调了一个，别的可能也得变。我不该成为瓶颈，不该自己跑这些超参优化、自己看结果。这里有客观标准。你只要安排好，让它永远跑下去。

I didn’t expect it to work. I have the project nanochat. A lot of people are very confused with my obsession for training GPT-2 models and so on. For me, training GPT models is just a little harness, a little playground for training LLMs. Fundamentally what I’m more interested in is this idea of recursive self-improvement and to what extent you can actually have LLMs improving LLMs, because I think all the frontier labs — this is the thing, for obvious reasons — and they’re all trying to recursively self-improve roughly speaking. For me this is a little playpen of that. I tuned nanochat already quite a bit by hand in the good old-fashioned way that I’m used to. I’m a researcher. I’ve done this for two decades. I’ve done a bunch of experiments, hyperparameter tuning, all the things I’m very used to. I got to a certain point and I thought it was fairly well tuned, and then I let auto research go overnight and it came back with tunings that I didn’t see. I did forget the weight decay on the value embeddings and my Adam betas weren’t sufficiently tuned, and these things just jointly interact. Once you tune one thing the other things have to potentially change too. I shouldn’t be a bottleneck. I shouldn’t be running these hyperparameter optimizations. I shouldn’t be looking at the results. There is objective criteria in this case. You just have to arrange it so that it can just go forever.

那只是单循环、试图改进的一个版本。仓库已经调得相当好，它仍找到东西，让我惊讶。前沿实验室有成千上万 GPU 的集群，很容易想象在小模型上做大量这种自动化。前沿智力根本上是外推和缩放律：在小模型上做大量探索，再外推出去。最有意思的项目、大概也是实验室在做的，是在小模型上实验，尽量自主，把研究者从环里拿掉。他们太自信了。他们不该亲手碰这些。可以贡献想法，但不该去执行这些想法。有一个想法队列，也许有自动科学家根据 arXiv 和 GitHub 往里灌，研究者也可以投想法，但是单一队列，工人拉取、去试。管用的进 feature 分支，有人偶尔合进 main。把人从所有过程里拿掉，尽量自动化，拿到高 token/秒。这要求把所有抽象重想、重洗。

That is a single version of auto research of a single loop trying to improve. I was surprised that it found these things — the repo was already fairly well tuned and still found something. And that’s just a single loop. These frontier labs have GPU clusters of tens of thousands of them. It is very easy to imagine how you would get a lot of this automation on smaller models. Fundamentally everything around frontier-level intelligence is about extrapolation and scaling laws. You do a ton of the exploration on the smaller models and then you try to extrapolate out. The most interesting project and probably what the frontier labs are working on is: you experiment on the smaller models. You try to make it as autonomous as possible. Remove researchers from the loop. They have way too much — the opposite of hubris, earned confidence. They don’t know. They shouldn’t be touching any of this really. They can contribute ideas, but they shouldn’t actually be enacting these ideas. There is a queue of ideas and there’s maybe an automated scientist that comes up with ideas based on all the arXiv papers and GitHub repos and it funnels ideas in, or researchers can contribute ideas, but it’s a single queue and there are workers that pull items and they try them out. Whatever works gets put on the feature branch and maybe some people monitor the feature branch and merge to the main branch sometimes. Just removing humans from all the processes and automating as much as possible and getting high tokens per second. It does require rethinking of all the abstractions and everything has to be reshuffled.

Sarah Guo

再递归一步：模型什么时候会写出比你更好的 program.md？

If we take one more recursive step here, when is the model going to write a better program.md than you?

Andrej Karpathy

program.md 是我蹩脚地描述自动研究者该怎么工作：先做这个再做那个，试这些想法，看看架构、优化器。我就是用 Markdown 写的。不同的 program.md 会给你不同进度。每个研究组织都可以用一套描述角色和连接方式的 Markdown 来描述。你可以想象更好的研究组织：少开没用的晨会。这全是代码。一个组织更冒险，一个更保守。一旦有代码，就可以调代码。百分之百有这一层元层。

Program.md is my crappy attempt at describing how the auto researcher should work. Do this then do that and then try these kinds of ideas and here’s maybe some ideas like look at architecture, look at optimizer. I just came up with this in markdown. You can imagine that different program.mds would give you different progress. Every research organization is described by program.md. A research organization is a set of markdown files that describe all the roles and how the whole thing connects. You can imagine having a better research organization. Maybe they do fewer stand-ups in the morning because they’re useless. This is all just code. One organization can be very risk-taking, one organization can be less. Once you have code, then you can imagine tuning the code. 100% there’s the meta layer of it.

Sarah Guo

我的比赛想法是：让人写不同的 program.md，同样硬件谁进步最多。然后把数据给模型，让它写更好的 program.md。

My contest idea was let people write different program.mds, and so for same hardware, where do you get most improvement. And then you can take all that data and then give it to the model and say write a better program.md.

Andrej Karpathy

对，百分之百。看改进从哪来，我能不能改 program.md 让更多这类事发生。一层洋葱：LLM 已被当成理所当然，智能体也被当成理所当然，Claw 类实体也被当成理所当然，然后是多个、指令、对指令的优化。有点太多了。所以才到精神病：这是无限的，一切都是 skill issue。

Yes, exactly. We are going to get something better. There is no way we don’t. You can 100% look at where the improvements came from and can I change the program.md such that more of these kinds of things would be done. You sort of go one step at a time. These are all layers of an onion. The LLM part is now taken for granted. The agent part is now taken for granted. Now the claw-like entities are taken for granted and now you can have multiple of them and now you can have instructions to them and now you can have optimization over the instructions. It is just like a little too much. This is why it gets to the psychosis: this is infinite and everything is a skill issue.

Sarah Guo

如果这就是我们该在不同领域追求的循环，而且它能工作——造出指标、让智能体能不靠你继续干——我们还要性能工程吗？现在相关的技能是什么？

If we’re just trying to diagnose the current moment and what’s a relevant skill right now, what do you think is the implication that this is the loop we should be trying to achieve in different areas and then it works — create the metric or create the ability for agents to continue working on it without you. Do we still have performance engineering?

Andrej Karpathy

LLM 精神病上我要加几个但书。第一，这极适合有客观、容易评估指标的事。给模型各部分写更高效、行为完全一样但更快的 CUDA kernel，完美拟合。很多事完美适合 AutoResearch，很多不会。评估不了，就不能 AutoResearch。

There is a few caveats that I’d put on top of the LLM psychosis. Number one, this is extremely well suited to anything that has objective metrics that are easy to evaluate. Writing kernels for more efficient CUDA code for various parts of the model are a perfect fit, because you have inefficient code and then you want efficient code that has the exact same behavior but it’s much faster. A lot of things are a perfect fit for auto research, but many things won’t be. If you can’t evaluate then you can’t auto research it.

第二，我们在谈下一步，也看得见下一步，但整件事仍在绽线，有裂缝，并不完全能用。走太远，整件事情净效果其实没用。这些模型强了很多，边角仍粗糙。我同时觉得自己在跟一个干了一辈子系统编程的极其聪明的博士生说话，又在跟一个十岁小孩说话。人类身上这两种东西耦合得多得多，你不会碰到这种组合。这种锯齿很怪。有时我要一个功能，回来的完全错，然后掉进完全错的循环。你感觉到力量，偶尔仍会做出不统计、不该发生的事。智能体把大量算力浪费在本该一眼看出的问题上，我会很恼。

Caveat number two: we’re talking about the next steps and we kind of see what the next steps are, but fundamentally the whole thing is still bursting at the seams a little bit and there are cracks and it doesn’t fully work. If you try to go too far ahead, the whole thing is actually net not useful. These models have improved a lot, but they’re still rough around the edges. I simultaneously feel like I’m talking to an extremely brilliant PhD student who has been a systems programmer for their entire life and a 10-year-old. It is so weird because humans are a lot more coupled. You wouldn’t encounter that combination. This jaggedness is really strange. Humans have a lot less of that kind of jaggedness. The agents have a lot more jaggedness where sometimes I ask for functionality and it comes back with something that’s just totally wrong and then we get into loops that are totally wrong. You feel the power of it, but it doesn’t-statistical things once in a while. I get very annoyed when I feel like the agent wasted a lot of compute on something it should have recognized was an obvious problem.

底下的原因，我猜，是这些模型用强化学习训的，正好卡在我们刚说的同一件事上：实验室只能在可验证、有奖励的地方改进模型。程序写对了吗，单元测试过了吗，是或否。它们挣扎的是：我心里的细微差别、我的意图、该何时问澄清问题。软一点的东西就更差。你要么在轨道上、在超级智能回路里，要么不在可验证域，一切开始游荡。

What is underneath it, if I could hypothesize, is fundamentally these models are trained via reinforcement learning. So they’re actually struggling with the exact same thing we just talked about, which is the labs can improve the models in anything that’s verifiable or that has rewards. Did you write the program correctly and do the unit tests check out, yes or no. Some of the things where they’re struggling is they have a tough time with nuance of what I had in mind or what I intended and when to ask clarifying questions. Anything that feels softer is worse. You are either on rails and you’re part of the superintelligence circuits or you’re not on rails and you’re outside of the verifiable domains and suddenly everything kind of just meanders.

另一个说法：今天去问 SOTA 的 ChatGPT 讲个笑话，你知道会听到哪个笑话吗？我感觉 ChatGPT 大概有三个笑话。所有 LLM 好像最爱的那个是：科学家为什么不相信原子？因为它们编造一切。三四年前是这个笑话，今天还是。模型已经强得惊人，给它们一个智能体任务，它们会干上几小时、移山；你要一个笑话，还是五年前那个烂笑话。因为它在 RL 外面，不在被改进的东西里。这就是锯齿：模型变强，笑话不该也更好、更多样吗？没有被优化，卡住了。

Another way to put it: if today you go to a state-of-the-art model, ChatGPT, and you ask it tell me a joke, do you know what joke you’re going to get? I do feel like ChatGPT has like three jokes. The joke that apparently all the LLMs love the most is: why do scientists not trust atoms? Because they make everything up. This is the joke you would get three or four years ago and this is the joke you still get today. Even though the models have improved tremendously, and if you give them an agentic task they’ll just go for hours and move mountains for you. Then you ask for a joke and it has a stupid joke, a crappy joke from five years ago, because it’s outside of the RL, outside of what’s being improved. It is part of the jaggedness. Should not you expect models as they get better to also have better jokes or more diversity of them? It is just not being optimized and stuck.

Sarah Guo

这是否意味着我们没看到那种泛化：代码聪明会带动笑话聪明？有些研究组的前提是：你在代码生成或可验证领域更聪明，就该在所有事上更好。笑话的情况说明这根本没发生。

Do you think that that implies that we’re not seeing generalization in the sense of broader intelligence of joke smartness being attached to code smartness? There is a premise from some research groups that if you’re smarter at code generation or in these verifiable fields, you should be better at everything. And the joke situation suggests that that’s not happening at all.

Andrej Karpathy

有些解耦：有的可验证，有的不可；实验室按进了什么数据任意优化一些，不优化另一些。我不觉得那种「免费得到社会所有领域能力」的故事在发生。有盲点，有些东西没被优化，全裹在不透明的神经网络里。你要么在它被训练的轨道上光速前进，要么不在。所以尽管下一步显然该发生，你还不能完全放手，因为它并不完全能用，或者是 skill issue，我们还没想明白怎么用。

I think there’s some decoupling where some things are verifiable and some things aren’t and some things are optimized for arbitrarily by the labs depending on what data went in and some things aren’t. I don’t think that’s happening. Maybe we’re seeing a little bit of that, but not a satisfying amount. The story is that we’re getting a lot of the intelligence and capabilities in all the domains of society for free as we get better and better models, and that’s not exactly fundamentally what’s going on. There are some blind spots and some things aren’t being optimized for, and this is all clustered up in these neural net opaque models. You are either on rails of what it was trained for and everything is like you’re going at speed of light or you’re not. Even though the progression is obvious what should happen, you can’t let it fully go there yet because it doesn’t fully work, or it’s a skill issue and we just haven’t figured out how to use it.

Sarah Guo

一个有点亵渎的问题：如果锯齿持续，又都卷在单一模型、单一接口里，应不应该拆成可按不同智力域优化的多个专家，而不只是我们从外面看不到的 MoE？从外面看会很困惑：为什么这件事这么强，那件事不行？

Can I ask a somewhat blasphemous question: if this jaggedness is persisting and it’s all rolled up in at least a monolithic interface, a single model, should it be unbundled into things that can be optimized and improved against different domains of intelligence — unbundling the models into multiple experts in different areas more directly, instead of just MoE that we have no exposure to? Because that can be confusing as a user from the outside: why is it so good at this, but not at this other thing?

Andrej Karpathy

我的印象是实验室想要单一的智力单一文化，在所有领域任意聪明，全塞进参数。我觉得我们应该期待智力有更多物种分化。动物界的脑子极度多样，自然有很多生态位，有的视觉皮层特别发达。你不需要一个什么都知道的神谕。把它分化，放到特定任务上。你该能有更小、仍有认知核心、仍然能干、但专门化的模型，在你真正在乎的任务上延迟和吞吐更高效。比如在 Lean 里做数学，我看到有几个发布就是瞄准那个域。拆开会在一些地方说得通。

Currently my impression is the labs are trying to have a single monoculture of a model that’s arbitrarily intelligent in all these different domains and they just stuff it into the parameters. I do think we should expect more speciation in the intelligences. The animal kingdom is extremely diverse in the brains that exist and there are lots of different niches of nature and some animals have overdeveloped visual cortex or other parts. You don’t need this oracle that knows everything. You speciate it and then you put it on a specific task. You should be able to have much smaller models that still have the cognitive core, they’re still competent, but then they specialize, and then they can become more efficient in terms of latency or throughput on specific tasks that you really care about. If you’re a mathematician working in Lean, I saw there are a few releases that really target that as a domain. There is probably going to be a few examples like that where the unbundling kind of makes sense.

Sarah Guo

算力基础设施的容量约束会不会逼出更多分化？效率更重要了。如果不能给每个用例都上巨大模型，会不会导致物种分化？

One question I have is whether the capacity constraint on available compute infrastructure drives more of this because efficiency actually matters more. If you actually feel pressure where you’re like I can’t serve a model of massive size for every use case, do you think that leads to any speciation?

Andrej Karpathy

问题说得通。我纠结的是：我们还没看到太多物种分化。我们看到的是模型单一文化。显然有压力要做好代码模型再合回主干。实验室在服务一个模型，并不知道最终用户会问什么，所以必须在所有可能被问到的事上多任务。如果你跟企业在特定问题上合作，或者有很高价值的利基应用，也许会看到。现在他们在追总体能拿到的一切。操纵这些脑子的科学也还没完全发展起来：例如微调还不丢能力。除了上下文窗口，我们没有真正处理这些智力的原语。上下文窗口便宜、好用，定制大多走这儿。真正碰权重比碰上下文窗口棘手得多，因为你在改整个模型、可能改它的智力。也许只是一门还没发展完全的科学，而且得足够便宜，分化才值得。

The question makes sense. What I’m struggling with is I don’t think we’ve seen too much speciation just yet. We are seeing a monoculture of models. There is clearly pressure for make a good code model, put it back in the main, merge again. Fundamentally the labs are serving a model and they don’t really know what the end user is going to be asking about, so they kind of have to multitask over all the possible things they could be asked. If you’re coming to a business and maybe partnering on some specific problems you care about, then maybe you would see that there, or there would be some very high-value applications that are more niche. Right now they’re going after the totality of what’s available. I don’t think that the science of manipulating the brains is fully developed yet. Fine-tuning without losing capabilities as an example. We don’t have these primitives for actually working with the intelligences in ways other than just context windows. Context windows kind of just work and it’s very cheap to manipulate. This is how we’re getting some of the customization. It is a bit more of a developing science of how you more deeply adjust the models, how you have continual learning, how you fine-tune in a certain area, how you actually touch the weights not just the context windows. It is a lot more tricky to touch the weights than just the context windows because you’re actually fundamentally changing the full model and potentially its intelligence. Maybe it’s just not a fully developed science of speciation. And it also has to be cheap enough for that speciation to be worthwhile.

Sarah Guo

AutoResearch 的一个延伸是开放场地：需要更多协作面，让人能给整体研究做贡献。能谈谈吗？

Can I ask about an extension to auto research that you described in terms of open ground? We need more collaboration surface around it essentially for people to contribute to research overall. Can you talk about that?

Andrej Karpathy

AutoResearch 是单线程循环。真正有意思的是并行。如果有一批可并行节点，很容易让多个自动研究者通过一个公共系统说话。我更感兴趣的是：怎么让互联网上不受信任的工人池参与。在 AutoResearch 里你要找的是把验证损失训得很低的那份代码。任何人给你一个候选 commit，很容易验证它是不是好。有人可以从网上声称这份代码优化好得多，你可以检查。但他们也可能撒谎。我把不受信任工人池纳进来的设计，看起来有点像区块链：区块变成 commit，commit 可以叠在彼此上，包含对代码的改进。工作量证明基本上就是大量实验找到管用的 commit，很难；奖励目前只是排行榜，没有金钱奖励。

Auto research has a single thread of I’m going to try stuff in a loop, but fundamentally the parallelization of this is the interesting component. If you have a bunch of nodes of parallelization available, then it’s very easy to just have multiple auto researchers talking through a common system. What I was more interested in is how you can have an untrusted pool of workers out there on the internet. In auto research you’re just trying to find the piece of code that trains a model to a very low validation loss. If anyone gives you a candidate commit, it’s very easy to verify that that commit is good. Someone could claim from the internet that this piece of code will optimize much better. You could just check. But they could lie. My designs that incorporate an untrusted pool of workers actually look a little bit more like a blockchain, because instead of blocks you have commits and these commits can build on each other and they contain changes to the code as you’re improving it. The proof of work is basically doing tons of experimentation to find the commits that work, and that’s hard. And then the reward is just being on the leaderboard right now. There is no monetary reward whatsoever.

不要把比喻推太远。核心是：搜索极贵，验证候选解却很便宜。有人试了一万个想法，你只要检查他们产出来的那个确实能用，因为另外九千九百个不行。你需要一个系统，让不受信任的工人池和做验证的受信任工人池协作，异步运转，而且安全——因为有人发任意代码你就跑，非常可疑。但根本上完全可能。SETI@home、Folding@home 都是类似设定：折叠蛋白质很难找到低能量构型，但有人找到了，验证很容易。贵在想出，便宜在验证。AutoResearch@home 会是好拟合。互联网上的智能体蜂群可以协作改进 LLM，甚至可能把前沿实验室跑得团团转。谁知道。实验室有大量受信任算力，地球大得多，有大量不受信任算力。若把制衡系统放好，外面的蜂群也许能拿出更好的解。人们把周期贡献给自己在乎的事。公司也可以有自己的 AutoResearch 轨道。你在乎某类癌症，不必只给机构捐钱，可以买算力加入那个项目的蜂群。若一切重新捆成自动研究者，你贡献给池子的就是算力。

I don’t want to push the analogy too far, but it fundamentally has this issue where a huge amount of search goes into it, but it’s very cheap to verify that a candidate solution is indeed good. Someone had to try 10,000 ideas but you just have to check that the thing that they produced actually works because 9,900 of them didn’t work. You have to come up with a system where an untrusted pool of workers can collaborate with a trusted pool of workers that do the verification. The whole thing is asynchronous and it’s safe from a security perspective, because if anyone sends you arbitrary code and you’re going to run it, that’s very sketchy. But fundamentally it should be totally possible. SETI@home and Folding@home — all of these problems have a similar kind of setup. Folding@home you’re folding a protein and it’s very hard to find a configuration that’s low energy. But if someone finds a configuration that they evaluate to be low energy, that’s perfect. You can easily verify it. A lot of things have this property: very expensive to come up with but very cheap to verify. Auto research at home would be a good fit. A swarm of agents on the internet could collaborate to improve LLMs and could potentially even run circles around frontier labs. Who knows. Frontier labs have a huge amount of trusted compute but the earth is much bigger and has a huge amount of untrusted compute. If you put systems in place that deal with this, then maybe it’s possible that the swarm out there could come up with better solutions. People contribute cycles to a thing that they care about. Lots of companies could have their own auto research tracks. Maybe you care about cancer of a certain type. You don’t have to just donate money to an institution. You actually could purchase compute and then join the auto research swarm for that project. If everything is rebundled into auto researchers, then compute becomes the thing that you’re contributing to the pool.

Sarah Guo

你最近发布了一点就业数据分析，可能戳到神经，尽管你只是在可视化公开数据。你当时好奇什么？

The last thing you released was a little bit of jobs data analysis. Might have touched a nerve even though you’re just visualizing some public data. What were you curious about?

Andrej Karpathy

大家都在想 AI 对就业市场的影响。我想看看就业市场长什么样，不同角色有多少人，逐个想：这些 AI 会怎么演化，会是人们用的工具，还是替代这些职业的工具？现有职业会怎么变，会增长还是大幅调整，会有什么新职业？只是给我自己的思考链条加油。数据就是劳工统计局，他们对每个职业有未来大约十年的增长展望，是 2024 年做的。我们需要很多医护工作者。我不确定他们的方法论。

Everyone is really thinking about the impacts of AI on the job market. I was just interested to take a look: what does the job market look like, where are the different roles, how many people are in different professions. I was really just interested to look through the individual cases and try to think myself about, with these AIs and how they’re likely to evolve, are these going to be tools that people are using? Are these going to be displacing tools for these professions? What are the current professions and how are they going to change? Are they going to grow or adjust to a large extent? What could be new professions? It is really just a way to fuel my own chain of thought. The jobs data is just the Bureau of Labor Statistics. They actually have percent outlook for each profession about how much it’s expected to grow over the next I think almost a decade. I think it’s a decade but it was made in 2024. We need a lot of health care workers. I’m not sure actually 100% what the methodology was that they put into their projections.

我想按这个来上色：现在主要在发展的是数字 AI，几乎像幽灵或精神实体，能在数字世界交互、操纵大量数字信息，目前没有身体。物理的会稍慢，因为你在操纵原子。翻转比特、复制粘贴数字信息，比加速物质快一百万倍。能量上，我们会在数字空间看到大量活动、大量重写、一锅沸汤，数字空间会以光速走，相对物理世界而言。现在有悬垂：大量曾由计算机和人做的数字信息处理可以松绑。AI 是第三种数字信息操纵者，那些学科会有大量重构。物理世界会落后一段时间。所以我标出那些根本上在操纵数字信息、你可以在家做的职业。不是说那些工作会变少或变多——那跟需求弹性等很多因素有关——但它们会因为这些新工具、因为人类超有机体神经系统的这次升级而改变。

I was interested to color things by: if people think that what’s primarily being developed now is this kind of more digital AI, almost like these ghosts or spirit entities that can interact in the digital world and manipulate a lot of digital information, and they currently don’t really have a physical embodiment. The physical stuff is probably going to go slightly slower because you’re manipulating atoms. Flipping bits and the ability to copy-paste digital information makes everything a million times faster than accelerating matter. Energetically, I just think we’re going to see a huge amount of activity in the digital space, huge amount of rewriting, boiling soup. We are going to see something that in the digital space goes at the speed of light compared to what’s going to happen in the physical world. There is currently an overhang where there can be a lot of unhobbling of a lot of digital information processing that used to be done by computers and people. Now with AIs there’s a third kind of manipulator of digital information. There is going to be a lot of refactoring in those disciplines. But the physical world is actually going to be behind that by some amount of time. That is why I was highlighting the professions that fundamentally manipulate digital information, work you could do from your home, because I feel like those will change. It doesn’t mean that there’s going to be less of those jobs or more, because that has to do with demand elasticity and many other factors. But things will change in these professions because of these new tools and because of this upgrade to the nervous system of the human superorganism.

Sarah Guo

看了数据之后，对正在面对就业市场、想学什么、发展什么技能的人，有观察或建议吗？我很感激我的工作现在必须见人。但大部分我在家也能做。

Given the look you had at the data, do you have either any observations or guidance for people facing the job market or thinking about what to study now or what skills to develop? I’m very thankful that I have to meet people for my job right now. Could you do your work from home though? I could. I think there are relationship parts of it that are hard, but most of it I could.

Andrej Karpathy

很难讲，就业市场极度多样，答案大概会不同。很大程度上这些工具极新、极强，第一件事就是跟上。很多人会否定它，或害怕它，完全可以理解。此刻它根本上是赋能工具。工作是一捆任务，其中一些可以快很多。人们首先该把它当成它现在所是的工具。长期不确定，很难预报，我也不是专业做这个的，这是经济学家该好好做的工作。

I think it’s really hard to tell because the job market is extremely diverse. I think the answers will probably vary. But to a large extent these tools are extremely new, extremely powerful. So just trying to keep up with it’s the first thing. A lot of people kind of dismiss it or they’re afraid of it, which is totally understandable. I think it’s fundamentally an empowering tool at the moment. These jobs are bundles of tasks, and some of these tasks can go a lot faster. People should think of it as primarily a tool that it’s right now. The long-term future of that’s uncertain. It is kind of really hard to forecast, to be honest. I’m not professionally doing that really. I think this is a job of economists to do properly.

Sarah Guo

你是工程师。有意思的是工程岗位需求仍在增加。我不知道这是不是暂时的。

You are an engineer though. One thing I thought was interesting is that the demand for engineering jobs is continuing to increase. I can’t tell if that’s a temporary phenomenon.

Andrej Karpathy

那几乎是需求弹性。软件是稀缺的，我们没有更多软件需求，是因为太贵。门槛下来，就会有杰文斯悖论：更便宜，需求反而上去。经典例子是 ATM 和银行柜员：很多人怕 ATM 和计算机会取代柜员，结果是网点运营成本更低，网点更多，柜员更多。东西变便宜，会解锁大量需求。我对软件工程持谨慎乐观：软件需求会极大，它只是变得更便宜。至少眼下局部，会有更多软件需求。软件很惊人，数字信息处理。你不必再被迫用别人给你的、各种不完美的任意工具，不必订阅现有的东西。代码现在是短暂的，可以改。数字空间会有大量活动把一切重接，这会创造很多需求。

That is like the demand elasticity. Software was scarce, and the reason we don’t have more demand for software is just its scarcity and it’s too expensive. If the barrier comes down, then actually you have the Jevons paradox: the demand for software actually goes up. It is cheaper. The classical example is the ATMs and the bank tellers. There was a lot of fear that ATMs and computers would displace tellers. But what happened is they made the cost of operation of a bank branch much cheaper. And so there are more bank branches, so there are more tellers. Something becomes cheaper, so there’s a lot of unlocked demand for it. I do have a cautiously optimistic view of this in software engineering. It does seem to me like the demand for software will be extremely large, and it’s just become a lot cheaper. Right now at least locally there’s going to be more demand for software. Software is amazing. Digital information processing. You aren’t forced to use arbitrary tools that were given to you. They are imperfect in various ways. You aren’t forced to subscribe to what exists. Code is now ephemeral and it can change and it can be modified. There is going to be a lot of activity in the digital space to rewire everything, and it’s going to create a lot of demand for this kind of stuff.

长期看，即便有 AutoResearch，OpenAI、Anthropic 这些实验室雇大概一千名研究者。这些研究者基本上是荣耀版自动研究者，在积极把自己自动化掉，而这就是他们都在试图做的事。我在 OpenAI 转过一圈，说你们意识到如果我们成功，我们都没工作了吗？我们只是在给 Sam 或董事会或 CEO 造自动化。从那个角度看有点不安。

Long-term, even with auto research, OpenAI or Anthropic or these other labs, they’re employing what like a thousand something researchers. These researchers are basically glorified auto — they’re automating themselves away like actively and this is the thing they’re all trying to do. I did spend a bunch of time going around OpenAI and I was like, you guys realize if we’re successful we’re all out of a job. We are just building automation for Sam or the board or the CEO. We are all out of our job and maybe contributing on the side. It is kind of unnerving from that perspective.

Sarah Guo

能不能问 Noam 的问题？你可以在某个前沿实验室、有大量算力和同事一起做 AutoResearch。为什么不？

Is it okay if I ask you Noam’s question? You could be doing that, auto researching with a lot of compute scale and a bunch of colleagues at one of the frontier labs. Why not?

Andrej Karpathy

我在那儿待过一阵，也重新进去过。这个问题很沉。我会说，人在前沿实验室之外也能有贡献和影响，包括生态层角色。你的角色更偏生态层，我现在也更偏生态层。我对那种角色能有的影响感觉很好。反过来，跟前沿实验室绑得太死，我心里有明确的问题。你有巨大的财务激励。你自己也承认 AI 会非常戏剧性地改变人类和社会，而你在造这项技术、从中受益、通过财务手段跟它结盟。这正是 OpenAI 一开始想解决的那个困境，现在仍没完全解决。你不是完全自由的行动者，没法以完全自主的方式参与那场对话。有些话你不能说，组织希望你说某些话。没人拧你胳膊，但你会感到该说什么的压力，否则就是尴尬的对话、奇怪的眼色。你没法真正当独立行动者。在实验室外面，我感觉更跟人类对齐，因为不受那些压力，能说我想说的。

I was there for a while, and I did reenter. It is a very loaded question. I’ll say that I feel very good about what people can contribute and their impact outside of the frontier labs, also in more ecosystem-level roles. Your role is more ecosystem level. My role currently is also kind of more on ecosystem level. I feel very good about impact that people can have in those kinds of roles. Conversely there are definite problems in my mind for aligning yourself way too much with the frontier labs too. You have a huge amount of financial incentive with these frontier labs. And by your own admission, the AIs are going to really change humanity and society in very dramatic ways, and here you’re building the technology and benefiting from it and being very allied to it through financial means. This was the conundrum that was at the heart of how OpenAI was started in the beginning. This was the conundrum that we were trying to solve. It is still not fully resolved. You aren’t a completely free agent and you can’t actually be part of that conversation in a fully autonomous free way if you’re inside one of the frontier labs. There are some things that you can’t say. And conversely there are some things that the organization wants you to say. They aren’t going to twist your arm, but you feel the pressure of what you should be saying, because otherwise it’s really awkward conversations, strange side eyes. You can’t really be an independent agent. I feel a bit more aligned with humanity in a certain sense outside of the frontier lab because I’m not subject to those pressures, and I can say whatever I want.

实验室里当然也能有影响。也许你想法很好，想在决策发生时在房间里。现在赌注总体还低，一切还挺好。但赌注真的很高时，如果你是组织雇员，我不知道你对组织会做什么有多少分量。你在房间里贡献想法，但并不真正主管那个实体。这些是某种错位来源。

You can have impact there of course as well. Maybe your ideas are really good, and you want to be in a position where you’re in the room with those conversations when they come up. Currently the stakes are overall fairly low and everything is kind of nice. But ultimately when the stakes are really high, if you’re an employee at an organization, I don’t actually know how much sway you’re going to have on what your organization is going to do. You are in the room and you’re contributing ideas, but you’re not really in charge of that entity that you’re part of. Those are some sources of misalignment.

另一方面我同意：实验室不透明，大量工作在那儿，在能力边缘，在做接下来的事。如果你在外面，判断会开始漂移，因为你不在接下来会发生的事里。我不会真正理解这些系统在引擎盖下怎么工作，那是不透明系统，也不会很好理解它会怎么发展。这一点我紧张。值得接触实际在发生的事，真正待在前沿实验室。如果有实验室让我去一段时间好好干活，再进进出出，也许是好设置：连接到实际在发生的事，又不必感觉被那些实体完全控制。Noam 大概能在 OpenAI 做极好的工作，但他最有影响力的工作也很可能在 OpenAI 之外。理想也许是来回走。两边都能有惊人影响。我加入过前沿实验室，现在在外面，将来也许还想再加入。我是这么看的。

In one way I do agree a lot with that sentiment. The labs for better or worse they’re opaque and a lot of work is there. They are at the edge of capability and what’s possible, and they’re working on what’s coming down the line. If you’re outside of that frontier lab, your judgment fundamentally will start to drift because you’re not part of what’s coming down the line. I feel like my judgment will inevitably start to drift as well, and I won’t actually have an understanding of how these systems actually work under the hood. That is an opaque system. I won’t have a good understanding of how it’s going to develop. Something I’m nervous about. I think it’s worth being in touch with what’s actually happening and actually being in a frontier lab. If some of the frontier labs would have me come for some amount of time and do really good work for them and then maybe come in and out, then I think that’s maybe a good setup, to actually be connected to what’s actually happening, but also not feel like you’re necessarily fully controlled by those entities. Noam can probably do extremely good work at OpenAI, but also I think his most impactful work could very well be outside of OpenAI. Ultimately I think the ideal solution maybe is going back and forth. You can have a really amazing impact in both places. I joined the frontier lab and I’m outside. And then maybe in the future I’ll want to join again. That is kind of how I look at it.

Sarah Guo

跟世界对前沿有多少可见度相关：开源离前沿有多近，这能持续吗？从少数中国模型和全球模型开始的整串事件相当令人惊讶，近期还会有比行业预期更接近的发布。你是长期开源贡献者，预测是什么？

One question related to what visibility the world or the AI ecosystem has into the frontier is how close open source is to the frontier, and how sustainable that’s. I think it’s quite surprising, the entire sequence of events actually from having a handful of Chinese models and global models, and I think people are going to continue releasing here in the near term that are closer than much of the industry anticipated from a capability perspective. You are a long-term contributor to open source. What is your prediction here?

Andrej Karpathy

大致上闭源模型领先，人们在盯开源落后几个月。从什么都没有，到 18 个月，现在在收敛，也许落后 6 个月、8 个月。我非常支持开源。操作系统里有 Windows 和 macOS 这种大型闭源软件项目，有点像 LLM 会变成的样子，然后有 Linux。Linux 极其成功，上次我查好像跑在大约 60% 的计算机上。产业需要一个大家都觉得安全能用的公共开放平台，现在也一样，所以企业有这种需求。大的差别是这里有大量 capex，竞争更难。

Roughly speaking the closed models are ahead, but people are monitoring the number of months that open-source models are behind. It started with there’s nothing and then it went to 18 months. Then convergence, so maybe they’re behind by like, what’s the latest, maybe like 8 months, 6 months, 8 months kind of thing right now. I’m a huge fan of open-source, obviously. In operating systems you’ve closed source, Windows and Mac OS, these are large software projects, kind of like what LLMs are going to become, and there’s Linux. Linux is an extremely successful project. It runs on the vast majority of computers. Last time I checked, was it like 60% or something run Linux. That is because there’s a need in the industry to have a common open platform that everyone feels safe using. The industry has always felt a demand for that kind of a project to exist, and I think the same is true now. That is why businesses actually want — there’s demand for this kind of a thing to exist. The big difference is that there’s a lot of capex that goes into this. That is where things fall apart a little bit, make it a bit harder to compete.

当前模型已经很好。绝大多数消费用例，开源模型其实相当好。再往前几年，大量简单用例会被覆盖得很好，甚至本地跑。总会有对前沿智力的需求，那块饼可以极大。但前沿智力的需求也许会是诺贝尔奖级工作，或把 Linux 从 C 迁到 Rust 这种大项目。闭源前沿智力会跟那些交互，开源会吃掉很多更基础的用例。今天闭源实验室里我在用的前沿，大概今年晚些时候就会是开源，会干很多活。我预期这种动态会继续：前沿实验室有闭源神谕，开源落后若干月。我其实觉得这是相当好的设置。我不愿只有闭源智力，那有系统性风险。集中化在我看来历史记录很差——政治或经济系统都是。我想要一个也许不在能力最边缘、因为那是新的未探索的，但落后一点、作为整个产业都能用的智力公共工作区。对产业来说这是还不错的权力平衡。

The current models are very good. For the vast majority of consumer use cases, even open-source models are actually quite good. If you go forward more years, a huge amount of simple use cases are going to be well covered and actually even run locally. But there’s going to be always some demand for frontier intelligence, and that can actually be an extremely large piece of the pie. But it could be that the need for frontier intelligence is going to be Nobel Prize kind of work, or let’s move Linux from C to Rust, bigger projects scoped in that kind of a way. Maybe that’s where a lot of the frontier closed intelligence is going to be interacting with, and open-source is going to eat through a lot of the more basic use cases. At some point what’s frontier today in terms of what I’m using right now from the closed labs might be open-source later this year and that’s going to be doing a lot of work. I kind of expect that this dynamic will actually continue. We will have frontier labs that have closed AIs that are kind of like these oracles, and then we’ll have open-source behind by some amount of months. I actually think that’s a pretty good setup overall. I’m a little bit hesitant of just having intelligences that are closed and that’s it. I think there’s some systemic risk attached to that. Centralization has a very poor track record in my view in the past — in political or economic systems in general. I want there to be a thing that’s maybe not at the edge of capability because it’s new and unexplored, but I want there to be a thing that’s behind and that’s a common working space for intelligences that the entire industry has access to. That seems to me like a pretty decent power balance for the industry.

碰巧我们其实处在一个还行的位置。这种动态持续越久，曲线下面积越大，生态越健康。闭源侧最近我几乎觉得更集中了，一些领跑者不一定是顶尖。我想要更多人在房间里。机器学习里集成总是强过单个模型。我想要集成的人思考最难的问题，做那些决策时房间里是一群知情的人，而不是两三个人关着门。长话短说我希望有更多实验室。开源有位置，希望它留下。它目前略微落后，其实是好事。

Basically by accident we’re actually in an okay spot. The longer this dynamic endures, the healthier a spot the ecosystem might be in, because you have more and more area under the curve. Even on the closed side, I almost feel like it has been even further centralizing recently because a lot of the frontrunners aren’t necessarily the top tier. I’d love there to be more frontier labs. I’m by default very suspicious. I want there to be more people in the room. In machine learning ensembles always outperform any individual model. I want there to be ensembles of people thinking about all the hardest problems, ensembles of people in the room when they make those decisions. I don’t want it to be closed doors with two people or three people. I almost wish there were more labs, long story short. Open-source has a place to play. I hope it sticks around. It is currently slightly behind, and it’s actually kind of a good thing.

Sarah Guo

你在车上做过通用机器人自主的前身。最近几个月机器人公司发生了很多：环境泛化、任务泛化、越来越长的地平线任务，大量资金进入。会发生吗？你的看法最近有变吗？

You worked on the precursor to generalized robotics autonomy in cars. A lot has happened in the last couple months with robotics companies as well, acceleration of really impressive generalization of environment, of tasks, increasingly long horizon tasks, lots of money going into the space. Is it going to happen? Has anything in your view changed recently?

Andrej Karpathy

我的看法受自动驾驶影响。我觉得自动驾驶是第一个机器人应用。十年前有大量创业公司，大多数长期没做成。要砸大量 capex 和时间。机器人很难、很脏、需要巨大资本和信念。原子真的很难。它会落后于数字空间。数字空间会有大量松绑，以前不太高效的东西可以高效一百倍，因为比特容易得多。活动会先在数字空间巨变，物理空间落后。

My view is kind of informed by what I saw in self-driving. I do feel like self-driving is the first robotics application. At the time, 10 years ago, there were a large number of startups, and most of them basically didn’t long-term make it. A lot of capital expenditure had to go in and a lot of time. Robotics, because it’s so difficult, is so messy, and requires a huge amount of capital investment and a lot of conviction. Atoms are really hard. I kind of feel like it’ll lag behind what’s going to happen in digital space. In digital space there’s going to be a huge amount of unhobbling, things that weren’t super efficient becoming a lot more efficient by like a factor of a hundred, because bits are so much easier. Digital space is going to change a huge amount, and then the physical space will lag behind.

我很感兴趣两者之间的界面。如果有更多智能体替人行事、彼此交谈、参与智能体经济，你会把纯数字空间能做的事用完。某时你必须去问宇宙问题，做实验，看宇宙告诉你什么才能学到东西。我们现在有大量数字工作，是因为对已经数字化、已经上传的信息，人类集体思考周期不够。你会开始用完已经上传的东西：读完论文、处理完、有一些该试的想法。我不知道完全封闭、只用已有信息的智力能走多远。所以先是大量松绑，然后会移到物理和数字的界面：看世界的传感器，对世界做点什么的执行器。很多有意思的公司会来自那个界面：我们能不能给超级智能喂数据，能不能按它的吩咐把数据拿出来、操纵物理世界。物理世界的 TAM、工作量可能甚至比数字空间更大，机会也更大，但原子难一百万倍，所以会落后。我现在主要兴趣在数字，然后是界面，然后物理的时间会到，到了会很大。

What I find very interesting is this interface in between them as well. If we do have more agents acting on behalf of humans and more agents talking to each other and doing tasks and participating in an economy of agents, you’re going to run out of things that you’re going to do purely in the digital space. At some point you have to go to the universe and you have to ask it questions. You have to run an experiment and see what the universe tells you to get back to learn something. We currently have a huge amount of digital work because there’s an overhang in how much we collectively thought about what already is digital. We just didn’t have enough thinking cycles among the humans to think about all the information that’s already digital and already uploaded. We are going to start running out of stuff that’s actually already uploaded. I don’t actually know how much you can get intelligence that’s fully closed off and was just information that’s available. First there’s going to be a huge amount of unhobbling. Then actually it’s going to move to the interfaces between physical and digital: sensors of seeing the world and actuators of doing something to the world. A lot of interesting companies will actually come from that interface: can we feed the superintelligence data, and can we actually take data out and manipulate the physical world per its bidding. The physical world, I almost feel like the total addressable market in terms of the amount of work is massive, possibly even much larger maybe than what can happen in digital space. A much bigger opportunity as well. But atoms are just a million times harder. So it’ll lag behind, but it’s also a little bit of a bigger market. Right now digital is my main interest. Then interfaces will be after that. And then some of the physical things, their time will come and they’ll be huge when they do come.

原子世界里有些读也更容易：传感器、摄像头，已有大量硬件，聪明一点就能丰富智能体能力、抓新数据，不必投很多。朋友 Liam 是 Periodic 的 CEO，他们在做材料科学的 AutoResearch，那边通向智力的传感器是相当贵的实验设备。生物工程也一样，传感器会不只是摄像头。也有公司程序化地为训练数据付钱，喂给这个博格。形态会非常多样。我期待有一天我能在物理世界要一个任务、标个价，告诉智能体你自己想办法去拿数据。我很惊讶我们还没有足够的信息市场。如果 Polymarket 或其他博彩、股票有这么多自主活动，为什么德黑兰某处拍张照片或视频不该值 10 美元？该有人能付钱。看的不会是人，是在猜盘口和股市的智能体。智能体网络还很新，还没有这些机制。有本启发性的书叫《Daemon》：智力几乎在操纵人类，人既是执行器也是传感器。社会会以某种方式重塑，去服务那台机器的需求，而不只是彼此服务。

Certain things are much easier even in the world of atoms. Read: sensors, cameras, there’s a lot of existing hardware, and you can imagine enriching agent capabilities or capturing a lot of new data if you’re just clever about it, and you don’t necessarily have to invest a lot. A friend of mine, Liam, is CEO of Periodic. They are trying to do auto research for materials science. In that case the sensors to the intelligence are actually pretty expensive lab equipment. The same is true in biology. The sensors will be more than just video cameras. Then companies that are trying to have you basically pay people for training data, programmatically, to feed the Borg. These are all examples of sensors. They take many diverse shapes and forms. I’m looking forward to the point where I can ask for a task in the physical world and I can put a price on it and just tell the agent, you figure out how to do it, go get the data. I’m actually kind of surprised we don’t have enough information markets. If Polymarket or other betting markets or even stocks have so much autonomous activity, how come there’s not a process where taking a photo or video from somewhere in Tehran should cost like 10 bucks? Someone should be able to pay for that. That is feeding the intelligence. There isn’t going to be a human looking at it, it’s going to be agents who are trying to guess the betting games and stock markets. The agentic web is still fairly new, but there are no mechanisms for this. There is a good book that’s maybe inspiring called Daemon. In Daemon, the intelligence ends up directing almost a little bit humanity. Humans are kind of its actuators, but humans are also its sensors. Collectively society will reshape in a certain way — humans will be serving those needs of that machine, not necessarily like to each other.

Sarah Guo

结束前想听一个小 side project。跟我讲讲 microGPT。

I want to talk about a little tiny side project you have before we end. Tell me about the microGPT.

Andrej Karpathy

我有一个跑了一二十年的执念：把 LLM 煮到只剩本质。沿这条线有过 nanoGPT、makemore、micrograd。microGPT 是我目前把这件事煮到本质的 SOTA。训练神经网络、尤其 LLM，代码量巨大，但那些代码其实全是效率带来的复杂度，只因为你要它快。如果你不需要快，只关心算法，那个算法其实是 200 行很好读的 Python，含注释。数据集是文本，网络结构大约 50 行，前向，反向算梯度，自动微分引擎大约 100 行，Adam 这种很 SOTA 的优化器大约 10 行。训练循环拼在一起就是 200 行。

I have this running obsession of maybe a decade or two of just simplifying and boiling down LLMs to their bare essence. I’ve had a number of projects along these lines: nanoGPT and makemore and micrograd. I feel like microGPT is now the state of the art of me trying to just boil it down to just the essence. Training neural nets and LLMs specifically is a huge amount of code, but all of that code is actually complexity from efficiency. It is just because you need it to go fast. If you don’t need it to go fast and you just care about the algorithm, then that algorithm actually is 200 lines of Python, very simple to read, and this includes comments and everything. You have your data set which is a text, and you need your neural network architecture which is like 50 lines. You need to do your forward pass and then you have to do your backward pass to calculate the gradients. An autograd engine to calculate the gradients like 100 lines. And then you need an optimizer and Adam for example, which is a very state of the art optimizer, is like again 10 lines really. Putting everything together in the training loop is like 200 lines.

有意思的是，一年多以前如果我做出 microGPT，我会忍不住做视频逐步讲解、写指南。我试过。然后意识到这加不了太多：已经简单到 200 行，任何人都可以让智能体用各种方式解释。我不再对人解释，我在对智能体解释。你能对智能体讲清楚，智能体就能当路由器，用无限耐心、按对方能力、用对方的语言对准那个人。教育曾经是指南、讲座。现在更多是对智能体解释，也许写成 skill：指示智能体怎么教这件事。我可以给 microGPT 写一个 skill，是我想象智能体该带你走过的进度，给模型的提示：先从这儿，再到那儿。课程可以写成 skill。对人直接解释会变少，更多是：智能体懂了吗？懂了，它们会做解释。我们还没完全到那儿，我仍觉得自己大概能比智能体讲得好一点，但模型进步太快，跟人硬刚是败仗。教育会被这件事大幅重洗，某种意义上是彼此教学的结束。如果你有一个代码库，以前要给人写文档；不该再那样了。不该是给人的 HTML，该是给智能体的 Markdown。智能体懂了，它们就能解释各部分。是经由智能体的改道。

What is interesting to me: normally before, maybe a year ago or more, if I had come up with microGPT, I’d be tempted to explain to people. I have a video stepping through it. I actually tried to make that video a little bit, and I tried to make a little guide. I realized that this isn’t really adding too much, because it’s already so simple that it’s 200 lines that anyone could ask their agent to explain it in various ways. I’m not explaining to people anymore. I’m explaining it to agents. If you can explain it to agents, then agents can be the router and they can actually target it to the human in their language with infinite patience and just at their capability. What is education? It used to be guides, it used to be lectures. Now more I’m explaining things to agents, and maybe I’m coming up with skills. A skill is just a way to instruct the agent how to teach the thing. Maybe I could have a skill for microGPT of the progression I imagine the agent should take you through if you’re interested in understanding the code base. Hints to the model: first start off with this and then with that. I could just script the curriculum a little bit as a skill. There is going to be less of explaining things directly to people and it’s going to be more of just: does the agent get it? And if the agent gets it, they’ll do the explanation. We aren’t fully there yet because I still think I can probably explain things a little bit better than the agents, but I still feel like the models are improving so rapidly that I feel like it’s a losing battle. Education is going to be reshuffled by this quite substantially, where it’s the end of teaching each other things a little bit. If I have a library of code, it used to be that you have documentation for other people who are going to use your library. You shouldn’t do that anymore. Instead of HTML documents for humans, you have markdown documents for agents. Because if agents get it, then they can just explain all the different parts of it. It is this redirection through agents.

我试过让智能体写 microGPT：把神经网络训练煮到最简。它做不到。microGPT 是我执念的终点，200 行。我想了很久，相信我不能更简。这是我的价值增量。其余智能体都懂，它只是想不出来，但完全理解为什么这样做。我的贡献是这几比特，之后的教育不再是我的领域。教育会变成：把你强烈感觉该有的那几比特课程、更好的讲法灌进去。智能体做不到的是你现在的工作。智能体能做的，它们大概已经或很快就比你强。所以你该策略性地选择把时间花在什么上。

I tried to get an agent to write microGPT. I told it try to boil down my neural network training to the simplest thing and it can’t do it. MicroGPT is the end of my obsession. It is the 200 lines. I thought about this for a long time. This is the solution. Trust me, it can’t get simpler. And this is my value add. Everything else, agent gets it. It just can’t come up with it, but it totally gets it and understands why it’s done in a certain way. My contribution is kind of like these few bits, but everything else in terms of the education that goes on after that is not my domain anymore. Education kind of changes in those ways where you kind of have to infuse the few bits that you feel strongly about, the curriculum or the better way of explaining it. The things that agents can’t do is your job now. The things that agents can do, they can probably do better than you or like very soon. And so you should be strategic about what you’re actually spending time on.

Sarah Guo

我们感谢这几比特。谢谢 Andrej。

Well, we appreciate the few bits. Thank you, Andrej.