OpenAI 金融负责人 Joseph Kim × Latent Space：ChatGPT 金融版、品味训练与金融评测

Joseph Kim

我真心觉得，会有人在一个 ChatGPT 插件之上做出一门 10 亿美元的生意。我们做的东西里有 95% 是创业公司、其他公司完全也能做的。在 OpenAI，我们曾在预训练上遇到问题，因此很难扩大模型规模。到了 2026 年，这些问题里很多我们已经解决了。

I really do think that there’s going to be a billion-dollar business built on top of a ChatGPT plugin. But 95% of what we built was something that absolutely a startup could do, another company could do. At OpenAI, we ran into problems with pre-training and therefore had difficulty scaling the model. In 2026, we’ve been able to solve many of these issues.

Latent Space 主持人

今天请到的是 OpenAI ChatGPT 金融业务的负责人之一 Joseph Kim，还有我们的联合主持、Latent Space 编辑主管 Richard 也在演播室。这是一期特别节目，我们在纽约的 AIE Finance。Joseph 刚做完一场很棒的主题演讲，我们不想重复里面的大部分内容——大家直接去看就好。不过请你用 30 秒讲讲演讲内容，我们留些线索，大家再去看主题演讲。

Today we have Joseph Kim, one of the heads of ChatGPT Finance from OpenAI, and our co-host and head of editorial from Latent Space, Richard, in the studio. It’s a special episode. We’re here in New York at AIE Finance. Joseph just gave a great keynote. I think we don’t want to repeat most of the information. Honestly, go watch it, but give like a 30-second of what you talked about and then we’ll leave the breadcrumbs. You guys can go watch the keynote.

Joseph Kim

好的，我可以快速用 30 秒介绍一下。第一，我们刚发布了一款新产品：ChatGPT for Financial Services。这是一款面向企业的产品，想把 ChatGPT 的能力——也就是大家在消费版里喜欢用的工具和功能——与企业层结合起来，包括各家机构自己的模板，以及 S&P 等数据源。非常令人兴奋。我们一直和设计伙伴 Morgan Stanley、Evercore 合作，反响好得惊人。这是演讲的前半部分。

Sure, yeah. I can give a quick 30-second intro. One, we just released a new product, ChatGPT for Financial Services. This is an enterprise-focused product that’s really trying to bring the mix of power of ChatGPT, the tools and things that you really like to use in the consumer product, but also add in the enterprise layer. That’s firm-specific templates, sources like … S&P, etc. Really exciting. We’ve been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that’s been incredible. That was kind of the first half of the talk.

后半部分更前瞻：认真看我们在模型进步上观察到的趋势线，把它们延伸到未来，想一想——到 2027 年年中、2027 年年底，这些模型能做到什么？这对在模型之上做产品的人意味着什么？你怎样提前为几个月后就会到来的智能做规划？

The second half of the talk was much more forward-looking: thinking deeply about some of the trend lines that we’re seeing in terms of model improvements, drawing those trend lines out to the future and trying to think, okay, what are these models going to be able to do by mid-‘27, end of ‘27? What does that mean for folks that are building products on top of that model? How can you start to plan ahead for the intelligence that’s coming in just a matter of months?

Latent Space 主持人

现在挺有意思的，因为你其实来自研究那一侧，现在却负责大量产品和工程。说说做一款金融专用产品需要什么。你在演讲里提到，ChatGPT Work 推出、Codex 开始面向大众时，用户出现了一波增长，人们开始用它做金融，现在你们发布了第一款金融产品。

I think it’s a fun time because you actually come from the research side of things. You now lead a lot of product and engineering, but tell us what it takes to make a finance-specific product. You mentioned in the keynote that there was an uptick in users when you launched ChatGPT Work. Codex started to become available to the masses. People started using it for finance, and now you’re shipping your first finance product.

Latent Space 主持人

金融推理数据是什么样的？金融专用模型里都放了些什么？

What is financial reasoning data like? What goes into finance-specific models?

Joseph Kim

好问题。我想说的一点是：我们为金融推理与分析做的所有训练数据，都放进上游我们所说的「big run」，也就是最终成为 GPT-6 这类模型的主训练。所以即使我们往模型里加了大量金融数据，它们也进了主模型，想在上面做产品的人都能用到。这对我们非常重要。

Yeah, it’s a great question. One thing I will say is that all of the training data that we do for financial reasoning and analysis, we do upstream into what we call the big run, which is the main run that becomes something like GPT-6 … Even though we’re adding a bunch of financial data to the model, it’s going into the main model, and therefore it’s available to folks that want to build on top of it. That is something that’s very important to us.

在 OpenAI，我们真的希望这些智能不只给我们自己内部、第一方的产品用，也要开放给想在上面做创业公司、做业务的人，以及想把它直接集成进公司里的金融机构。我们会让所有这些智能都能被用到。

At OpenAI, we really do want that intelligence to be accessible not only to our products that we’re building in-house in first-party ways, but also for folks that want to build startups and businesses on top of it, as well as financial firms that want to integrate it directly into their companies. We will make all that intelligence accessible there.

你提到了产品与研究，这正是 OpenAI 的核心。我们内部称自己是一家「研究与部署公司」，这个顺序是刻意的：研究在先，部署在后。我们最终想改变人们做知识工作的方式，所以按这个顺序来：第一，思考模型能力，它在往哪里走，能解锁哪些新事物；第二，深入思考应该做哪些产品功能和产品界面，让更多人用上这些模型智能。

I think you touched a little bit on product and research, and that’s really core to what we do at OpenAI. Internally, we call ourselves a research and deployment company. We structure that in order on purpose: research comes first, and deployment comes second. As we think about trying to change the way that people do knowledge work, we work in that order. We think about the model capabilities, where they’re headed, what new things they can enable. That’s one. Then two, we think very deeply about the product features and surface areas that we should build to make that model intelligence accessible to more people.

Joseph Kim

你提到我演讲里有张幻灯片，画的是 OpenAI 内部的 Codex 采用曲线。挺有意思的：软件工程的采用曲线一路稳定向上，大家逐渐把它融入日常。可你看金融、法务和销售，几乎一直是零，然后突然跳到 90%。去看它跳升前后发生了什么——我们在 2 月发布了 Codex 桌面应用，采用率就是那时候跳上去的。我认为这说明了模型与产品协同开发的重要性。产品的目的，就是让尽可能多的用户用上模型智能。

I think you alluded to a slide in my keynote where we have this chart of Codex adoption within OpenAI. It’s funny because you can see the software engineering adoption curve, and it’s just up and to the right, pretty steady. People are picking it up and incorporating it into their daily routines. If you look at finance, legal and sales, it’s pretty much at zero, and suddenly it jumps up to 90%. If you look at what happened around the time it jumped up, we had released the Codex desktop app in February, and that was when that adoption jumped up. I do think it shows the importance of co-developing those things. The purpose of the product is to make that model intelligence accessible to as many users as possible.

Latent Space 主持人

在 Codex 用于金融场景的那波数据增长里，你们学到了什么，又用到了金融数据的后训练里？

In that uptick of data for using Codex for financial applications, what sort of things did you learn that you then applied into the post-training of financial data?

Joseph Kim

好问题。我可以讲讲我们怎么做 ChatGPT for Financial Services，这次很特别。我们做的第一方产品并不多，Codex 是最大、最主要的一个。这是我们第一次在一个屋檐下同时掌握产品、模型 harness 和研究，这给了很大的灵活度。

Yeah, it’s a great question. I could talk a little bit about how we went about building ChatGPT for Financial Services. It was very unique. We haven’t built a whole lot of first-party products. I would say Codex was really our biggest and main one. It was the first time we developed a product where the product, the model harness and the research were all under one house and one roof, and it really gives a lot of flexibility to go about building things.

过去六个月做这件事，我最大的收获是：在企业市场，你必须面对所在行业的现实，并围绕它去构建。这一点很容易被忽略。只从第一性原理出发，你可以得出一个合理的结论，用逻辑和推理去辩护它；但如果开始构建时忽视了所处世界的状态、行业里起作用的力量——竞争力量、每个人有自己的目标——你就做不出对的产品。

I think the biggest takeaway that I have from having worked on this for the last six months or so is: in enterprise, you have to deal with the reality of the industry that you’re in and go build around that. I think that piece is easy to ignore. If you just think about it from first principles, you can come to a conclusion that makes sense, and you can defend it with logic and reason. But if you ignore the state of the world that you’re in when you start building, and the forces that are at play within the industry, whether competitive forces or people having their own objectives, you’re not going to build the right product.

Joseph Kim

我们做 ChatGPT for Financial Services 时就看到了这一点，也想了很多。举个例子：数据。数据是我们构思产品时非常大的一块。从第一性原理看，这个决定一开始很难辩护：你说「我要把季度财报电话会文字稿接进产品、打包进来」，人们第一反应是：「应用商店里已经有 MCP 连接了，你打包进来的增量价值是什么？」纸面上看很糟：他们可能已经有授权，会觉得付了两次钱；他们可能已经有 MCP。你为什么要做这个？

We saw that with ChatGPT for Financial Services, and we thought about that a lot. I’ll give you one example: data. Data is a huge part of how we thought about building the product. Data was one of those things where on first principles it was a little bit tough to defend when I was talking about building this product, because you say, hey, I want to go and bring quarterly earnings transcripts into the product and bundle it. The first thing people are going to say is, look, there’s an MCP connection in the store. What is the incremental value that you’re providing by bundling that in? On paper, it looks bad. They might already have a license for it, so they might feel like they’re paying twice, and they might have an MCP. Why are you building this?

现实是，我们把这些 MCP 都测了一遍，发现错误率非常非常高。我们接入的一些 MCP，其中有些大牌，错误率在 40% 以上。MCP 就是那种纸面上很漂亮的东西，但你必须去看数据，看实际发生了什么。对数据，我们的结论是：这太糟了，规模化解决它的唯一办法就是把数据收进来、端到端掌控体验，多付的打包费是值得的。这是一个非常行业特定的产品决策，而且在客户那里很有共鸣，客户很喜欢。

The reality is we went and tested all these MCPs. What we found was the error rates were really, really high. Some of the MCPs that we connected, some big names, had a 40%-plus error rate. MCPs are one of those things where it looks great on paper, but you have to go look at the data and see what’s actually going on. For data, we were like, hey, look, this is really bad. The only way we can go about fixing this at scale is to bring the data in-house and control that experience end-to-end. The extra money that you pay on top to bundle that data is worth it. That was a product decision that we had to make that was very industry-specific. It’s been resonating with customers. Customers love it.

Latent Space 主持人

数据这一点值得强调，而且奏效了。说说技术层面：对于想为某个特定服务改造 harness 的人，你们改了 Codex、做了后训练，做了各种事，才有了 ChatGPT for Financial Services。如果有人要在别的领域——比如法律、销售——做同样的事，最佳实践是什么？把现成的东西拿来、真正做成一款产品，需要什么？

One thing to call out is the data, and it worked. Tell us about the technical side for people building a harness adaptation for a specific service. You went out, you changed Codex, you did post-training, you did whatever you did to make ChatGPT for Financial Services. If someone was to do the same for another domain, say legal or sales, what are the best practices? What goes into taking something off the shelf and actually making a product around it?

Joseph Kim

我很喜欢这个问题，我在演讲里也想点到这一点。有趣的是，谈后训练很好玩，因为那是我们独有的、让人兴奋的东西。但我们做的东西里有 95% 是创业公司、其他公司完全也能做的。我们并不需要动用修改模型的能力，就能做出很棒的产品。

Yeah, I like the question a lot. I tried to allude to this in my keynote as well. The interesting thing is, it’s fun to talk about the post-training stuff because it’s unique to what we do and gets people excited. But 95% of what we built was something that absolutely a startup could do, another company could do. We didn’t really have to leverage our ability to make changes in the model to go about building a fantastic product.

我想讲几点。第一，我们做的大部分产品都建立在 Codex——现在应该说 ChatGPT——的插件架构和插件原语之上，这是我们投入大量时间精力的地方。OpenAI 是一家有意思的公司：我们内部面对的问题，和一家在 OpenAI 之上创业的公司会遇到的问题一样。在很多方面，我的团队就是一家在 OpenAI 技术之上创业的公司。即使在内部，我们也需要找到可扩展的方式，让像我这样的团队站在巨人肩膀上、在 Codex 之上构建。

What I would say is a few-fold. One is most of the product that we built was built on top of the Codex, or I should say ChatGPT now, plugin architecture and plugin primitives. That is something that we’ve been investing a lot of time and energy in. OpenAI is an interesting company where we face the same problems internally that a startup would face if it were to build on top of OpenAI. In a lot of ways, my team is a startup that’s building on top of OpenAI technology. Even internally, we need to find scalable ways for teams like mine to stand on the shoulders of giants and build on top of Codex.

Codex 很棒：记忆好、技能好、能持续运行，有很多优点。我们团队的论点是：Codex 能把一款在市场上成功的产品做到约 80%，但缺了 20%。我们得想办法补上那 20%，因为论点的另一半是：如果你直接把 Codex 当成金融产品发布，它会失败。那 20% 至关重要。

Codex is fantastic. It’s got great memory. It’s got good skills. It’s persistent. There are a lot of things that are wonderful about Codex. The thesis of my team was: Codex will get you about 80% of the way to a product that would be successful in the market, but 20% is missing. We need to find some way for my team to go build that 20%, because the other half of the thesis was, if you just took Codex and launched it as a finance product, it would fail. That 20% is actually pivotal.

我们是在插件架构上搭建的，用的是所有想在 ChatGPT 上构建的开发者都能用的同一套原语。我演讲里有句话：我真心觉得会有人在一个 ChatGPT 插件之上做出 10 亿美元的生意。你可以借用所有横向功能，只在上面加你的独门配方——可能是数据，可能是 UI 改动，也可能是一个集成——那份独门配方很有价值。我认为那里有 10 亿美元的价值可做。

We went and built on top of the plugin architecture. It’s using the same primitives that are available to any of the builders that want to build on top of ChatGPT. I had a line in my presentation: I really do think that there’s going to be a billion-dollar business built on top of a ChatGPT plugin. You’re going to be able to leverage all of that horizontal functionality and just add your secret sauce on top. That secret sauce might be data. It might be UI changes. It could be an integration. But that secret sauce is really valuable. I think there’s a billion dollars’ worth of value to be built there.

Latent Space 主持人

现在业界正处在一个有意思的时刻：编排越来越自动化，一组组智能体出去办事。金融当然有很多风险。能谈谈你们在产品里内置的安全机制吗？

It’s an interesting time for the industry because orchestration is becoming a lot more automated, teams of agents going out and doing things. For finance, obviously, there are a lot of risks involved in that. Can you talk a bit about the safety that you’ve baked into the product?

Joseph Kim

安全非常非常重要，而且只会更重要，因为智能体已经越来越自主、越来越独立，我们预计这个趋势会延续到 2027、2028 年。就我们上个月发布的产品而言，关于安全、关于审计工作、确保模型不犯影响业务结果的错误，已经有一套现成范式，我们大量借用了它们。

Safety is super, super important. I would say that it’s only going to get more important, because agents are already getting more autonomous and more independent, and we expect that trend to continue into ‘27 and ‘28. For the product that we built and launched last month, there were a bunch of paradigms that already existed as it pertains to safety, being thoughtful about auditing the work and making sure that the model is not making a mistake that’s going to impact business outcomes. We were able to lean heavily into those.

我们发布的现有产品，是给这个领域知识工作者用的生产力工具，背后有一个很深的假设：人始终在环。你是一名金融分析师，当天有一长串待办事项。就像软件工程师用 Codex 那样，每一项你先试着用 ChatGPT 来做，视任务不同，它也许能做到 30%，也许能做到 80%，但前提始终是有人在监督这些工作。这让我们很容易把审计和复核的责任交到人手里，这套做法效果好，也容易沟通。

The existing product that we launched is a productivity tool for knowledge workers in the space. There’s a deep assumption that there’s always a human in the loop. You’re a financial analyst and you have a laundry list of things you want to get done that day. Just like software engineers do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT. It might get you 30% of the way there, it might get you 80% of the way there, depending on the task, but there’s always this presumption of humans in the loop presiding over that work. That made it very easy for us to build a product where we could pass the responsibility of auditing and review into people’s hands. That worked well for us and was easy to communicate.

不过我们在认真思考下一步，而下一步看起来不再是一个永远被人盯着的生产力工具。我们在想两件事。第一，怎样把产品的安全形态塑造成人们能直觉理解的样子？人们最直觉理解的，是怎么和其他人合作。所以我们的论点是：如果产品用起来很像和同事、实习生、身边的人共事，人们就能很快上手。因此我们需要把它的安全形态做得像一个人。

I will say we’re thinking very deeply about what comes next, and what comes next looks different from a productivity tool that is always being monitored by a human. As we think about what comes next, there are two things we’re thinking about. One is: how do we shape the security and safety profile of the product into something that people understand intuitively? Something people understand very intuitively is how to work with other people. So our thesis is, if we make the product very similar to how you work with your coworkers, your interns, the people around you, that is something people can pick up very quickly and easily. Therefore, we need to shape the security profile to look like a person.

我想点出几件事。一是读写权限的概念，即使对人也存在。你是公司新来的软件工程师——我在 Google 时，需要一个叫 readability 的资格——在你开始动系统之前，有一套制衡机制。监督人的工作的系统早已存在。如果模型的行为更像人，你就能借用这些系统，这让我们和金融机构沟通、推进起来容易得多。这是我们接下来会重点投入的方向。

A few things I’ll call out. One is there’s this notion of read/write, even for people. When you’re a new software engineer at a company, oftentimes, when I was at Google, you needed something called readability. So there are checks and balances and systems in place before you can start mucking around and touching the system. There are already systems in place to monitor the work of people. So if models behave more like people, you can leverage those systems, and that makes it very easy to talk through this with these financial institutions. That’s something we’re leaning into pretty heavily coming up.

Joseph Kim

最后一点：至少在银行业——我们就是从这里起步的——已经有一套体系：初级分析师做大量基础工作，交给经理（associate）复核，经理再加上自己的独门配方和推理，交给 VP，VP 再到总监、董事总经理（MD）。这是一种层级体系，内含隐性的工作检查，本来就是为了防范人会犯的错误——人犯的错和模型一样多。这些体系已经建好，我们在做金融产品时也在大量借用它们。

The very last thing I would say is, at least in the context of banking, which is where we started, there’s already this system in place: junior analysts do a lot of the groundwork. Their work gets passed off to associates. Associates review the work, then add their secret sauce, their reasoning. That gets passed off to the VP. The VP goes up to the director, managing director. So there’s this hierarchical system with implicit work-checking that’s already been built out to prevent the mistakes that humans make. Humans make just as many mistakes as models. Those systems are built, and we’re leaning heavily on them as it pertains to building financial products.

Latent Space 主持人

金融圈有一种误解：一个错误会层层放大，交错一次东西，整个职业生涯都有风险。那人们该怎么起步？如果你是分析师，该怎么用这些工具？我们现在处在什么位置，有没有类比——比如自动驾驶的自主等级？你有张幻灯片好像写着 2027 年不再有 PowerPoint，很多人会让 AI 工具进入工作流。人们该怎么看现在和未来，又该怎样采用这些工具，尤其当他们需要把越来越多工作交出去的时候？

I’d say there’s this sort of misconception in the finance world that one mistake really compounds: your whole career is at risk if you submit something wrong. How should people think about getting started? If you’re an analyst, how should you use these tools? Do you have any analogies for where we’re at right now, like levels of autonomy in self-driving? One of your slides, I think, said by 2027 you see no more PowerPoint; a lot of people will be using AI tools in the loop. So how should people think about where we are now, where things are going, and how they should adopt these tools as they need to offload more and more?

Joseph Kim

很好的问题。我有两个视角，起点假设不同。我认为「犯一次错职业生涯就完了」是不正确的看法，后面我有一条思路接着讲。但先假设它成立。那样的话，我想说：不是所有错误都等价，也不是所有模型能力都等价。可能终结职业生涯的错误，通常发生在你到了一定级别之后，比如你是 VP，赌注比初级分析师高，因为默认此前的工作已经被层层核对过。如果 MD 发现一个错误，那就是大问题，意味着很多安全网都失效了，错误才一路传到 MD。

Yeah, that’s wonderful. I have two perspectives on this, and they start with different assumptions. I think the notion that your career is over if you make one mistake is incorrect, and I have a line of thought that follows that. But let’s first assume that it’s true. In that case, what I would say is: not all mistakes are created equal, and not all model capabilities are created equal. The mistake that could be career-ending usually comes once you get up to a certain level. If you’re a VP, the stakes are higher than when you’re a junior analyst, because there’s a presumption of work having been checked up to that point. If an MD sees a mistake, that’s a huge problem. A lot of safety nets have failed for that mistake to make it up to an MD.

终结职业生涯的错误，是你把一条事实弄错了，而这条事实改变了结论——这显然是最严重的一类错误。我们在注意细节方面看到了很大进步。「幻觉」这个词在 2022、2023 年常被提起，现在少多了。模型在事实性错误上的可靠性正在大幅提升。所以那种会让你被开除的严重错误，模型越来越少犯，并会继续进步。我其实相当有信心：持续的进步加上现有的层层检查，会让这类错误一路传到 MD 的概率非常低。

The career-ending mistake is something where you get a piece of information factually incorrect, and that piece of factual information changes the conclusion. That is obviously the biggest type of mistake you can make. What I would say is we’re seeing a lot of progress in attention to detail. Hallucination used to be a big term in 2022, 2023; it doesn’t come up nearly as often nowadays. We’re starting to see huge improvements in the reliability of models as it pertains to factual mistakes. So the kind of egregious mistake that would get you fired, models are getting much better at avoiding, and they continue to get better. I actually feel quite confident that the combination of continual improvements plus the existing system of checks will make the chance of that type of mistake propagating up to an MD very low.

Joseph Kim

我想说，模型今天欠缺、仍需提升的，其实不是那种致命错误，而是涉及人的判断与品味的部分。比如「这张幻灯片很丑」——它没错，但排版、颜色有点不对，看起来很粗糙，所以不是一张好幻灯片。这不像事实错误那样会终结职业生涯，但仍然重要，关系到这份工作有多有用。

What I would say the models are lacking today, and still need to improve on, is actually not the career-ending mistake, but the bit that involves human judgment and taste. It’s: this slide looks ugly. It’s not wrong, but the formatting, the colors look a little bit off, something about it looks sloppy, and therefore this is not a good slide. That’s not career-ending in the way that a factual mistake is, but it still matters, in terms of how useful that work is.

展望 2027 年，我预计模型在人类品味这部分会好很多。这就是我在演讲里暗示的：在那些更软的目标函数上，模型的品味会好得多。问题不再是对或错，而是比较性的：有没有比这个更好的？我认为模型在这个维度上会进步很多。

As we think about the progression into 2027, I expect the model to get much better at the human-taste component. That’s what I was alluding to in my keynote: I think the model will get much better human taste on things that are softer objective functions. It’s not “is it right or wrong?” It’s comparative: is there something better than this? I think models are going to get much better along that dimension.

Latent Space 主持人

具体怎么做？怎么解决品味问题，怎么把品味训练进模型？

How so? How do we solve taste? How do we train taste into the models?

Joseph Kim

说来好笑，在早期 RLHF 的年代，我们训练模型用的基本上只有品味。基于人类反馈的强化学习，人类反馈始终是成对信号：给两个选项，哪个更好？所以强化学习的早期正是建立在训练人类品味之上。后来有了可验证奖励、环境，我们搭了很多机器去做其他类型的奖励和 RL，人们就忘了强化学习早期全是围绕人类品味的。

It’s funny. Very early on, in the early RLHF days, the only thing we trained models on was essentially taste. It was reinforcement learning from human feedback, and the human feedback was always a pairwise signal: given two options, which one’s better? So the early days of reinforcement learning were built on the notion of training on human taste. Then obviously came things like verifiable rewards and environments, and we built a lot of machinery for other types of rewards and other types of RL, so people kind of forgot that the early days of reinforcement learning were all about human taste.

那怎么训练人类品味？技术答案很简单，实践答案显然更复杂。实践答案就是：训练一个奖励模型，找一批人看成对的东西，说哪个更好。这招今天依然有效。一个冷知识：我们给幻灯片就是这么做的。怎么让模型更会做演示文稿？就是找一批人看成对的幻灯片，说哪张更好看。

So how do you train on human taste? I think the technical answer is quite simple; the practical answer is obviously more complicated. The practical answer is you just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. Fun fact: we did this for slides. How do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better.

人类的审美判断其实非常非常好，我们发现比模型好得多。人几秒就能看出两张幻灯片里哪张更好；而如果让一个评分模型来做，它会花大量推理去抠像素这里那里，最后也许还得出错误结论。所以人在改进模型上依然非常有价值，我们会大量依赖人类偏好来赋予模型品味这类东西。

Humans are actually really, really good at aesthetic judgment, actually much better than models, which is something that we discovered. A human can look at two slides and within seconds say this one looks better. If you have a model, like a grading model, try to do that, it’s going to spend a whole bunch of reasoning digging into the pixels this way and that, and then maybe come to the wrong conclusion. So humans are still really valuable in making these models better, and we will rely a lot on human preference to imbue things like taste.

最后补充一点：这有先例，让我们有信心这个策略会奏效，先例就是编程。编程还没完全解决，但假设我们走了 80%，正在攻最后 20%。编程是可验证奖励与人类品味的结合：可验证奖励是代码能不能编译、有没有语法错误、函数输出是否符合预期；但好的软件工程很大一部分在于：系统设计是否干净？是否便于长期维护？是否容易做单元测试？是否遵循业界大多数人认为正确的范式？

The last thing I’ll add is there’s precedent for this, which makes us confident this strategy will work, and that precedent is coding. Coding is not fully solved, but let’s say we’re 80% of the way there and making progress on the last 20%. Coding is a combination of verifiable rewards and human taste. The verifiable rewards are: does the code compile, are there syntax errors, is the output of the function what’s expected? But a lot of good software engineering is: is this clean system design? Did you build a system that’s maintainable over time? Is it easy to unit test? Is it following the paradigms that the vast majority of the industry thinks are right?

从这个角度看，很多金融工作非常相似。我有一个 Excel 模型，加总对不对？有没有引用错误？有一大堆可验证的东西能检查，确保工作通过第一轮合理性检查。但也有很多是：这是我会搭的方式吗？我开始做金融后学到一个概念，叫模型的「手感」：扰动输入、看输出，经验丰富的金融从业者能感觉到数据在模型里的流动是否顺。这种人类偏好，和「这段代码干不干净、写得好不好」非常像。我们在编程上的成功，让我有信心在金融上也能解决很多这类问题。

If you look at it from that lens, a lot of financial work looks very similar. I have an Excel model: does it sum up? Are there reference errors? There’s a whole bunch of verifiable things we can check to make sure the work passes a first-pass sanity check. But there’s also a lot of: is this the way I would have built this model? Something I learned as I started working in finance is the notion of the feel of the model: you perturb the inputs and look at the outputs, and an experienced financial professional can understand whether the data flows through this model in a way that feels good. That is a human preference that looks a lot like: does this code look clean? Is it well written? Our success in coding gives me confidence that we can solve a lot of these problems in finance as well.

Latent Space 主持人

这里可以提一下 Cognition 的 Frontier Code。它不只像 SWE-bench 那样问答案对不对，还问「你会不会合并这段代码？」你可能有多个都通过测试的合格 PR，但你会合并哪一个？这也是为什么有时会看到「推理量更低反而更好」的图表——它同时也是一个品味基准。

Yeah, we can shout out Frontier Code from Cognition. It’s not only SWE-bench-style “is the answer correct?” It’s also “would you merge the code?” You can have multiple answers that are proper PRs that pass, but which one would you merge? That’s why you sometimes see charts where lower reasoning is better, but it’s a taste benchmark as well, right?

Latent Space 主持人

你另一个 2027 年预测我觉得很有意思：你们训练的模型会更擅长形成投资论点、生成投资和交易想法。但这显然是经验丰富的金融专业人士非常擅长、有直觉的事，而且往往要与主流看法相反，否则获得高回报的机会就更小。你们怎么训练智能体做这类工作？

One of your other 2027 predictions I thought was interesting was that the models you’re training will become much better at investment theses and generating ideas for investment and trading. But obviously that’s something experienced financial professionals are very good at; they have an intuition for it. And it’s often counter to the prevailing wisdom out there; otherwise there’s less of a chance of a great return. So how are you tackling that problem in terms of training the agents to do that kind of work?

Joseph Kim

我想说，到目前为止，AI 在金融里的绝大多数应用都集中在初级分析师的工作上，原因有两个。第一，你必须基于今天模型的能力来做产品。今天的模型很擅长接受相当具体的指令并执行：给我做一张幻灯片，我想要这样，做得好看点，用我们的模板。模型很能胜任这些事，所以我们围绕它做了产品体验。

I would say that, up until now, the vast majority of applications of AI in finance have been really focused on the junior analyst’s work, and I think that’s for two reasons. One is you have to build the product based on the capabilities of the model of today. If you look at the capabilities of the model today, they’re really quite good at taking pretty concrete instructions and executing: build me a slide, I want it to look this way, make it look good, use our template. The model is quite capable of doing those things, and so we’ve built product experiences around that.

第二，人们强烈希望让模型擅长那些人不擅长或不喜欢做的事。我接触的大多数人——当然有例外——真的不喜欢搭 Excel 模型，那是繁琐、讲究细节、容易出错的工作。幻灯片也一样：复制 logo、把它们精确放到页面某个位置，很多人都不爱做。所以一部分动力就是聚焦这些事：现有模型擅长某些任务，加上人们喜欢做和不喜欢做什么，共同决定了产品的方向。

The other thing I would call out is there’s a strong preference to make the model good at things that humans are not good at or don’t like to do. Most people I talk to, with obvious exceptions, really don’t like building Excel models. It’s tedious work, very detail-oriented, easy to get wrong. Similarly with slides: you’re copying logos and placing them very precisely on a specific part of the page. It’s very tedious work that a lot of people don’t like to do. So that’s been part of the impetus for us to focus on those things. The existing model is pretty good at certain types of tasks, and human preferences about what they like and don’t like to do help drive the direction of the product as well. It’s a combination of those two things.

我认为这两件事里低垂的果实我们已经摘了很多。进入 2027 年，我们会开始沿着链条往上走：从初级分析师能做的任务，到经理能做的部分任务，再到 MD 会做的部分任务。往上走时，目标会变得模糊得多：从「这张幻灯片明显好或坏、能不能放到客户面前」，变成「我不知道这个投资想法好不好，可能要等三年看市场怎么走才能给你明确答案」。

I think we’ve solved a lot of the low-hanging fruit from those two things. Going into 2027, we’re going to start to move up the stack: from the tasks a junior analyst can do, to some of the tasks an associate can do, to some of the tasks an MD would do, and move up that chain. As you move up the chain, the objectives become much fuzzier. It goes from “this is clearly a good or bad slide, something I would or wouldn’t put in front of a client” to “well, I don’t know if this investment idea is good, and it would take me three years of waiting to give you a clear, concrete answer about what the market has done.”

目标变模糊，训练模型、让模型在这些事上变好就更难了，从研究角度更难。但我想说，并非不可能。它需要大量研究投入、大量优质数据、大量领域专家的反馈——我们真的想拿到 P99.9 那种水平的反馈，让模型和人一样好。我不认为这是无法逾越的问题。它需要基座模型持续扩展，以及专注：研究实验室要认为这是一个值得解决的重要问题。我在团队里做的事之一，就是让这个领域成为 OpenAI 的重点与优先事项。基于这两点，我有信心 2027 年会取得很大进展。

So the objectives start to become fuzzier, and the way you go about training the model and making it better at those things becomes more difficult, more difficult from a research perspective. But it’s not impossible, is what I would say. What it takes is a lot of research focus, a lot of wonderful data, a lot of feedback from experts in the field. We really want to get the P99.9 type of feedback to make the model as good as humans. I don’t think it’s an insurmountable problem. What it takes is continued scaling of the base model, and focus: the research labs thinking this is an important problem to go solve. One of the things I’m doing on my team is making this area a focus and priority for OpenAI. Because of the combination of those two things, I feel confident we’ll make a lot of strides in 2027.

Latent Space 主持人

你对现有的金融基准有什么看法？我觉得我们还停留在「数字对不对得上」的阶段。你怎么考虑做自己的评测、把品味加进去？这也是向这个领域的人发出的公开邀请，去做自己的基准。顺带一个问题：你说要基于今天的模型做系统，同时又做了很多对未来的预测。对今天交付给用户的金融服务，你们怎么设置护栏、限制和检查，让人们放心它不会失控？

Do you have any comments on the current financial benchmarks? I think we’re still in the state of “does this check out numbers-wise?” How do you think about making your own and adding in taste? I think this is also an open call for people working in this space to work on their own benchmarks. A follow-on: you said build systems for the models of today, but you also give a lot of predictions for the future. How do you put guardrails, limitations and checks on the financial services you build for people today, so they’re confident this is not going off the rails?

Joseph Kim

好，那我们聊聊评测，这是我很关心的事，我在评测上花了很多时间。我想分享的先验是：做出一个好的评测极其困难，没做过评测的人往往意识不到。要做出一个真正与你想衡量的技能、能力相关的评测非常难，因为你拼出来的东西只是那件事的一个代理，而且只是一个维度。这是第一点。第二，把答案键或评分方式做对非常繁琐；如果评分方式不对，这个评测就很容易被 hack、很容易被奖励投机，你可能在评测上得分很高，却在真正想衡量的底层技能上表现很差。

Yeah, makes sense. Let’s talk a little about evals, something near and dear to my heart; I spent a lot of time working on evals. The prior I would share with people is that building a good eval is incredibly hard. That’s not obvious to people who haven’t worked on an eval before. To build an eval that actually correlates with the skill or capability you’re trying to measure is very difficult, because whatever you put together is just a proxy for that thing, and it’s just one dimension. So that’s one. Two, it’s very tedious to get the answer key, or the way you grade, correct. And if you don’t grade the answers correctly, it’s an eval that’s very easy to hack, very easy to reward-hack. You could score very highly on the eval but actually do really poorly on the underlying skill or objective you’re trying to measure.

所以我的先验是：评测非常难，不只是金融，全都一样。我认为今天并没有一个真正好的金融评测。已有一些前人工作，我可以提我们做过的 GDPval，它更宽泛地覆盖知识工作。如果你们看 GPT-6 的发布，我们没提 GDPval。我们知道这个评测已经完全饱和，没饱和的部分基本是坏的，所以我们不再报告、也不再跟踪它。GPT-6 发布时，一个棘手之处就是我们没怎么谈金融评测。我们内部有一些，没多讲，也还不想现在分享，它们会来的；但当时没有好办法在这些评测上展示金融任务的进展。

So the prior I would share is that evals are really hard, not just in financial work, but across the board. And we don’t think there’s a really good eval today for financial work. There’s a bunch of existing prior work. I’ll reference some of the work we did, GDPval, which is a little more broadly about knowledge work. If you look at the GPT-6 announcement, we don’t talk about GDPval. We know that the eval is completely saturated, and the pieces that are not saturated are essentially broken. So we actually don’t report on it; we don’t track it anymore. One of the tricky things for GPT-6 was that we didn’t talk a whole lot about financial evals. We have some internally that we didn’t want to share yet. They’re coming. But there was no good way for us to show progress on financial tasks on these evals.

有件好笑的事：评测里有种偏差——如果数字好看，研究员就默认模型变好了，不去深究；如果你做出了一个强得多的模型，评测数字却难看、没有达到你想要的涨幅，你才会去调查。这种偏差是内嵌在评测调查里的。GPT-6 就发生过：我们在一组金融评测上跑 GPT-6，数字没怎么变。我们想：怎么回事？深入看了这些评测，发现很多地方是坏的。基本上，我们已经把这些评测做饱和了，答错的题目是由错误的答案键、描述不清的提示，或评测本身的其他缺陷导致的。

There’s a funny thing: I think there’s this bias in evals where, if the numbers look good, researchers assume the model has gotten better and they don’t investigate. If you’ve made a much more capable model and the eval numbers look bad, or don’t increase at the scale you want, then you go investigate. So there’s this bias baked into investigating evals. Something like this happened with GPT-6: we ran GPT-6 on a set of financial evals and the numbers didn’t really change much. We were like, what’s going on here? We did a deep dive into these evals, and lots of things were broken. Basically, we found that we had saturated the evals, and the questions it was getting wrong were driven by incorrect answer keys, poorly specified prompts, or some other flaw with the eval.

所以说，我的团队目前在做的一件事，就是打造一个好得多的评测，真正能预测「能否自动化某个职业的工作」。比如我们会做一个投资银行评测：如果你在这个评测上拿高分，你大概就能自动化投资银行初级分析师的大量工作。

All that to say, one of the things my team is working on currently is building a much better eval that actually predicts the ability to automate the work of a specific profession. The idea is we’ll have something like an investment banking eval where, if you score well on it, you could probably automate a lot of the work of a junior analyst in investment banking.

Latent Space 主持人

我们得收尾了。对金融圈、对 Latent Space 的听众有什么要喊话的？你们团队大概在招人。最后说说，大家能怎么帮到你？

We’ve got to wrap here soon. Any call-outs for people in the finance space and the Latent Space audience? Your team is probably hiring. Last thing to put out there: how can people be helpful to you?

Joseph Kim

好的。首先，请大家去试试 ChatGPT for Financial Services。它是企业产品，但我们在努力把它开放给更多 Pro 计划上的专业消费者，希望很快有好消息。第二，是的，我的团队在招人。产品的客户增长势头非常好，所以我们在快速招人、扩张团队。团队很多元，岗位很多，我快速点几个：当然有标准的前端、后端工程，也就是传统的产品开发；第二，我们非常需要优秀的产品经理。某种意义上，优秀的产品管理是在这个领域取胜的主要瓶颈，所以我们在投资银行、股票研究、资产管理等各个方向招产品经理。

Yeah, wonderful. First and foremost, please check out ChatGPT for Financial Services. It is an enterprise product, but we’re trying very hard to make it available to more prosumers on the Pro plan, so hopefully we’ll have some cool announcements there. Two, yes, my team is hiring. We’re seeing incredible customer traction with the product, and because of that we’re hiring and scaling the team rapidly. It’s a very diverse team with lots of different roles. I’ll call out a few quickly. There’s obviously standard front-end and back-end engineering, traditional product building. Two, we’re very much looking for wonderful product managers. In some ways, great product management is the main bottleneck in trying to win in this space. So we’re trying to hire product managers in investment banking, equity research, asset management, across the board.

另一个我想特别点名的岗位叫「领域专家」（subject matter expert），非常重要。我们在编程上做得好的一点是：训练模型、改进模型的人自己知道好代码长什么样，这在我们改进模型时是巨大的顺风。在金融上我们没有这个优势。所以我们在招很多能带来金融专业知识的人，和工程师、研究员、产品经理一起，把模型和产品在你的专业领域里做得更好。如果其中有打动你的，强烈建议你联系我，我在 LinkedIn 上能找到，职位也挂在我们的招聘页面上，很好找。

The other role I want to give a special shout-out to is something called a subject matter expert, and it’s really important. One of the things we did well in coding is that the people training the models and making them better understood what good code looks like. That was a huge tailwind for us as we made the models better. We don’t have that advantage when it comes to making the models better at finance. So we’re building out and trying to hire a lot of folks who can bring that financial expertise and work with the engineers, the researchers and the product managers to make the model and the product better in your specific fields. If any of those resonate with you, I highly recommend you reach out to me; I’m available on LinkedIn. We also have these job listings posted on our careers page. It’s pretty easy to find.

Latent Space 主持人

太好了，你已经回答了我想问的问题——我本来想问什么时候能试用。它会到 Pro 用户那里。

Great. Well, you answered my question. I was going to ask, when can I have a play with this? But it is coming to Pro.

Joseph Kim

是的，我们在非常努力地让更多人用上它。

Yeah, we’re working very hard to make it available to more people.

Latent Space 主持人

这很 OpenAI：先对企业有意义、先给企业，然后逐步下沉到常规模型，来到 Pro 计划。所以可以期待，日常使用中它在金融上也会越来越好。

That’s very common OpenAI stuff: it makes sense for enterprise first, and then it trickles down to the regular models and comes to the Pro plans. So expect, for your general usage down the line, it’ll get better at financial stuff.

Joseph Kim

没错，完全是这样。

Yes, absolutely.

Latent Space 主持人

太棒了，很高兴请到你。大家去看主题演讲，你在里面给了不少不错的预测，还稍微透露了点猛料——「我们修好了预训练的问题」。线索留给大家去看。感谢参与。

Awesome. Well, very excited to have you. Check out the keynote. You gave good predictions. You talked a little bit about some juicy stuff: “we fixed pre-training issues.” I’ll leave breadcrumbs for you guys to go watch. Thanks for joining.

Joseph Kim

太好了，非常感谢。

Yeah, awesome. Thank you so much.