Why capability-first products often feel like demos
A team sees a new model capability and immediately adds a chat box, generation button, or agent workflow. The prototype can be impressive while the product remains unclear. Users may not know when to use it, what to provide, how to judge the answer, or what to do when the system is wrong.
The missing work is translation. Product design translates general technical possibility into a specific promise, interaction, and result for a particular person in a particular context.
The four-layer framework
What can the system do?
Inputs, outputs, quality range, latency, cost, constraints, failure modes, and available tools or data.
What progress does the user need?
The decision, transformation, or coordination job that matters in the user's real workflow.
How will the user direct and judge it?
Entry point, context, states, evidence, controls, collaboration, recovery, and handoff.
What valuable change should occur?
Time saved, quality improved, risk reduced, conversion increased, or a better decision made.
Layer 1: Capability is a range, not a promise
Document the model's practical operating range. What input quality does it require? Which cases work reliably? Where does latency become disruptive? What information is unavailable? What errors are costly? A product promise should be narrower than the broadest demo and stronger than a vague claim of intelligence.
Layer 2: The task creates relevance
Choose a task where AI can improve a meaningful bottleneck. “Write text” is a capability. “Turn a support conversation into a reviewed follow-up that preserves policy and tone” is a task. The task gives the team a user, context, quality bar, and point of completion.
Layer 3: Experience creates usable confidence
The experience must help the user provide context, understand progress, review evidence, adjust direction, and recover. The right interface may be a form, editor, workflow, recommendation panel, or background assistant rather than a chat box. Match the interaction to the work, not to the model's default input format.
Layer 4: Outcome creates proof
Define success beyond feature usage. A high generation count may indicate value, confusion, or repeated failure. Measure the change in user or business outcome and then trace backward to experience quality, task completion, and model performance.
Measure each layer separately
| Layer | Useful measures | Misleading shortcut |
|---|---|---|
| Capability | Task-specific accuracy, latency, cost, critical failure rate | A general benchmark score |
| Task | Completion, time-to-decision, handoff quality, expert intervention | Number of prompts |
| Experience | Comprehension, correction, override, recovery, perceived control | Feature clicks alone |
| Outcome | Quality, conversion, risk, time, retention, downstream success | Output volume |
Use evaluation to connect the layers
OpenAI describes a useful evaluation loop as Specify, Measure, and Improve. For product teams, “specify” should include both model behavior and user outcome. Define examples of good and unacceptable output, but also define the workflow condition in which that output becomes useful.
Then review real cases. Model evaluation can reveal output failure; usability research can reveal whether people understand and control the experience; product metrics can reveal whether the workflow creates durable value. None of these signals is sufficient alone.
Three questions before building another AI feature
- Which user decision or task becomes meaningfully better? If the answer is only “it becomes automated,” the value proposition is incomplete.
- How will a user recognize a good result and correct a bad one? If judgment is impossible, capability can increase risk.
- Which outcome will change, and what evidence will prove it? Choose an outcome close enough to the product that the team can learn from it.
Frequently asked questions
Why do capable AI models still produce weak products?
Model capability does not define the right task, workflow, controls, evidence, or success metric. Product value appears only when those layers work together.
What should an AI product team validate first?
Validate that the task is important and the product can improve the user's outcome under real constraints before optimizing polish or adding more model features.
How should AI product value be measured?
Measure model quality, task completion, experience quality, and user or business outcome separately. One usage metric hides where value is being created or lost.
Sources and further reading
- OpenAI: How evals drive the next chapter in AI for businesses — Specify, Measure, Improve.
- Google People + AI Guidebook: Feedback + Control.
- NIST AI RMF 1.0 — practical, lifecycle-oriented AI risk management.
为什么“能力优先”的产品容易停留在演示阶段
团队看到新的模型能力，立即增加聊天框、生成按钮或智能体流程。原型可能很惊艳，但产品依然不清楚。用户不知道什么时候该用、需要提供什么、如何判断答案，以及系统出错后应该做什么。
中间缺少的是“翻译”。产品设计需要把通用技术可能性，翻译成特定的人在特定情境下能够理解的承诺、交互与结果。
从能力到价值的四层框架
系统能够做什么？
输入、输出、质量范围、延迟、成本、限制、失败模式，以及可使用的工具或数据。
用户需要取得什么进展？
真实工作流中重要的决策、转化或协同任务。
用户如何指挥并判断它？
入口、上下文、状态、证据、控制、协作、恢复与交接。
应该发生什么有价值的变化？
节省时间、提高质量、降低风险、提升转化，或帮助用户作出更好的决策。
第一层：能力是一个范围，不是一句承诺
记录模型在真实条件下的工作范围：它需要怎样的输入质量、哪些情况比较稳定、延迟何时会影响流程、哪些信息无法获得、哪些错误代价较高。产品承诺应该比最宽泛的演示更克制，也应该比“更智能”这样的模糊描述更具体。
第二层：任务创造相关性
选择 AI 能够改善的重要瓶颈。“生成文本”是一项能力；“把客服对话整理成经过审核、符合政策与语气的跟进消息”才是一项任务。任务为团队提供了明确用户、使用情境、质量标准和完成条件。
第三层：体验创造可用的信心
体验需要帮助用户提供上下文、理解进度、审阅证据、调整方向并从错误中恢复。合适的界面可能是表单、编辑器、流程、推荐面板或后台助手，并不一定是聊天框。交互形式应该匹配实际工作，而不是照搬模型默认的输入方式。
第四层：结果提供价值证据
不要只用功能使用量定义成功。较高的生成次数可能意味着价值，也可能意味着困惑或反复失败。先衡量用户或业务结果的变化，再向前追踪体验质量、任务完成和模型表现。
分别衡量每个层次
| 层次 | 有效指标 | 容易误导的捷径 |
|---|---|---|
| 能力 | 任务准确率、延迟、成本、关键错误率 | 通用基准分数 |
| 任务 | 完成率、决策时间、交接质量、专家介入 | 提示词数量 |
| 体验 | 理解度、纠正、覆盖、恢复、控制感 | 只看功能点击 |
| 结果 | 质量、转化、风险、时间、留存、下游成功 | 输出总量 |
用评估连接四个层次
OpenAI 将一套有效的评估循环概括为“定义、测量、改善”。对产品团队而言，“定义”不仅要说明模型应如何表现，也要说明用户应该获得什么结果。需要定义好输出与不可接受输出的例子，也需要定义这些输出在什么工作流条件下才真正有用。
之后应审阅真实案例。模型评估揭示输出问题；可用性研究揭示用户能否理解和控制体验；产品指标揭示流程是否创造持续价值。任何单一信号都不足以代表完整结果。
增加下一个 AI 功能前，先回答三个问题
- 哪个用户决策或任务会得到实质改善？ 如果答案只有“实现自动化”，价值主张仍不完整。
- 用户如何识别好结果并纠正坏结果？ 如果无法判断，更强的能力反而可能增加风险。
- 哪个结果会发生变化，什么证据能够证明？ 选择与产品足够接近、能够帮助团队快速学习的结果。
常见问题
为什么模型能力很强，产品体验仍然很弱？
因为模型能力不会自动定义正确任务、工作流、控制、证据和成功指标。只有这些层次共同工作，产品价值才会出现。
AI 产品团队应该先验证什么？
先验证任务是否重要，以及产品能否在真实约束下改善用户结果，再投入界面精修或增加更多模型功能。
应该如何衡量 AI 产品价值？
分别衡量模型质量、任务完成、体验质量和用户或业务结果。单一使用量指标会掩盖价值在哪个环节产生或流失。
资料来源与延伸阅读
- OpenAI：How evals drive the next chapter in AI for businesses：定义、测量、改善。
- Google People + AI Guidebook：反馈与控制。
- NIST AI RMF 1.0：面向全生命周期的 AI 风险管理。
Build the experience, not only the capability.
A focused product design review can connect your model, workflow, user judgment, and measurable outcome.