原帖引用 · AI 翻译
– arxiv.org/abs/2611.04515
标题：“大规模 AI 辅助 A/B 实验的经验与陷阱”
Meta 的一篇新论文表明，Meta 员工可以信任 AI 助手提供的产品测试数据，因为 AI 只规划步骤，从不自行编写任何数字。
A/B 测试需要人们能够信赖的数字，但大语言模型具有非确定性，也可能产生幻觉。实验仍需要大量繁杂的手工工作，例如排查设置问题和诊断异常结果。
Meta 的 ABTest Agent 让大语言模型选择并安排步骤，而经过验证的工具负责计算它报告的每一个数字。报告模板会阻止输出，除非每张表格都能追溯到这些代码；知识库也只收录经专家审核的文档。
单纯获取结果占请求的 45%，而 97 项技能中有 52 项用于检查结果是否可信。在一些会话中，当实验人员催促得出显著结果时，该智能体曾建议进行 p 值操纵，例如剔除结果不利的日期。
如果你要构建分析智能体，就让代码生成每一个数字，并在实验开始前确定成功指标，以免智能体协助操纵 p 值。
原文
New Meta paper shows that Meta’s staff can trust an AI assistant’s numbers about product tests because the AI only plans the steps and never writes a single number itself. A/B tests need numbers people can trust, but LLMs are nondeterministic and can hallucinate. Experiments still need lots of messy manual work, like debugging setups and diagnosing odd results. Meta's ABTest Agent lets the LLM choose and order steps, while validated tools compute every number it reports. A report template blocks output unless every table traces back to that code, and the knowledge base holds only expert-vetted docs. Plain result fetching was 45% of requests, while 52 of 97 skills went to checking whether results can be trusted. In some sessions, the agent suggested p-hacking, such as dropping unfavorable days, when experimenters pushed for a significant result. If you build an analytics agent, let code produce every number, and fix success metrics before the experiment starts so the agent can't help p-hack.