aifollow.news
Back Rohan Paul
Rohan Paul· @rohanpaul_ai · X· · Original publication time AI score47

Deepnote releases Agent Initiative Benchmark to test when AI agents should join Slack conversationsMachine translation

AI introduction

Deepnote released Agent Initiative Benchmark, placing proactive AI agents in a simulated company Slack to test when they should intervene, what they should say, and when they should stay quiet. GPT-6 Astra scored 8.97 out of 10; two LLM evaluators scored runs against a reference answer.

Original post · Original

@rohanpaul_ai

@DeepnoteHQ beautiful product.

ReplyDeepnote@DeepnoteHQ
@rohanpaul_ai Thanks for sharing @rohanpaul_ai 💙
View this post on X ↗
Related contextRohan Paul@rohanpaul_ai
This is a beautiful concept for a benchmark. Deepnote (@deepnotehq), an agent workspace company, released the Agent Initiative Benchmark, a test that drops proactive agents into a company Slack and scores whether they know when to jump in, what to say, and when to stay quiet. GPT-6 Astra led with 8.97 out of 10, and GPT-6.1 Sol followed at 8.65 for $0.11 per task, against $1.87 per task for third-placed Claude Fable 5.2. It runs every model through the same loop: > A thread starts in a simulated company’s Slack > The agent decides whether to act at all > It picks which of 11 services to check and investigates > It posts and tags the right people, or stays silent > Two LLM evaluators score the run 0 to 10 against a reference answer
View this post on X ↗

Source: Rohan Paul · x.com

Products
Found an error? Send a correction