Sakana AI Fugu Ultra vs Fable 5: Benchmark Comparison Questioned Over Testing Scaffold Differences

According to monitoring by Beating, Sakana AI's multi-agent system Fugu Ultra's claimed wins over Anthropic's Fable 5 in scientific reasoning and coding benchmarks face widespread skepticism from the AI community.

Critics argue that benchmark scores are highly dependent on testing scaffolds used during evaluation. Different scaffold implementations can introduce 10-20 point variations, meaning reported performance differences may reflect system engineering optimization rather than fundamental model capability advances. Both Sakana AI and Anthropic released results based on proprietary, vendor-specific scaffolds without unified third-party testing environments, limiting the reliability of direct comparisons.