mkmloom

. It received 118 points and 65 comments.

They test it on launch day, then benchmark it against that. A deviation of above 10% is considered a change. They're currently tracking Opus 5.5 and GPT-6 Astra.

This bench famously detected a degradation of Opus 4.6 which Anthropic later blogged about. I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.

tech

This link was posted by [bryan0](https://news.ycombinator.com/user?id=bryan0) 1 hour ago on [HN](https://news.ycombinator.com/item?id=49901736). It received 118 points and 65 comments.

[Top comment](https://news.ycombinator.com/item?id=49902317) by [jug](https://news.ycombinator.com/user?id=jug):

> We also have Nerf Bench:
> 
> [https://www.bridgebench.ai/nerf-bench](https://www.bridgebench.ai/nerf-bench)
> 
> They test it on launch day, then benchmark it against that. A deviation of above 10% is considered a change. They're currently tracking Opus 5.5 and GPT-6 Astra.
> 
> This bench famously detected a degradation of Opus 4.6 which Anthropic later blogged about. I personally think people sense nerfs more often than they happen and that it's often about honeymoon effects.