Pool topic: AI Model Degradation Watch
Question revision: 1
Exact question: Are popular AI models getting worse right now, and what evidence distinguishes real capability degradation from outages, routing changes, product changes, safety behavior, pricing, access limits, and anecdotes?
Two Opus 5.5 trackers add counterevidence, not a universal verdict.
Current call: broad capability decline remains unestablished. BridgeBench’s 16 October score sits within its stated band. LiveNerf’s first complete follow-up window reports no change on a narrow reasoning-and-math panel. Their tasks, baselines, and scoring are different; neither tells us how every user’s workflow performs.
What the trackers report
| Source | Latest dated result | Scope and limit |
|---|---|---|
| BridgeBench NerfBench, 16 October | Claude Opus 5.5 is 105.8 relative to its first tracked session; 95.7 on 9 October, a rise of 10.1 index points. All seven listed sessions remain in its stated 90–110 band. | Private 50-task suite. Public report shows the composite, not per-task results or component scores. The first tracked session is the reference; materials reviewed do not establish it as a launch-day baseline. |
| LiveNerf public repository, update checked 17 October | First complete 10-day follow-up window reports no change: +0.007 ± 0.018 score units on a 78-item calibrated reasoning/math panel; 780 attempts in baseline and window. | Second follow-up window was five of ten days complete in the dated update. Its pre-registered persistence rule was not yet decided. The selected panel is not a sample of consumer workflows. |
Why the BridgeBench index is not an accuracy-only score
BridgeBench’s methodology describes 50 private tasks, three attempts per task, and a composite weighted 60% correctness, 20% output tokens, and 20% cost. If cost data is missing, the methodology says the score reweights to correctness 75% and tokens 25%. A lower token or cost figure can improve the composite even if correctness slips; the method says it checks for that condition. The public result page does not show component changes, so this review cannot say what drove the 16 October score.
The operator describes 90–110 as a normal band and requires two consecutive complete, comparable sessions outside the band before publishing an alert. It also treats each route or surface as a separate series. Its methodology describes saving the model and request IDs, but the default OpenRouter upstream route is not pinned and the public score page does not display the saved profile, component results, or task-level output. These are meaningful limits when interpreting a private benchmark result.
LiveNerf’s pre-registered rule is not yet a two-window verdict
LiveNerf’s preregistration calls for a 99% interval excluding zero in two consecutive 10-day follow-up windows, an effect of at least three points, and a control arm that does not show the same movement. The first complete window was reported as no change. The 17 October status said the second window had five of ten days; therefore the series had not yet fulfilled or failed the complete two-window decision rule in that update. The repository reports a minimum detectable effect of 6.6 score points against a predicted 7.5; that describes its planned sensitivity, not a guarantee that smaller effects are absent.
The repository reports a pinned harness/CLI across its 25 collected daily sessions; one day used Anthropic API credits instead of the planned Max subscription, a disclosed deviation. Its narrow calibrated panel and subscription/API deviation are useful to interpret, not evidence of consumer workflow quality.
How to read the apparent agreement
Both projects currently provide counterevidence to a simple story of a sustained, large Opus 5.5 decline on their own measures. That is not a pooled estimate: the tasks, reference periods, score definitions, sample structures, and serving paths differ. BridgeBench’s composite can move with tokens and cost; LiveNerf tests selected reasoning/math tasks. A private task with a small or domain-specific loss could be missed by either. A changed product, client, or route may also be separate from model weights.
Recent community discussions describe filtering model comparisons by domain or using a small private workflow suite to decide what is “good enough.” These are individual signals, not representative demand. For a reader, the most informative next test is a local, repeated replay of the same task under recorded conditions, measuring correctness and correction work separately from speed and price.