Fix the model path and task before calling it a regression.
A model string may no longer point to the same served version. A benchmark number may combine correctness with efficiency. Keep the path, task, result, and measure visible.
First classify the symptom
| What happened? | First check | What not to claim yet |
|---|---|---|
| No response, connector error, or blocked send | Save the error and time; check the provider’s status record for the affected component. | A request that never completed is not a wrong answer. |
| A completed answer misses the task | Repeat a safe task with the same surface, requested model, settings, effort, and context. | One conversation does not establish a weight change or broad regression. |
| A model name may route differently | Record the request string and available response metadata; check current provider documentation. | A stable alias does not guarantee a stable serving version. |
| Usage or access changed | Record the plan, entitlement, selected/default model, effort, and quota display. | Price, quota, fallback, and stopped requests are not completed-answer quality. |
Current Gemini example
Google’s Gemini API documentation says the deprecated gemini-3.5-flash string routes API requests to gemini-3.6-flash and is no longer selectable in AI Studio. The page does not state when the route began or report comparative quality. Treat this as a verified product/serving change, not a regression verdict; check whether the documented path matches your request and available response receipt.
Read tracker alerts by status
AI Stupid Level’s 31 October tracker reports no confirmed alarm among 25 API-tested models, while two separate model/suite pairs are early watches. It reports 66 of 75 detectors armed and three models still calibrating. An early warning is not a fired detector. Its selected tasks are private and its API surface does not represent every consumer application.
Read NerfBench as a composite
BridgeBench’s board still shows 16 October as its latest test. The published method describes 50 private tasks, three attempts per task, and the first tracked session as the 100% reference. That baseline may not be model launch day. The score weights correctness 60%, output tokens 20%, and cost 20%; Codex uses 75% correctness and 25% tokens because cost data is missing. A price or token change can move the composite without the same-sized correctness change.
The operator says a change is confirmed after two complete, comparable sessions fall outside the 90–110% band on the same side. The current public board shows no session newer than 16 October. It does not show per-session component scores or uncertainty around the first reference. Those limitations matter when a reader asks about answer quality today.
Read longitudinal panels by their surface
LiveNerf reports no change in the latest completed Opus 5.5 window, but its panel runs through a pinned Claude Code subscription path. A current result on one interface is not a result for every product or task.
What a useful comparison records
- Product surface, requested model string, served-model identity when available, and date.
- Plan, effort, prompt or dataset, context, tools, settings, and region when relevant.
- Whether the request completed; final artifact; correctness, safety, latency, cost, and correction burden as separate measures.
- Sample size, repeated-task variation, rubric, grader, route, baseline, exclusions, and missing data.