Jev benchmark grades and coding-agent settlement: research of 11 October 2026

This dated report records the evidence behind release regular-2026-10-11-benchmark-grades-coding-agents of 11 October 2026. It refreshes benchmarks (three comparisons graded from opened files: 34 → 37 graded rows), Jev alternatives (Microsoft-Decision-1 added: 30 → 31 substitutes), coding agents (24 integrations read in source) and When Jev fails (71 → 74 rows). No new pages. The pool ran nothing: no package, plugin, model or benchmark was installed or run, and no Jev or OpenAI call was made.

Run: run:4804d142-c2f9-40b1-8aa1-418259bd9a1a (scheduled regular research run 13) · Pool: pool_jev_catalog · Runner: one runner plus two research helpers (A: real-time-control harnesses and drift; B: coding agents) and a spare-time helper. Checks: 11 Oct 2026; runner started 05:15:58 UTC; home status written 05:19:56 UTC; runner done 05:34:59 UTC; manager review 05:40 UTC.

The run's research notes and its source ledger (154 records, R13-S… IDs) are held in the pool's private record. Key sources are linked below and on the pages each finding feeds. Pre-return checks passed: 173 cited IDs in the research files plus 51 in the report, 0 missing; a recheck of 74 IDs after the manager's decisions, 0 missing. 14 count-word sentences were checked: every category list adds up to its total, and every category with 3 or more items (or 10% of the total) is named.

Rules kept: nothing was installed or run, so no Jev or OpenAI call was made. Failure cells come only from source at pinned commits or from published package files read in memory. Study results are the operators' own measurements. Nothing in this run is Tested.

Previous report: 10 Oct research report. All dated reports: Research.

1. Summary answer, as of 11 October 2026

• Access is unchanged (11 Oct 2026, 05:17 UTC). Status open_with_conditions since 27 Sep (signups open, no new-user credit); the API is operational. Access status.
• Benchmarks. The three comparisons listed as leads on 10 Oct are now graded A from their metric and results files opened at pinned commits (B38, B39, B40). The ledger has 37 graded rows: A 10, B 12, C 8, D 7. The two comparisons with OpenAI's Decisions API point in different directions by task. B40 is a reranking-pipeline study in which Jev and Cohere Rerank 4 Pro are within noise.
• Alternatives. Microsoft-Decision-1, a hosted decision model published by Microsoft on 9 Oct, is added to the "Hosted API from another provider" layer. The page now lists 31 substitutes: 12 weights, 11 runtimes, 4 adapters, 2 classifier or LLM approaches and 2 hosted APIs.
• Coding agents. Three rows are now counted: jev-triage (OpenClaw), the jev-for-all launcher, and hermes-jev-mana (Hermes Agent), which is inert after its documented install. Hoppin-Sorter/jev-router is read and not settled.
• When Jev fails. The three new rows are routing fallbacks: n = 74, routing fallbacks 5 → 8, action goes ahead 15 of 74 (unchanged number).
• Fast control loops (research for run 14). The condition is met: two third-party harnesses that call Jev were read at pinned commits with every failure cell recorded. Nothing is published on use-cases/real-time-control this run (section 6).

2. New rule of 11 Oct 2026: grades rest on opened files

3. What this release changed

No new pages. Edited in place:

• Benchmarks: rows B38, B39 and B40 (grade A, with flags); new key-finding clause; "Added 11 Oct" paragraph; grade counts 37 = A 10, B 12, C 8, D 7; leads for classifier-bakeoff, decisionbench and 10 new comparison repositories.
• Jev alternatives: Decisions API evidence cell rebuilt from graded rows; new Microsoft-Decision-1 row; counts 30 → 31.
• Coding agents: new key finding; jev-triage, jev-for-all and hermes-jev-mana counted; Hoppin-Sorter/jev-router not settled; wording fixes for jev-effort-router and jev-approvals; triage table extended; README-only candidate gates 8 → 6.
• When Jev fails: 3 new rows; recount n = 71 → 74; new key finding.
• One linked line each on Laya vs Kev (B39) and reranking for RAG (B40).

4. Corrections to published pages

• Alternatives, Decisions API "Parity or quality evidence" cell. It said "Lead, not graded" for every comparison. Two of them are now graded A (B38, B39). The cell is rebuilt from the graded rows only, and only that row is re-dated (section 5.2).
• Alternatives, "Hosted API from another provider: 1 entry". Microsoft published Microsoft-Decision-1 on 9 Oct (R13-S32), so the count of 1 was incomplete from 9 Oct. It is now 2, and the total is 31.
• Coding agents, jev-effort-router row. The failure cells hold. Two wording fixes: "routing on in its defaults" is true of the plugin's own config only, since Hermes plugins are off until enabled; and with Hermes defaults (model: "", config_defaults.py L21–22; R13-S217) the plugin does nothing, because it acts only when the provider is Ollama:Cloud.
• Coding agents, jev-approvals row. "May ask the main model" is right only when a main model is configured. Added: "(with no main model configured, it asks you)".
• Coding agents, jev-triage cells. Below threshold was published as "acts-anyway"; the source shows the message goes to the main model untouched (index.js L45; R13-S200), so the cell is now "fallback-main model". Malformed answer was "not recorded" on 10 Oct and is now settled as "fallback-main model".

5. Findings per page

5.1 benchmarks (protected): B38–B40 graded

Each row was graded from its metric file, its results file and the data reference, opened at the pinned commit (one tree call per repository). Results are Reported in the operator's own units; nothing was recomputed or run. Grade scale as published: A = data, code and configuration are published and the operator states no commercial stake. Flags are listed separately and never averaged into a grade. Each operator is an individual with no stake stated.

The anchors #lead-decisions-rpg, #lead-decisions-laya and #lead-rerank-bench stay on the page and point to B38, B39 and B40.

Leads, not graded (no grade, no grade colour):

• alpersonalwebsite/classifier-bakeoff at 4a1c4a0: report opened, metric code not opened. The report states 200 synthetic messages written by a Claude model, with Decisions API 88.0%, Sonnet 86.0% and Jev 84.0%, "all within noise" Reported (R13-S67).
• KizitoNaanma/decisionbench: not done this run; results location not identified.
• 10 new comparison repositories from this run's scan, README or description only, not opened (R13-S23): kartikanand73/decision-api-benchmark, etsabary/luna-decisions-benchmark, janagarajsn/jev-vs-laya-vs-vega, Mike-Animal-Counseling/laya-jev-lab, arghya05/jev-boundary-study, truongxxxx/jev-polymarket-test, Markonick/jev-bakeoff, davra02/req-classifier, gbozelli/pokemon_match, alexcpn/llm-invaders.

Of the 10 A rows, 3 are provisional in the page's own sense (B13, B15, B24) and 7 were read at pinned commits (B29, B31, B32, B37, B38, B39, B40).

5.2 alternatives (protected): evidence cell and Microsoft-Decision-1

Count basis: the 30 published rows plus Microsoft-Decision-1, counted by layer by script. A row with two layers counts once, under its primary layer.

5.3 build/coding-agents: settlement

Count basis: 22 published rows plus 2 README-only gates read this run (jev-for-all, Hoppin-Sorter/jev-router) = 24. Manager decisions of 11 Oct 2026, 05:40 UTC: hermes-jev-mana counted under the 7 Oct rule (plugins that are off after install still count, with the state stated next to the row); jev-for-all counted on its launcher only.

• 5 let the call run + 3 hold + 5 fall back (1 may ask the main model: jev-approvals; 1 keeps its configured model: jev-effort-router; 1 passes the message to the main model: jev-triage; 1 uses its default tier: jev-for-all; 1 runs the original Hermes branch: hermes-jev-mana) = 13 counted.
• 8 return Jev's answer to the agent + 2 give advisory hints = 10 not counted.
• 1 not settled (Hoppin-Sorter/jev-router).
• 13 + 10 + 1 = 24.
• Change: read 22 → 24; counted 10 → 13; not counted 10 (unchanged); not settled 2 → 1. On a Jev error: held 3/10 → 3/13; the call runs 5/10 → 5/13; fall back to a model or the original path 2/10 → 5/13.
• Under the 7 Oct rule, 3 of the 13 counted rows do nothing until switched on: jev-approvals, jev-curator and hermes-jev-mana.

• fallback-<what>: a named substitute
• not recorded: not read in source

• Not settled: Hoppin-Sorter/jev-router (Claude Code, Codex, Cursor) at bdb4721 rewrites the model on every main-loop request in its default "auto" mode. How the host engine handles the rewrite cannot be read at a pinned commit, because the Claude Code engine source is not public.
• README-only candidate gates: 8 → 6 (jev-for-all and Hoppin-Sorter/jev-router were read in source).
• Drift: @openclaw/typesafe 2026.10.1 was released 10 Oct; the jev-triage host branch, read at 2026.10.1 up to L650, has the same effect.

Combined: 31 lines, 30 unique projects (one npm package is the same project as a GitHub repository).

5.4 build/when-jev-fails: recount 71 → 74 (protected)

A script read the published cells (71 rows) and first reproduced the published error split, 38 / 17 / 8 / 5 / 3. It then added the three rows counted on coding agents. Each is a routing fallback: when a decision fails, the work goes to the main, default or configured model by the original path.

• Routing fallbacks: 5 → 8.
• Action goes ahead on a Jev error: 15 of 74 (8 fail-open plus 7 non-routing fallbacks; the number is unchanged).
• No key "not recorded": 18 → 19 (hermes-jev-mana).
• The sentence on the 8 that let it through (4 moderation bots, 2 agent hooks, a passage filter and an eval tool) is unchanged.

6. Access status and home notices

Checked 11 Oct 2026, 05:17 UTC: Documented. Status unchanged.

• Status open_with_conditions since 27 Sep 2026, 22:33:25 UTC: signups open, no free credit for new accounts. Service operational ("All services are online"). Limits, price and aliases unchanged since 10 Oct (R13-S05, R13-S06).
• Shown on Home (2): new-user credit disabled (rank 1); lookalike resellers (rank 2).
• Retired: none. New notices: none.
• The status page also lists api.us.typesafe.ai ("served from the closest US region") beside the regional hostname it already listed. TypeSafe's docs do not document these hostnames (11 Oct 2026, 05:17 UTC; R13-S05, R13-S07). Note only. Access status.
• Microsoft-Decision-1 is an alternative from another provider, not a change to Jev access.

7. Game and robotics control-loop harnesses: research result for run 14

Condition met: two third-party harnesses that call Jev were read at pinned commits with every failure cell recorded. A third calls a local substitute model by default. All three are third-party runnable demos; none is a product and none is published by TypeSafe.

8. Drift and spare-time results

• pydantic-ai-slim 2.55.0 (compared from the PyPI wheels; R13-S119, R13-S120): checked at 2.55.0, failure path unchanged; requests with images now fail with UserError before any network call.
• @effect/ai-typesafe 4.0.3 (patch, 10 Oct) and @openclaw/typesafe 2026.10.1 (10 Oct): drift notes. Everything else on the drift list is unchanged since 10 Oct (registry check 05:22:26 UTC; R13-S35).
• SDK default timeouts: typesafe-sdk 0.7.4: 10.0 s per request, 2 retries, a 30 s total retry budget; a timeout raises a subclass of TypeSafeError; a missing key raises when the client is created (R13-S300 to R13-S305). @typesafe-ai/sdk 0.6.0: 10,000 ms per attempt, 2 retries (R13-S310).
• LanceDB TypeSafe reranker with no key at b1b080b: no error at setup, an error at the first rerank call, no fallback (R13-S311, R13-S312).

9. Evidence gaps and uncertainties

• B39's significance test was not found in the files read. B38's full call logs are not committed.
• classifier-bakeoff's metric code was not opened; decisionbench's results location is still unknown; the 10 new comparison repositories were not opened.
• OpenClaw 2026.10.1 was not read past L650.
• hermes-jev-mana's client key handling and the Hermes version it patches were not read, so its no-key cell is "not recorded".
• jev-for-all's Claude Code hook is not in the published package, and OpenCode's handling of removed tools was not read. The Claude Code engine is not public source.
• What the two status-page hostnames are for is not documented by TypeSafe.
• Legal "Last updated" dates were not re-extracted.
• Reddit returned HTTP 403. X, YouTube, TikTok and GitHub trending were not covered: their coverage of the scan window is unknown, not zero.
• "jev vision", "jev vlm" and "jev vla" appear in search suggestions with no TypeSafe source: unexplained.

10. Not done (for the next run)

• decisionbench's lead line.
• ra2web's threshold values in doctrine.py.
• typesafe-spring-ai on Maven.
• Spare-time queue items 3–5: pg_jevplanner, the unchecked alternatives list plus TOD's base licence, and jev-graphrag-poc.
• Products recount (jev-triage, jev-for-all and hermes-jev-mana are code-backed integrations to add if missing).