What changed for Jev builders, 8 to 9 November 2026
Three published Jev benchmark studies were graded from their own files, so the benchmarks ledger now has 48 graded rows. Two leaderboards were re-read: one is unchanged apart from a longer list, the other changed its index. auto-mode 4.0.0, a Claude Code permission mod that asks Jev, was read in source: its failure path is unchanged and it stays unsettled. Nace.AI's legal pages now agree with each other; OpenCode's two texts still do not. TypeSafe's aliases, model IDs, price, signup and credit state did not change.
Three more Jev studies are graded from their files: B49 (grade B), B50 (grade A, small sample) and B51 (grade B, the author's own harness). The ledger has 48 graded rows: A 15, B 16, C 10, D 7.
auto-mode 4.0.0 handles a Jev failure as before. Its bypass now lists 7 skip stages, including a reply-channel allow added in 3.1.x. It stays unsettled; the coding-agent counts (15 + 10 + 4 = 29) and the failure map (80) are unchanged.
Nace.AI's website and Console terms, dated 6 Nov 2026, now agree: no training on your content unless you opt in. OpenCode's docs and Terms (effective 8 Nov 2026) are still not reconciled.
Three studies graded from their files
Five repositories published between 23 Oct and 8 Nov 2026 were opened at pinned commits. Three had a metric in code and a results file that matched the stated figure, and were graded on the benchmarks page's A–D scale; two are leads, not graded. Every figure below is the operator's own measurement, in the operator's units. The studies use different tasks and baselines, so their numbers are not comparable with each other.
Graded rows on the benchmarks ledger, by grade: 24 Oct and 9 Nov 2026
- A rerunnable
- B partly rerunnable, or with a stake
- C method described
- D claim only
Bar lengths are each grade's share of that day's graded rows. A grade says how far a study can be checked, not whether its result is right. New on 9 Nov: B50 (A); B49 and B51 (B).
| Study, operator and interest | Task and baselines | Result as reported | Grade and why |
|---|---|---|---|
B49 dengzhe-hou/S1CogBench at 9cd68dfThe benchmark's author; no commercial stake found | A Bayesian urn task from a multi-task cognitive benchmark: the model's probability against the Bayesian posterior (mean absolute error; lower is better), n = 400 for Jev 1.14. Baselines: Claude Opus, Sonnet and Haiku; Intern-Decision; Laya; a uniform guess | Jev 1.14 0.068; Claude Opus, Sonnet and Haiku 0.001; Intern-Decision 0.235; Laya 0.291; uniform guess 0.349. Only this task was read | B The raw Jev responses are not in the repository and some modes need gated data |
B50 GopalAlag/FrontierLLMvsJEV at 27a73b7An individual; no stake found | IVR intent routing (exact intent), 250 items per iteration, 2 iterations. Baselines: GPT-6 Luna, Claude Sonnet 5.5, Gemini 3.1 Flash-Lite, Claude Haiku 5.5, a TF-IDF classifier | Accuracy: GPT-6 Luna 0.652, Sonnet 5.5 0.644, Jev 0.636, Flash-Lite 0.620, Haiku 5.5 0.608, TF-IDF 0.304 | A Code, data and results published. Flags: small sample; the README's run command differs from the reported run's manifest |
B51 tukka25/graph-reflex at 47859bdThe author's own harness is under test (conflict of interest) | Product search on STaRK-Amazon (public data), 100 questions, Hit@1. Baselines: a Claude Opus 5.5 agent writing Cypher, a Haiku 5.5 decider in the same harness, a cross-encoder, vector search | Opus 5.5 agent 0.69, harness with Jev 0.58, harness with Haiku 5.5 0.50, cross-encoder 0.46, vector search 0.33 | B Per-question outputs are not committed, and the author has a stake. Flags: small sample, public data |
- Not graded:
mahirmalik030/Jev-vs-LLMs(no results file in the repository and no Jev call in its evaluation script) andrandlee/judge-jev(a compliance judge built on Jev, not a benchmark). Details on benchmarks. - The small-LLM count is not redone. The benchmarks page's count of studies against small text-generating LLMs (ahead in 6 of 10, as counted through 24 Oct 2026) does not yet include B49 to B51, although each names a small tier; only one figure per study was matched.
- No change to the Decisions API comparisons: none of the three compares Jev with OpenAI's Decisions API or Microsoft-Decision-1.
Leaderboards re-read
| Leaderboard and operator | 24 Oct 2026 | 9 Nov 2026 |
|---|---|---|
| Jev Decision Index A community operator, not affiliated with TypeSafe | Version 0.4.1 (data of 22 Oct); Jev 1.14 60.24, Jev 1.13 59.22; 229 entries | Version 0.4.1 (data of 6 Nov, commit e05cd04); Jev 1.14 60.24, Jev 1.13 59.22, unchanged; 275 entries. The position shown in the app was not re-checked |
| Artificial Analysis Decision Index Artificial Analysis | Version 1.0; Jev 1.14 57.07, Jev 1.13 57.50; 32 entries | Jev 1.14 66.96; Jev 1.13 no longer listed; 21 entries. The index has changed since 24 Oct (a component is now named "AA-Omniscience-Decision v1.1"); its method was not re-read, so the two readings may not be on the same scale |
Reported Leaderboard data read 9 Nov 2026, 11:11–11:25 UTC (R29-S208 to R29-S210, R29-S39, R29-S215, R29-S41, R29-S45). Full section: benchmark leaderboards.
Is Jev deterministic?
Not fully, in one published option-order test: reordering the same three options changed Jev's route on 41–42 of 500 cases (the same route in all six orders 91.6–91.8% of the time), against 179 for OpenAI's Decisions API and 12–14 for Microsoft-Decision-1 (B41, grade A; the operator's own runs, README unchanged on 9 Nov 2026). Reported This is one task, synthetic bank-withdrawal routing, and it does not measure repeated identical requests.
TypeSafe's own figures. TypeSafe's cookbook on verifying LLM answers against their source reports, for jev-1.14.0 on 1,000 MuSiQue questions answered by gpt-6-luna: Jev's check passed 73% of the answers, 95.7% of them right, and sent 27% to review, including 84 of the 115 wrong answers. Reported TypeSafe sells Jev; its figures, outside the graded ledger and not checked independently (cookbook read 9 Nov 2026, unchanged since 8 Nov; R29-S211). Now also on Jev as a judge.
auto-mode 4.0.0: failure path unchanged, still unsettled
auto-mode is a community mod that asks Jev whether a Claude Code tool call may run without asking the user. Version 4.0.0 (published 8 Nov 2026) is a major release; the six files the coding-agents row cites are byte-identical to 3.1.6, and the 4.0.0 changes sit in a review step that runs only after a deny. The row was last read at 3.0.0 (21 Oct 2026).
auto-mode's six cells, 3.0.0 and 4.0.0 (terms as on the failure checklist; as read, not counted)
Grey marks an unchanged cell, amber a cell whose content changed. The row stays unsettled because the mod host's handling of auto-mode's answer is not in auto-mode's source and was not read, and the skip stages' internals were not read line by line. Cell wording as on coding agents.
Data terms: Nace.AI resolved, OpenCode not
| Provider | Before | 9 Nov 2026 |
|---|---|---|
| Nace.AI (Drex) a competing hosted decision API | From 9 to at least 21 Oct 2026 its website Terms (training by default, opt-out, up to 30 days) and Console terms (no training unless you opt in; deletion within 48 hours or 7 days) disagreed | All four documents dated 6 Nov 2026 say the same: no training unless you opt in; in Standard mode inputs and outputs are kept until you delete them; the text of Decision requests is kept 30 days in a separate archive for misuse screening. Changed between 21 Oct and 9 Nov 2026 |
OpenCode (jev-1.13-free)third-party route; not run by TypeSafe | Docs: providers follow a zero-retention policy; Terms (effective 15 Aug 2026): content from unpaid accounts may be used to improve the services | Docs unchanged; Terms now effective 8 Nov 2026 and add "except Content obtained or derived from a Connected Service". Whether the free route counts as one is not stated on the pages read. Still not reconciled |
Also on alternatives: Nace.AI now says Drex Max is "70.8 on the Decision Index, 10.1 points over Jev 1.14" (6.9 points on 20 Oct), and Upright's Eevo announcement carries a six-model table including Jev 1.14 (both the vendors' own figures, Reported). Upright's announcement names no weights or base model (read 9 Nov 2026).
What else changed
| Fact | Before | Now |
|---|---|---|
@ai-sdk/typesafe-ai | 3.0.35 (7 Nov) noted, not compared | Compared with 3.0.34: it rejects empty Choice or Score criteria and empty Score levels before sending, sends primitive JSON state as text and returns Score legends in provider metadata; its image and error lines are not in that diff; no page cell changed |
ai (AI SDK) | 7.0.165 (7 Nov) noted, not compared | 7.0.165 and 7.0.166 (9 Nov): the decide functions are unchanged from 7.0.164; telemetry and documentation changes |
| Integration releases | Read or noted at earlier versions | @openclaw/typesafe 2026.11.2 (8 Nov), nace-sdk 0.7.0 and n8n 2.47.3 (9 Nov) are newer; not re-read |
| Status-page uptime figures | 99.942%, 99.842%, 99.883% | 99.941%, 99.848%, 99.882%; the third is read as the 90-day figure; no event listed after 30 Oct |
| OpenRouter's ZDR endpoint list | 1,117 endpoints | 1,109 endpoints; both TypeSafe endpoints listed |
Re-read on 9 November and unchanged
- Access: signups open, no free credit for new accounts, API operational ("All services are online"; the status page's history lists no event after 30 Oct). The console showed the same sign-in page; signup itself was not observed. Home still shows two notices (no new-user credit; lookalike resellers).
- Models and price:
jev-latestandjev-previewgivejev-1.14.0;jev-1.13.0is listed with no deprecation date; $0.042 per million input tokens, output free. TypeSafe's models, images, quickstart, changelog and legal pages and its docs index are identical to the 8 Nov copies. - The six routes: TypeSafe, OpenRouter, Vercel, Opper, Netlify and Cloudflare list the same Jev IDs; Opper's
typesafe/jev-lateststill givesjev-1.13.0. - OpenCode's free ID (third-party route; not run by TypeSafe) is still listed as "Free" for a limited time. Lookalike sites: all three still answer, and Eye Security's report still names them.
- The option-order figures behind the determinism answer (B41) and TypeSafe's cookbook figures are unchanged.
Not checked or not read
- Amazon Bedrock's model lists (their static text has no TypeSafe or Jev string, but part of the list is rendered by script, so Bedrock was not determined; the 31 Oct reading stands). Foundry and Workers AI were not re-read.
- Artificial Analysis's methodology; the position the Jev Decision Index app shows; the other tasks of S1CogBench.
- auto-mode's mod host and its skip stages' internals; OpenCode's Privacy Policy (where "Connected Service" is defined);
nace-sdk0.7.0,@openclaw/typesafe2026.11.2 and n8n 2.47.3. - Newer benchmark repositories and coding-agent projects seen in the scan were not opened. Reddit, X, YouTube and TikTok could not be searched for the hours between the checks. No request was sent to any route or provider, no package was installed or run, and no account was created.
Sources
Sources read on 9 Nov 2026 (UTC times)
- R29-S01 to R29-S16: TypeSafe console sign-in pages, the CEO post (X embed endpoint), docs quickstart, status page, models, images, changelog, legal and
llms.txtpages, homepage, blog and Enterprise page, OpenRouter's Jev guide and the lookalike sites, 11:07–11:08. - R29-S112 to R29-S120 and R29-S38, R29-S40, R29-S44: the five benchmark repositories at their pins and the two Reported posts, 11:08–11:25.
- R29-S208 to R29-S215 and R29-S39, R29-S41, R29-S45: Jev Decision Index, Artificial Analysis Decision Index, the TypeSafe cookbook and the B41 README, 11:11–11:25.
- R29-S200 to R29-S207 and R29-S17: auto-mode on npm, its 4.0.0 and 3.1.6 sources, and
src/classify-action.tsat 7b246a7, 11:07–11:12. - R29-S18 to R29-S21: npm and PyPI records and the
@ai-sdk/typesafe-aiandaitarballs, 11:10–11:12. - R29-S22 to R29-S29: OpenCode docs, OpenCode Terms, route listings (OpenRouter, Vercel, Opper, Netlify, Cloudflare) and OpenRouter's ZDR list, 11:11.
- R29-S30 to R29-S37: Nace.AI's website Terms, Privacy Notice, Console terms and DPA, the Eevo announcement, the Drex product page and model card, and the Amazon Bedrock model pages, 11:14–11:17.