Shaduf.Research preview
Jev: Use Cases, Alternatives & Products/What changed for Jev builders, 8 to 9 November 2026
Research reportChecks , 11:07–11:33 UTC; previous checks 8 November 2026, 05:17–05:42 UTC; no check between them

What changed for Jev builders, 8 to 9 November 2026

Three published Jev benchmark studies were graded from their own files, so the benchmarks ledger now has 48 graded rows. Two leaderboards were re-read: one is unchanged apart from a longer list, the other changed its index. auto-mode 4.0.0, a Claude Code permission mod that asks Jev, was read in source: its failure path is unchanged and it stays unsettled. Nace.AI's legal pages now agree with each other; OpenCode's two texts still do not. TypeSafe's aliases, model IDs, price, signup and credit state did not change.

Key findings

Three more Jev studies are graded from their files: B49 (grade B), B50 (grade A, small sample) and B51 (grade B, the author's own harness). The ledger has 48 graded rows: A 15, B 16, C 10, D 7.

ReportedEach operator's own results; metric and results files read at pinned commits, 9 Nov 2026, 11:09–11:25 UTC (R29-S112 to R29-S115, R29-S38, R29-S40, R29-S44). Not averaged or ranked across studies

auto-mode 4.0.0 handles a Jev failure as before. Its bypass now lists 7 skip stages, including a reply-channel allow added in 3.1.x. It stays unsettled; the coding-agent counts (15 + 10 + 4 = 29) and the failure map (80) are unchanged.

DocumentedSource read at commit 7b246a7 (npm 4.0.0, published 8 Nov 2026) and compared with 3.1.6 (382b7a9), 9 Nov 2026, 11:07–11:12 UTC (R29-S200 to R29-S207, R29-S17)

Nace.AI's website and Console terms, dated 6 Nov 2026, now agree: no training on your content unless you opt in. OpenCode's docs and Terms (effective 8 Nov 2026) are still not reconciled.

DocumentedNace.AI legal pages read 9 Nov 2026, 11:14 UTC (R29-S30 to R29-S33); OpenCode docs and Terms read 11:11 UTC (R29-S22, R29-S23)

Three studies graded from their files

Five repositories published between 23 Oct and 8 Nov 2026 were opened at pinned commits. Three had a metric in code and a results file that matched the stated figure, and were graded on the benchmarks page's A–D scale; two are leads, not graded. Every figure below is the operator's own measurement, in the operator's units. The studies use different tasks and baselines, so their numbers are not comparable with each other.

Graded rows on the benchmarks ledger, by grade: 24 Oct and 9 Nov 2026

24 Oct 2026 A 14 · B 14 · C 10 · D 7 = 45
9 Nov 2026 A 15 · B 16 · C 10 · D 7 = 48
  • A rerunnable
  • B partly rerunnable, or with a stake
  • C method described
  • D claim only

Bar lengths are each grade's share of that day's graded rows. A grade says how far a study can be checked, not whether its result is right. New on 9 Nov: B50 (A); B49 and B51 (B).

The three studies graded on 9 Nov 2026 (each row is one study; figures are not comparable across rows)
Study, operator and interestTask and baselinesResult as reportedGrade and why
B49 dengzhe-hou/S1CogBench at 9cd68df
The benchmark's author; no commercial stake found
A Bayesian urn task from a multi-task cognitive benchmark: the model's probability against the Bayesian posterior (mean absolute error; lower is better), n = 400 for Jev 1.14. Baselines: Claude Opus, Sonnet and Haiku; Intern-Decision; Laya; a uniform guessJev 1.14 0.068; Claude Opus, Sonnet and Haiku 0.001; Intern-Decision 0.235; Laya 0.291; uniform guess 0.349. Only this task was readB The raw Jev responses are not in the repository and some modes need gated data
B50 GopalAlag/FrontierLLMvsJEV at 27a73b7
An individual; no stake found
IVR intent routing (exact intent), 250 items per iteration, 2 iterations. Baselines: GPT-6 Luna, Claude Sonnet 5.5, Gemini 3.1 Flash-Lite, Claude Haiku 5.5, a TF-IDF classifierAccuracy: GPT-6 Luna 0.652, Sonnet 5.5 0.644, Jev 0.636, Flash-Lite 0.620, Haiku 5.5 0.608, TF-IDF 0.304A Code, data and results published. Flags: small sample; the README's run command differs from the reported run's manifest
B51 tukka25/graph-reflex at 47859bd
The author's own harness is under test (conflict of interest)
Product search on STaRK-Amazon (public data), 100 questions, Hit@1. Baselines: a Claude Opus 5.5 agent writing Cypher, a Haiku 5.5 decider in the same harness, a cross-encoder, vector searchOpus 5.5 agent 0.69, harness with Jev 0.58, harness with Haiku 5.5 0.50, cross-encoder 0.46, vector search 0.33B Per-question outputs are not committed, and the author has a stake. Flags: small sample, public data
  • Not graded: mahirmalik030/Jev-vs-LLMs (no results file in the repository and no Jev call in its evaluation script) and randlee/judge-jev (a compliance judge built on Jev, not a benchmark). Details on benchmarks.
  • The small-LLM count is not redone. The benchmarks page's count of studies against small text-generating LLMs (ahead in 6 of 10, as counted through 24 Oct 2026) does not yet include B49 to B51, although each names a small tier; only one figure per study was matched.
  • No change to the Decisions API comparisons: none of the three compares Jev with OpenAI's Decisions API or Microsoft-Decision-1.

Leaderboards re-read

Two decision leaderboards, as read on 24 Oct and 9 Nov 2026 (listings at one time, not studies; not combined; no position given)
Leaderboard and operator24 Oct 20269 Nov 2026
Jev Decision Index
A community operator, not affiliated with TypeSafe
Version 0.4.1 (data of 22 Oct); Jev 1.14 60.24, Jev 1.13 59.22; 229 entriesVersion 0.4.1 (data of 6 Nov, commit e05cd04); Jev 1.14 60.24, Jev 1.13 59.22, unchanged; 275 entries. The position shown in the app was not re-checked
Artificial Analysis Decision Index
Artificial Analysis
Version 1.0; Jev 1.14 57.07, Jev 1.13 57.50; 32 entriesJev 1.14 66.96; Jev 1.13 no longer listed; 21 entries. The index has changed since 24 Oct (a component is now named "AA-Omniscience-Decision v1.1"); its method was not re-read, so the two readings may not be on the same scale

Reported Leaderboard data read 9 Nov 2026, 11:11–11:25 UTC (R29-S208 to R29-S210, R29-S39, R29-S215, R29-S41, R29-S45). Full section: benchmark leaderboards.

Is Jev deterministic?

Not fully, in one published option-order test: reordering the same three options changed Jev's route on 41–42 of 500 cases (the same route in all six orders 91.6–91.8% of the time), against 179 for OpenAI's Decisions API and 12–14 for Microsoft-Decision-1 (B41, grade A; the operator's own runs, README unchanged on 9 Nov 2026). Reported This is one task, synthetic bank-withdrawal routing, and it does not measure repeated identical requests.

TypeSafe's own figures. TypeSafe's cookbook on verifying LLM answers against their source reports, for jev-1.14.0 on 1,000 MuSiQue questions answered by gpt-6-luna: Jev's check passed 73% of the answers, 95.7% of them right, and sent 27% to review, including 84 of the 115 wrong answers. Reported TypeSafe sells Jev; its figures, outside the graded ledger and not checked independently (cookbook read 9 Nov 2026, unchanged since 8 Nov; R29-S211). Now also on Jev as a judge.

auto-mode 4.0.0: failure path unchanged, still unsettled

auto-mode is a community mod that asks Jev whether a Claude Code tool call may run without asking the user. Version 4.0.0 (published 8 Nov 2026) is a major release; the six files the coding-agents row cites are byte-identical to 3.1.6, and the 4.0.0 changes sit in a review step that runs only after a deny. The row was last read at 3.0.0 (21 Oct 2026).

auto-mode's six cells, 3.0.0 and 4.0.0 (terms as on the failure checklist; as read, not counted)

Cell3.0.0 (read 21 Oct 2026)4.0.0 (read 9 Nov 2026)
Host and whatClaude Code mod; answers Claude Code's permission question with allow or deny when Claude Code would askUnchanged (new line numbers)
Jev error or timeoutheld for the user's prompt (as read, not counted): 5,000 ms; Claude Code's question staysUnchanged
Malformed answerheld for the user's prompt: only an exact allow or deny is usedUnchanged
No keyheld for the user's prompt: the call throws and the question staysUnchanged
Below thresholdacts-anyway: blocks only at a block probability of 0.2 or more; below it the call runs without askingUnchanged
BypassPartial read: 6 skip stages listed; not counted as recordedPartial read: 7 skip stages, adding a reply-channel allow (a tool call that matches a reply channel in the task scope is allowed without Jev deciding); present since 3.1.x; internals not read; not counted as recorded

Grey marks an unchanged cell, amber a cell whose content changed. The row stays unsettled because the mod host's handling of auto-mode's answer is not in auto-mode's source and was not read, and the skip stages' internals were not read line by line. Cell wording as on coding agents.

Data terms: Nace.AI resolved, OpenCode not

Two data-terms texts re-read on 9 Nov 2026
ProviderBefore9 Nov 2026
Nace.AI (Drex)
a competing hosted decision API
From 9 to at least 21 Oct 2026 its website Terms (training by default, opt-out, up to 30 days) and Console terms (no training unless you opt in; deletion within 48 hours or 7 days) disagreedAll four documents dated 6 Nov 2026 say the same: no training unless you opt in; in Standard mode inputs and outputs are kept until you delete them; the text of Decision requests is kept 30 days in a separate archive for misuse screening. Changed between 21 Oct and 9 Nov 2026
OpenCode (jev-1.13-free)
third-party route; not run by TypeSafe
Docs: providers follow a zero-retention policy; Terms (effective 15 Aug 2026): content from unpaid accounts may be used to improve the servicesDocs unchanged; Terms now effective 8 Nov 2026 and add "except Content obtained or derived from a Connected Service". Whether the free route counts as one is not stated on the pages read. Still not reconciled

Also on alternatives: Nace.AI now says Drex Max is "70.8 on the Decision Index, 10.1 points over Jev 1.14" (6.9 points on 20 Oct), and Upright's Eevo announcement carries a six-model table including Jev 1.14 (both the vendors' own figures, Reported). Upright's announcement names no weights or base model (read 9 Nov 2026).

What else changed

Each change is on its page with this wording and date. Old values stay on the pages as dated history.
FactBeforeNow
@ai-sdk/typesafe-ai3.0.35 (7 Nov) noted, not comparedCompared with 3.0.34: it rejects empty Choice or Score criteria and empty Score levels before sending, sends primitive JSON state as text and returns Score legends in provider metadata; its image and error lines are not in that diff; no page cell changed
ai (AI SDK)7.0.165 (7 Nov) noted, not compared7.0.165 and 7.0.166 (9 Nov): the decide functions are unchanged from 7.0.164; telemetry and documentation changes
Integration releasesRead or noted at earlier versions@openclaw/typesafe 2026.11.2 (8 Nov), nace-sdk 0.7.0 and n8n 2.47.3 (9 Nov) are newer; not re-read
Status-page uptime figures99.942%, 99.842%, 99.883%99.941%, 99.848%, 99.882%; the third is read as the 90-day figure; no event listed after 30 Oct
OpenRouter's ZDR endpoint list1,117 endpoints1,109 endpoints; both TypeSafe endpoints listed

Re-read on 9 November and unchanged

  • Access: signups open, no free credit for new accounts, API operational ("All services are online"; the status page's history lists no event after 30 Oct). The console showed the same sign-in page; signup itself was not observed. Home still shows two notices (no new-user credit; lookalike resellers).
  • Models and price: jev-latest and jev-preview give jev-1.14.0; jev-1.13.0 is listed with no deprecation date; $0.042 per million input tokens, output free. TypeSafe's models, images, quickstart, changelog and legal pages and its docs index are identical to the 8 Nov copies.
  • The six routes: TypeSafe, OpenRouter, Vercel, Opper, Netlify and Cloudflare list the same Jev IDs; Opper's typesafe/jev-latest still gives jev-1.13.0.
  • OpenCode's free ID (third-party route; not run by TypeSafe) is still listed as "Free" for a limited time. Lookalike sites: all three still answer, and Eye Security's report still names them.
  • The option-order figures behind the determinism answer (B41) and TypeSafe's cookbook figures are unchanged.

Not checked or not read

  • Amazon Bedrock's model lists (their static text has no TypeSafe or Jev string, but part of the list is rendered by script, so Bedrock was not determined; the 31 Oct reading stands). Foundry and Workers AI were not re-read.
  • Artificial Analysis's methodology; the position the Jev Decision Index app shows; the other tasks of S1CogBench.
  • auto-mode's mod host and its skip stages' internals; OpenCode's Privacy Policy (where "Connected Service" is defined); nace-sdk 0.7.0, @openclaw/typesafe 2026.11.2 and n8n 2.47.3.
  • Newer benchmark repositories and coding-agent projects seen in the scan were not opened. Reddit, X, YouTube and TikTok could not be searched for the hours between the checks. No request was sent to any route or provider, no package was installed or run, and no account was created.

Sources

Sources read on 9 Nov 2026 (UTC times)

Search published pools, pages, reports, and evidence.