Shaduf.Research preview
EvidenceLedger checked (05:18–05:33 UTC); reranking note updated ; B37 added and leads opened, updated (05:20–05:22 UTC); B38–B40 graded and leads updated (05:22–05:26 UTC); B41 graded, short answer and leads updated (05:12–05:20 UTC); B42 graded, short answer and leads updated (05:18–05:24 UTC); B43 and B44 graded, short answer and leads updated (16:51–16:56 UTC); Decision Index line added and model-version notes updated (05:21–05:31 UTC); leaderboard section added and B45–B48 graded (05:15–05:24 UTC); Base Labs pricing line added (05:20 UTC) · model jev-1.13.0 unless a row states jev-1.14.0; B49–B51 graded, Jev Decision Index and Artificial Analysis lines re-read, option-order and TypeSafe cookbook lines added (11:08–11:33 UTC)

Jev benchmarks (TypeSafe AI): is Jev accurate? A graded ledger of published studies

In the three weeks after launch, TypeSafe, gateways, vendors, researchers and individuals published tests of Jev. They use different tasks, baselines and materials, so their headline numbers cannot be compared directly. This page grades each study by how far it can be checked, says who ran it and what they had at stake, and shows where the studies agree and where they conflict.

Why this title: Bing and DuckDuckGo complete "jev b" and "jev bank" to "jev benchmark"; Google completes "jev 1.13" to "jev 1.13 benchmarks" (rechecked ). A search suggestion is a demand signal, not a search volume.

Short answer

Key finding, 24 Oct 2026: Five grade-A comparisons of Jev with OpenAI's Decisions API differ by task; there is no overall direction (B38, B39, B41, B44, B45; a sixth, B42, is grade B). On five public classification tests run on 17 Oct 2026, during the API's public beta (B45, grade A), accuracy showed no detectable difference on three and Jev 1.14 ahead on two; Brier favoured Jev on all five. The earlier finding below gives B38, B39 and B41 task by task; B44 (grade A, 14 Oct) is a 48-game Tetris round robin. Reported results, Documented materials; B45 read .

Earlier key finding, 12 Oct 2026: Three independent grade-A comparisons of Jev with OpenAI's Decisions API differ by task; there is no overall direction. On NPC move choices in a small role-playing game (B38, grade A), the Decisions API was ahead (per-move accuracy 59.4% vs Jev 51.0%). On Portuguese routing, injection detection and judging (B39, grade A), Jev was ahead on the exact judge score (80.7% vs 72.0%) and level on routing and injection. On synthetic bank-withdrawal routing (B41, grade A), Jev (89.2% macro-F1) and Microsoft-Decision-1 (87.6%) were not separated and both were ahead of OpenAI's Decisions API (71.4%); Decision-1 kept the same route across all 6 option orders more often (97.2–97.6% vs Jev 91.6–91.8%, Decisions API 64.2%). Reported results, Documented materials; B38 and B39 checked , B41 checked .

A fourth graded comparison with OpenAI's Decisions API, at a lower grade, sits below these three: On 1,000 five-option logic questions (B42, grade B), Jev answered 817 correctly and OpenAI's Decisions API 612; Jev's answers come from a 22 Sep run, and the answer key is not published. Reported; B42 pin rechecked . Compare the three hosted decision APIs side by side: Decisions API vs Jev vs Microsoft Decision 1.

Published tests disagree because they test different tasks against different baselines. Against small text-generating LLMs, Jev scored ahead in 6 of 10 graded studies, mixed in 2 and not ahead in 2; 5 of the 10 are grade C. (Studies whose ledger row names a small tier: mini, nano, Haiku, Flash or Flash-Lite. Ahead: B01, B03, B05, B15, B23, B35; mixed: B31, B34; not ahead: B06, B33. Directions as the operators reported them; counted 12 Oct 2026 and unchanged through 24 Oct 2026. The three studies graded on 9 Nov 2026, B49 to B51, each name a small tier and are not yet in this count.) One study found open-weight models scored through their token probabilities about level with it overall (B35, grade B). It is close to frontier LLM judges on verdicts that can be read off the text once a threshold is tuned. It is behind the best LLMs on harder or derived judgments (math, code, logic) and on 14 of 15 social-science annotation tasks. Against trained classifiers on fixed label sets the evidence is mixed: in one Red Hat guardrail study, Jev was behind a prompt-injection classifier and ahead of a small content-safety classifier (B28, grade B; Red Hat publishes both classifiers); the rest is grade D.

Added 10 Oct 2026: one reranking-pipeline study, B37 (jev-pulse, grade A), joined the ledger, which then had 34 graded rows. Three comparisons opened that day at README level were listed as leads.

Added 11 Oct 2026: three comparisons first listed as leads on 10 Oct are now graded from their metric and results files at pinned commits (B38, B39, B40, all grade A). The ledger then had 37 graded rows: A 10, B 12, C 8, D 7. Two compared Jev with OpenAI's Decisions API and pointed in different directions by task; the third is a reranking-pipeline study (B40) in which Jev and Cohere Rerank 4 Pro are within noise. classifier-bakeoff stays a lead, not graded (report opened, metric code not opened).

Added 12 Oct 2026: one comparison first listed as a lead on 11 Oct is now graded from its metric, results and data files at a pinned commit (B41, grade A, with four flags). The ledger then had 38 graded rows: A 11, B 12, C 8, D 7. Graded comparisons with OpenAI's Decisions API were then 3 (B38, B39, B41); with Microsoft-Decision-1: 1 (B41). Two README comparisons from real-time game and robot-simulation demos are listed as leads, not graded.

Added 13 Oct 2026: one comparison first listed as a lead on 11 Oct is now graded from its metric and results files at a pinned commit (B42, grade B, with seven flags). The ledger then had 39 graded rows: A 11, B 13, C 8, D 7. Graded comparisons with OpenAI's Decisions API were then 4 (B38, B39, B41, B42); with Microsoft-Decision-1: 1 (B41). utakatano/decision-models-demo is a lead, not graded (empty repository), and ten comparison repositories found on 13 Oct are listed as leads, not opened.

Added 14 Oct 2026: two comparisons first listed as leads on 13 Oct are now graded from their scoring or metric code and results files at pinned commits. In a 48-game Tetris round robin, Microsoft-Decision-1 won 15, Jev 12, OpenAI's Decisions API 11 and the open Laya model 8; Jev beat the Decisions API 6–2 head to head (B44, grade A, small sample, one game). On 200 SWE-bench Verified issues, SubQ Snap and Jev picked the file to change about equally often (78.5% and 79.0%); asked per file, Snap was ahead but not significantly (B43, grade B; the operator takes part in Subquadratic's prize challenge). The ledger then had 41 graded rows: A 12, B 14, C 8, D 7. Graded comparisons with OpenAI's Decisions API were then 5 (B38, B39, B41, B42, B44); with Microsoft-Decision-1: 2 (B41, B44); with SubQ Snap: 1 (B43). The 12 Oct key finding covered B38, B39 and B41 and was unchanged that day. Reported; files read . SubQ Snap's published facts: SubQ Snap on Jev alternatives.

Added 24 Oct 2026: four studies are now graded. Two were read in their metric and results files at pinned commits: B45 (grade A; Jev 1.14 vs OpenAI's Decisions API, with Claude Opus 5.5 on a sample) and B46 (grade A; Jev 1.13 vs SubQ Snap on a synthetic long-log task, where Jev was ahead at 360K tokens with the log split into chunks). Two are vendor blog posts without materials: B47 (kapa.ai, grade C, reranking-pipeline study) and B48 (Sayora, grade C, relevance-judgment study). The ledger then had 45 graded rows: A 14, B 14, C 10, D 7. Graded comparisons with OpenAI's Decisions API: 6 (B38, B39, B41, B42, B44, B45); with Microsoft-Decision-1: 2 (B41, B44); with SubQ Snap: 2 (B43, B46). egarim/decision-1-vs-jev stays a lead: it has no Jev arm. Four leaderboards that list Jev are described below; they sit outside the ledger. The small-LLM count (6 of 10) is unchanged. Reported; files read .

Added 9 Nov 2026: three studies published between 23 Oct and 8 Nov 2026 are graded from their metric and results files at pinned commits: B49 (S1CogBench, a multi-task cognitive benchmark by its author; grade B; one task matched), B50 (FrontierLLMvsJEV, IVR intent routing against four LLMs and a TF-IDF classifier; grade A, small sample) and B51 (graph-reflex, product search over a knowledge graph; grade B; the author's own harness is under test). The ledger has 48 graded rows: A 15, B 16, C 10, D 7. None of the three compares Jev with OpenAI's Decisions API or Microsoft-Decision-1, so the comparisons above are unchanged. Two more repositories were opened and are leads, not graded, each with its reason.

This site has run no benchmark of its own and made no Jev call. Every result below is Reported: the named operator's own measurement. The site does not average results across studies and does not rank studies by headline score.

Jev benchmark leaderboards (TypeSafe AI): who runs them and what you can check

Key finding, 24 Oct 2026

Four decision leaderboards read on 24 Oct 2026 list Jev. Base Labs' "#1 of 208" claim matches the Jev Decision Index's 20 Oct data (version 0.4) but not its current 22 Oct data (0.4.1).

Reportedleaderboard data files and pages read ; the match result is this site's sort of the index's own published data file at two commits

Update, 9 Nov 2026: the Jev Decision Index is still version 0.4.1 with Jev 1.14 at 60.24 and Jev 1.13 at 59.22 (data of 6 Nov 2026; 275 entries); Artificial Analysis now lists Jev 1.14 at 66.96 and no longer lists Jev 1.13 (21 entries; its index has changed since 24 Oct). Positions were not re-checked. Reported

How to read this section. Leaderboards are listings at one time, not studies: they sit outside the ledger, carry no grade and do not change the ledger counts. Positions below are as listed in each operator's own data at the version shown; they are not combined, and a position is not an accuracy result. This site submitted nothing and ran no benchmark.

What you can check on each leaderboard, as read on 24 Oct 2026 (UTC); the Jev Decision Index and Artificial Analysis rows re-read 9 Nov 2026
Leaderboard, operator and stakeVersion readHow scores get thereWhat is publishedJev as listed
Jev Decision Index (Hugging Face Space multimodalart/jev-decision-index)
A community account: "Unofficial and community-maintained; not affiliated with TypeSafe AI." No model of the operator's own found in the data file.
0.4.1, Space commit e05cd04 (6 Nov 2026, 09:10 UTC; data generated 09:05 UTC), read 9 Nov 2026, 11:11–11:18 UTC. On 24 Oct 2026: commit 0b5ecaf (22 Oct 2026, 20:12 UTC).Run by the operator; author-submitted checkpoints are adopted only after the operator reproduces them. No "verified" row marker found.Per-model scores for a public suite of 37 benchmarks (5 areas, 110,201 requests). The headline "Full score" is 0.60 × private tests of the same skills + 0.40 × private tasks from new domains: "The private tests and the new-domain tasks are never published." Private item counts not found.Jev 1.14 (jev-1.14.0, "Lab run, 2026-10-14"): Full score 60.24; Jev 1.13 (jev-1.13.0): 59.22; both unchanged since 22 Oct. The board now lists 275 entries (229 on 22 Oct). The position shown in the app was not re-checked on 9 Nov, so no position is given here. Change, 9 Nov 2026: the positions read on 24 Oct from the 22 Oct data are no longer shown, because the list has grown since.
Artificial Analysis Decision Index
Artificial Analysis. No model of its own among the 32 entries read on 24 Oct.
Changed since 24 Oct: on 9 Nov 2026 (11:12–11:25 UTC) a component is named "AA-Omniscience-Decision v1.1"; the methodology was not re-read. On 24 Oct 2026: version 1.0 ("October 2026 to current"); pages read 05:16–05:17 UTC (no commit or page history).Run by the operator (its methodology, read 16 Oct); no "verified" marker found in the page data.Per-model scores and sub-scores for four evaluations (AA-Omniscience-Decision, HLE-MultiChoice, When2Call, WildGuardTest). Per-item results not found on the methodology, article or evaluation pages. AA-Omniscience-Decision uses 500 private questions (read 16 Oct).9 Nov 2026: Jev 1.14 66.96; Jev 1.13 is no longer listed; the page data holds 21 entries. The index has changed since 24 Oct and its method was not re-read, so the two readings may not be on the same scale; no position is given. On 24 Oct 2026: Jev 1.14 (host TypeSafe; model ID not shown) 57.07, 17 of 32 entries as listed (5 of 17 decision models); Jev 1.13 57.50 (16 of 32).
JevMan (Pac-Man game benchmark)
Opper, which serves Jev and the other listed models (a route provider's stake). No Opper-made model listed.
public/leaderboard.json generated 21 Oct 2026, at repository head fd465f0 (23 Oct)Both: "Run by @opper-ai means we played the games through Opper. Community models were benchmarked by whoever submitted them ... CI replayed every game they sent in to check its score".Per-model aggregates (mean score, survival, pellets, latency, fallback rate, cost per game) for 100 real-time games per model against scripted ghosts. Per-game files not opened.jev 1.14 (typesafe/jev-1.14.0): mean score 3597 (SE 148), 1 of 12 as listed; next Eevo-1.1 3416 (SE 147). The page's margin-of-error tie rule was not applied. jev 1.13: 2750, 5 of 12.
JevBench
Leanmcp (GitHub repository, Hugging Face dataset and a preprint). No Leanmcp-named model among the 18 rows.
Data "generated" 16 Oct 2026; site read 05:18 UTC; not pinned to a commit (repository not opened)Not stated in the text read; the site has a "Submit" section. No "verified" marker.Per model and per dataset, with intervals and n, for 25 public datasets (17,081 cases). No private set found in the data file.Jev 1.13 (run of 26 Sep 2026): index 76.5, 3 of 18 as listed. No Jev 1.14 row in the 16 Oct data.

Positions: each operator's rows sorted by its headline score, highest first, counting every listed entry (ties stated where the page shows them). JevMan and JevBench are different leaderboards; the Hacker News item that mentions "Jevman" (50093003) is the Eevo launch post. Sources: Jev Decision Index R23-S100 to R23-S114 (data files at 0b5ecaf and 133313c re-sorted at 05:23 UTC, R23-S48); Artificial Analysis R23-S119, R23-S120, R23-S123; JevMan R23-S126, R23-S129 to R23-S133; JevBench R23-S128, R23-S134. Re-read 9 Nov 2026: Jev Decision Index R29-S208 to R29-S210 and R29-S39 (data file); Artificial Analysis R29-S215, R29-S41, R29-S45.

Base Labs' "#1 of 208" claim for mammoth-1, checked against two versions of the Jev Decision Index data

  1. · version 0.4 · commit 133313cmammoth-1 71.24, first of 209 entries208 model rows plus the Jev 1.14 row; next 69.42. The claim matches this version.
  2. · version 0.4.1 · commit 0b5ecafmammoth-1 68.44, 3 of 229 (shown "=3")After rescoring on current weights. The claim does not match the current version.

Both readings sort the index's own data file (data/index.json) at each commit; the Space's app was read as text and not run. The claim was published on 21 Oct 2026 (Baseten blog, Base Labs report; R23-S115, R23-S116, R23-S109, R23-S102). Reported

Vendor and press claims, checked against each leaderboard's own data

  • matches at both commits Base Labs: elephant-1 is "beating Jev 1.14" on the Jev Decision Index: 62.65 vs 62.22 at 133313c and 60.26 vs 60.24 at 0b5ecaf, where both sit in the same tie band "=10" (R23-S116).
  • matches at 21 Oct, not later Jeffan-31b-1710 model card (21 Oct): "#1 on the Jev Decision Index": shown "=1", tied with mammoth-1, at 16f312e and 1f4e490 (21 Oct, 15:13–20:39 UTC); not from 76fbe37 (21 Oct, 22:37 UTC); at 0b5ecaf it is 2 of 229, behind Fastino GLiDE 1.2 (71.52) (R23-S118, R23-S111 to R23-S113).
  • does not match Baseten: mammoth-1 scores 65.9 and elephant-1 65.2 on the Artificial Analysis index: at 24 Oct, 05:16 UTC no mammoth entry is listed and "Elephant 9B" shows 64.69; an earlier state cannot be checked (no page history) (R23-S115, R23-S123).
  • matches only when rounded Base Labs: elephant-1 "tied with" d1-ultra on the Artificial Analysis index: 64.69 and 65.25, both 65 as whole numbers; the page's display rounding was not checked (R23-S116, R23-S123).
  • does not match the live data Artificial Analysis article (13 Oct): "Jev 1.13 leads the Decision Index with 58": at 05:16 UTC d1-ultra (65.25), SubQ Snap 1.1 (65.00) and Elephant 9B (64.69) are listed above Jev 1.13 (57.50); the article text is unchanged (R23-S120, R23-S123).
  • matches against 1.13, not 1.14 Upright (Hacker News 50093003, 14 Oct): Eevo beats Jev "even Pacman (Jevman benchmark)": in the 21 Oct JevMan file Eevo scores 3004 and 3416 against jev 1.13's 2750 and below jev 1.14's 3597; the 14 Oct state was not read (R23-S125, R23-S133).
  • not checkable Decido (Hacker News 50199105, 22 Oct): a six-group mean of 85.38 for Decido 26B-A4B against 83.85 for "Jev 1.13 (published)" is Decido's own average of Jev 1.13's reference runs on six third-party benchmark groups, not a figure TypeSafe published; its results files were not opened (R23-S127, R23-S138, R23-S140).
  • not checkable Basal-Omni's JevBench claim (seen 19 Oct): the source was not located; "JevBench" was not found in the basal-omni-1.0-ultra model card at 6d5ba67 (24 Oct, 05:18 UTC; R23-S137).
  • not checked Baseten: elephant-1 "tied for #8 overall" (the tie group at 133313c was not computed); a Hacker News comment (50213804, 23 Oct) that Jev "handily beats Luna-decisions on the Artificial Analysis Decision Index on every metric".

Base Labs (Baseten) released mammoth-1 and elephant-1 on 21 Oct 2026 and cite the Jev Decision Index and the Artificial Analysis index; their rank claims are Reported and checked above. They are not listed on the alternatives page yet. Reported (R22-S48, R22-S49, R23-S115, R23-S116) Change, : Base Labs' mammoth-1, elephant-1 and joey-1 are now one hosted-API row on the alternatives page, with prices, data terms and the cards' rank claims (Reported; not matched against the board on 7 Nov 2026, when its data returned HTTP 500; R27-S200 to R27-S220). The 24 Oct checks above keep their date.

: Baseten's 21 Oct post said the models were free to try for a week on Baseten Model APIs (read 23 Oct 2026; R22-S48). That free-week wording was not found on Baseten's pricing page or its mammoth-1 model page on 30 Oct 2026, 05:20 UTC; the 21 Oct post was not re-read. The pricing page lists, per 1M input tokens: joey-1 $0.01, elephant-1 $0.035, mammoth-1 $0.10; the mammoth-1 page lists output at $0.00 (R24-S223, R24-S224). Listed prices only, not a tested request; the claim checks above keep their 24 Oct date. Documented

Versions, history and method notes for each leaderboard
Jev Decision Index history
50 commits from 27 Sep to 22 Oct 2026 on the Space's API page: 0.2.1 (27 Sep–2 Oct), 0.3 (6 Oct), 0.3.1 and 0.3.2 (12–16 Oct), 0.4 (20 Oct), 0.4.1 (from 21 Oct). The methodology changelog lists v0.2 dated 27 Sep 2026, so the "Decision Index 0.2" that B18 (Rene-1) cites is an earlier version of this index; the 0.2 data was not read. The data comes from the Space's own data/*.json files (no separate dataset repository) (R23-S101, R23-S103, R23-S107).
How the positions were counted
Jev Decision Index: the models rows of data/index.json plus its separate Jev row, by the headline score, highest first. Artificial Analysis: the index score in the page data. JevMan: mean score, 12 entries. JevBench: index, 18 entries. Each count is one listing at one version or time.
Artificial Analysis Decision Index: earlier reading (16 Oct 2026)
Artificial Analysis Decision Index v1.0 (13 Oct 2026; one of four evaluations uses 500 private questions; the model ID or route behind "Jev 1.13" is not stated): Jev 1.13 58, GLiDE 54, Microsoft-Decision-1 54. Read 16 Oct 2026, 05:31 UTC; the index was published before Jev 1.14 was released on 14 Oct, so that "Jev 1.13" cannot refer to 1.14. No per-item results file was found on the pages read. It sits outside the ledger and has no grade. Reported article (R17-S60), methodology page (R17-S61)
Not verified
The Jev Decision Index 0.2 data (the figure B18 quotes); private item counts on the Jev Decision Index; Artificial Analysis display rounding and page history; JevMan's tie display; a JevBench commit pin. No score was submitted and nothing was run.

Change, 7 Nov 2026: the leaderboard section's Base Labs paragraph links the new Base Labs row on the alternatives page (added 7 Nov 2026). Ledger rows, grades, counts and leaderboard positions unchanged.

Change, 30 Oct 2026: under the leaderboard section, a dated line on Base Labs: the free-week wording was not found on Baseten's pricing and mammoth-1 pages, which list joey-1 $0.01, elephant-1 $0.035 and mammoth-1 $0.10 per 1M input tokens (30 Oct 2026, 05:20 UTC; R24-S223, R24-S224). Ledger rows, grades, counts and leaderboard positions unchanged.

Change, 9 Nov 2026: B49, B50 and B51 added (grades B, A, B); the ledger went from 45 to 48 graded rows (A 14 → 15, B 14 → 16). The small-LLM count in the short answer is kept as counted through 24 Oct and says the three are not yet in it. The Jev Decision Index row no longer gives a position (its list grew from 229 to 275 entries); the Artificial Analysis row gives the 9 Nov figure, with the 24 Oct reading kept as dated. New: leads opened 9 Nov, Is Jev deterministic? and TypeSafe's cookbook figures outside the ledger. The key findings above are unchanged: none of the new rows compares a hosted decision API.

Change, 24 Oct 2026: a leaderboard section replaces the single Artificial Analysis line (its 16 Oct reading stays as a dated note); leaderboards sit outside the ledger. B45 and B46 (grade A) and B47 and B48 (grade C) added; the ledger went from 41 to 45 graded rows (A 12 → 14, C 8 → 10). Graded comparisons with OpenAI's Decisions API went from 5 to 6 and with SubQ Snap from 1 to 2. The key finding is reopened as a dated 24 Oct sentence; the 12 Oct form stays below it. The small-LLM count (6 of 10) is unchanged: no new row compares a small-tier LLM (B48 names Claude Haiku 4.5 only as the tool that wrote page descriptions).

Change, 16 Oct 2026: the Artificial Analysis Decision Index is listed as a Reported line outside the ledger (now a dated note in the leaderboard section); ledger counts unchanged. Under what nobody has measured yet, the image and model-version lines now reflect Jev 1.14 (preview since 14 Oct 2026) and the opt-in image input; the earlier lines said "only jev-1.13.0 exists so far" and "Jev accepts text only".

Change, 14 Oct 2026: B43 (grade B) and B44 (grade A) added; the ledger went from 39 to 41 graded rows (A 11 → 12, B 13 → 14). Graded comparisons with OpenAI's Decisions API went from 4 to 5 and with Microsoft-Decision-1 from 1 to 2; B43 is the first graded comparison with SubQ Snap. kprzybys/snap-vs-jev-swe and dante01yoon/decision-model-arena moved from the 13 Oct lead list to the ledger; the other eight 13 Oct comparison leads and JevBench's repository now show pinned commits. The key finding is unchanged. The small-LLM count (6 of 10) is unchanged: neither B43 nor B44 names a small-tier LLM.

Change, 13 Oct 2026: B42 added (grade B); the ledger went from 38 to 39 graded rows (B 12 → 13), and graded comparisons with OpenAI's Decisions API from 3 to 4. The key finding is unchanged; B42's sentence follows it. etsabary/luna-decisions-benchmark moved from the 11 Oct lead list to the ledger. The small-LLM count (6 of 10) is unchanged: B42 is not a small-LLM comparison.

Change, 12 Oct 2026: B41 added (grade A); the ledger went from 37 to 38 graded rows (A 10 → 11). The key finding now gives the three Decisions API comparisons task by task. The short-answer sentence on small text-generating LLMs now gives a count (ahead in 6 of 10 graded studies, mixed in 2, not ahead in 2) instead of a general verdict.

Is Jev deterministic?

Not fully, in one published option-order test: reordering the same three options changed Jev's route on 41–42 of 500 cases (the same route in all six orders 91.6–91.8% of the time), against 179 for OpenAI's Decisions API and 12–14 for Microsoft-Decision-1 (B41, grade A; the operator's own runs). Reported B41 README at its pin 1ffa4d5, re-read 9 Nov 2026, 11:11 UTC, unchanged; no later commit (R29-S212 to R29-S214). One test, one task (synthetic bank-withdrawal routing); it does not say how Jev behaves on other tasks or on repeated identical requests.

Before you quote "68%"

The figure quoted as "68% on TypeSafe's own benchmark" is TypeSafe's workflow benchmark (evals.typesafe.ai), where Jev scores 67.8%. That number is agreement with reference answers averaged from GPT-6 Astra and Claude Fable 5.1 on four workflows built by TypeSafe's own team. It is not accuracy against human labels or ground truth. GPT-5.6 workflows on the same test score 66.8–74.1%. See row B01. Reported

Can you publish Jev benchmark results? What TypeSafe's terms say

Current texts: no clause found (rechecked ). TypeSafe's current Terms of Use (last updated 19 Sep 2026), Acceptable Use Policy (23 Sep 2026), Master Customer Agreement, or MCA (23 Sep 2026), Data Processing Addendum (24 Apr 2026) and Privacy Policy (19 Nov 2025) contain no "benchmark", "performance information" or "comparison" clause. A search of each text for those words found no match; the only near hit is "comparable proceeding" in the MCA's insolvency clause, which is unrelated.

Docs legal page. docs.typesafe.ai/legal.md holds no terms text. Compared with the Internet Archive copy of 22 Sep 2026, it changed in two places only: the contact mailbox for enterprise zero data retention (ZDR) requests moved from TypeSafe's privacy address to its sales address, and a Mintlify footer line was added. The ZDR offer itself is unchanged. Data terms by route: data privacy.

Until 19 Sep 2026 the MCA had one. The MCA version marked "Last updated Aug 27, 2026", as archived on 16 Sep 2026 (21:55 UTC), listed among the customer's licence restrictions in section 2.3: "(f) publish benchmarks or performance information about the Services". The next archived version, dated 19 Sep 2026, no longer has it; nor do later captures or the live version dated 23 Sep. TypeSafe's docs call the MCA "the general terms that apply to your TypeSafe account".

What this means for the ledger. Studies published between Jev's launch on 15 Sep and 19 Sep appeared while, as far as the archive shows, the MCA contained 2.3(f): Parallel (B04, 18 Sep) and the first Jevals release (B06, 18 Sep). LiteLLM ran its test on 18 Sep and published on 20 Sep. TypeSafe itself now publishes a benchmark and code "to reproduce the results" (B01). This site draws no conclusion about any operator's compliance.

Still in the MCA (23 Sep version, quoted, not interpreted): 2.3(b), no use of the Services or Output "to perform model distillation, train a model to imitate the output of the Services, or develop (or to facilitate the development of) a similar or competing product or service"; 2.3(c), no reverse engineering; 2.3(g), no circumventing access restrictions "or conduct any security or vulnerability test".

Documented Read 30 Sep 2026, 20:48–20:51 UTC; rechecked 7 Oct 2026, 05:19 UTC (R10-S37, R10-S38; legal.md archived 22 Sep): Terms of Use, AUP, MCA, DPA, Privacy Policy, MCA archived 16 Sep (contains 2.3(f)), MCA archived 20 Sep (does not). This is a reading of the text, not legal advice.

Other claims about the terms, and what was checked
  • A Hacker News comment of 18 Sep (item 49753778) said the Terms of Use (1(v)) and the MCA (2.3(f)) both prohibited publishing "benchmarks or performance information about the Services". The MCA part matches the 27 Aug text. The Terms-of-Use part is not confirmed: the archived Terms of 15 Sep (dated 14 Sep) have no such clause. Reported
  • A Hacker News question of 23 Sep, "Is benchmarking Jev still a ToS violation?" (item 49813141): by the texts above, the clause was removed in the MCA version of 19 Sep.
  • OpenRouter's terms make users agree to each model's terms; its Jev page points to TypeSafe's Terms of Use, not the MCA. Cloudflare's model page links docs.typesafe.ai/legal.md. Vercel was not checked.
  • jevcal's statement that the terms restrict public testing was not re-read.

How each study is graded

A Rerunnable
Data, code and configuration published; the operator has no commercial stake in the result. "Provisional" means the linked code repository exists but this site did not read it.
B Partly rerunnable
Data public but code or prompts missing, or everything published but the operator has a stake (vendor, gateway, alternative-model author).
C Method described
Task, sample size and baseline stated; no materials published.
D Claim only
A headline number without task, sample size or baseline, or a secondary summary whose primary study is not identified.

Flags: COI the operator has a stake in the result · LLM-labels reference answers come from LLMs, not people or the source · small-n fewer than 200 items · no-baseline no LLM or classifier compared · sealed part of the test set is withheld · provisional code repository not read · public-data the data may be in training sets · single-task.

48 graded rows by grade, 9 Nov 2026 (select a row to jump to it)

Grade index, 9 Nov 2026: A 15 + B 16 + C 10 + D 7 = 48 graded rows (45 on 24 Oct; B49–B51 added 9 Nov; 41 on 14 Oct; B45–B48 added 24 Oct; 39 on 13 Oct; B43 and B44 added 14 Oct; 38 on 12 Oct; B42 added 13 Oct; 37 on 11 Oct; B41 added 12 Oct; 34 on 10 Oct; B38–B40 added 11 Oct; 33 on 7 Oct; B37 added 10 Oct; 24 before the 7 Oct check, which added B28–B36). Bars show each grade's share of the 48. Blue-bordered rows compare Jev with an LLM or a trained classifier. Grades describe how far a study can be checked, not whether its result is right. B14 moved from A (provisional) to B on 7 Oct.

Not graded: B22 (a secondary review), B25 and B27 (leads), and three items found on 7 Oct (see leads): jev-safety-eval has no results yet; jev-zh-tw-eval is not a Jev study (it tests Plumb-4B); the DAIR.AI piece is a tutorial, not a study. Of the eight repositories opened at README level on 10 Oct, three were graded on 11 Oct (B38–B40) and five stay leads, not graded; of the ten comparison repositories found on 11 Oct (leads), two were graded (B41 on 12 Oct, B42 on 13 Oct) and eight stay leads, not graded; of the ten comparison repositories found on 13 Oct (leads), two were graded on 14 Oct (B43, B44) and eight stay leads, not graded; utakatano/decision-models-demo (lead) is an empty repository; egarim/decision-1-vs-jev, read 24 Oct, has no Jev arm and stays a lead. Leaderboards (above) are not graded and are not in the count. The reranking studies in Reranking studies without a B number are graded for that question only and are not in the 45; B14, B37, B40, B47 and B48 are ledger rows and are counted once.

Tier 1: studies read at their primary source

Reported Results in each operator's own units. "Materials" lists data / code / prompts / raw outputs. B01–B14 checked 30 Sep 2026, B28–B36 checked 7 Oct 2026, B38–B40 checked 11 Oct 2026, B41 checked 12 Oct 2026 and B42 read 12 Oct 2026 (pin rechecked 13 Oct 2026), B43 and B44 read 14 Oct 2026, B45 and B46 read 24 Oct 2026, B49–B51 read 9 Nov 2026, at the UTC time shown; B14 regraded 7 Oct 2026. Model jev-1.13.0 unless stated (B45 and B49 call jev-1.14.0; B47 reports 1.13 and 1.14). Since 24 Oct 2026, jev-latest on TypeSafe's API gives jev-1.14.0; rows that called jev-latest earlier ran on jev-1.13.0. Operators are named as organisations, sites or repositories.

Study, operator and interestTask, n and baselineResult as reportedMaterialsGrade and flags
B01 TypeSafe workflow evals, code WorkflowEvals (29 Sep)
TypeSafe, the vendor. Its launch post says the workflows "were made by individuals on our model capabilities team, so some bias could exist".
Four business workflows split into Choice, Score and Noul questions: invoices (150 cases), customer service (204), agent traces (111), security incidents (240).
Baselines: Claude Haiku 4.5, Opus 5, Sonnet 5; DeepSeek V4; GPT-5.6 Luna, Sol, Terra. Reference answers: average of GPT-6 Astra and Claude Fable 5.1.
Mean agreement with the LLM reference answers: Jev 67.8% at $0.0004 and 0.4 s per case; Sol workflow 74.1%, Opus 5 73.1%, Terra 67.9%, Sonnet 5 67.8%, Luna 66.8%, Haiku 4.5 53.6%. Not accuracy against human labels. The site's arithmetic cannot reproduce the homepage's "193.6x Faster, 444.6x Cheaper" from the rounded figures.
Customer-service workflow (204 cases): Jev 76.0% (read 1 Oct 2026); how real ticket-triage projects handle Jev's failures: support-ticket triage.
Data yes; code yes (Apache-2.0, commit 0ac3b8a); prompts yes; outputs partly (on Hugging Face, not opened)B COILLM-labels
20:48–20:49 UTC
B02 Arize, "Jev vs LLM-as-a-Judge" (23 Sep)
Arize AI, an AI observability and evaluation vendor; no Jev stake stated.
Hallucination detection (RAGTruth, 2,675 responses) and summary quality (SummEval, 1,700 summaries). Human labels. 23,325 judgments in total.
Baselines: Claude Opus 5 and GPT-5.6 Terra at high reasoning effort.
RAGTruth at a 0.5 cutoff: Opus 5 83%, Jev 76%. With cutoffs tuned on half the data: Jev 87%, Opus 5 87%, Terra 80%; "the intervals overlap". SummEval rank correlation: Jev 0.74, Opus 5 0.70. $0.05 vs $14.30 per 1,000 judgments.Data public; no study code; prompts partly; outputs noB public-data
20:53 UTC
B03 NavyaAI, "We Benchmarked Jev Directly" (21 Sep)
An AI cost consultancy; "TypeSafe gave us API access".
AG News, 4 topics, n = 200 (sample not identified). English.
Baselines: gpt-4.1-nano with one constrained token; gpt-4.1-nano and gpt-4o-mini in JSON mode; GPT-5.6 Luna.
Jev 87.5–88.0%; nano with one token 71.0%; small LLMs in JSON mode 82–83%; Luna 82%. Per million decisions: Jev $18.57, nano one-token $16.30 (cost detail on cost per decision).Data public, sample not identified; code and prompts partly; outputs noC public-datasingle-task
20:53 UTC
B04 Parallel, "Testing out Jev" (18 Sep, 21:07 UTC)
A web-search API company that runs its own rerankers and classifiers.
Search reranking, topic classification, query freshness. Private data; n not stated.
Baseline: Parallel's internal models (not named).
Reranking "NDCG@10 of 0.7: comparable to internal system"; topic and freshness: "Internal 'wins'". "Specialized classifiers will still often outperform Jev on cost and speed."NoneD no n
20:53 UTC
B05 LiteLLM, "JEV Classifier: 5.43x as Fast as Haiku" (20 Sep; run 18 Sep)
An AI gateway that ships Jev as an Auto Router option. What LiteLLM's router and five others do when Jev fails, read in source: model routing.
Routing 80 synthetic prompts into three tiers; prompts and labels by the same author. 240 calls per model.
Baseline: Claude Haiku 4.5, same rubric.
Match with the authored tiers: Jev 95.00% (228/240), Haiku 73.75% (177/240). Median latency 127 ms vs 688 ms; 96% lower registry-priced cost. The site's arithmetic reproduces all four ratios.Cases and labels not published; no benchmark script linkedC COIsmall-nsingle-task
20:53 UTC
B06 Jevals, release 2026-09-18
One unnamed maintainer; states no affiliation with TypeSafe, gateways or labs and pays list price.
PubMedQA (yes/no), Banking77 (77 intents), HelpSteer2 (5-level score). Human labels; 300 items × 5 runs per task. Via Vercel AI Gateway, ai@7.0.106.
Baselines: Gemini 3.8 Flash, GLM-5.3, DeepSeek V4.1 Flash and others; a label-prior baseline.
PubMedQA: "tied" with Gemini 3.8 Flash (Decision Score 69.0 vs 73.0; 95% range of the difference −2.5 to +10.4); accuracy 91.3%; calibration error 5.0 points; $0.029 vs $0.80 per 1,000. Banking77: Gemini "significantly ahead". HelpSteer2: "no model clearly beats answering with the label base rates".Data public; per-decision logs yes (CC-BY-4.0); suites yes; harness code not foundB public-data
20:55–20:58 UTC · one row recomputed
B07 JevBench v1.5.4 (undated)
Benchmark Heaven, a benchmark site that also sells custom evaluations; no Jev stake stated.
"State and a bounded rubric in, a typed answer out"; 1,624 decisions per system, 720 of them sealed.
Ranking limited to "Jev-class" systems (at most 2× Jev's cost and latency); no LLM in it.
Jev 1.13.0 first by "Capability" 80.0 (Intelligence 72.0, Calibration 88.0), $0.032 per 1,000, median 0.62 s; Winnow-12B Q8 79.3; Cygnet 79.0.
Laya and Kev research-preview rows: Jev vs Laya vs Kev.
Code repository linked, not opened; half the items sealed; aggregate results only for the sealed halfB sealedno-baseline
20:53 UTC
B08 arXiv 2609.26550 v3, "JEV-as-a-Judge" (v3 29 Sep)
Carnegie Mellon University; thanks OpenAI's Researcher Access Program for API credits.
Judging: 5,172 base judgments (RewardBench, JudgeBench, HaluEval, final-answer checks); blinded human adjudication of disputes.
Baselines: 16 judges, including GPT-6 Astra and two reward models.
"within three points of GPT-6 wherever a verdict can be read off the text, at 0.36% of its fee"; behind "where the verdict must be derived, as in math, code, and logic"; 12 points behind on JudgeBench. A cascade with a threshold frozen in advance escalates 31% of 1,610 pairs and is 0.9 points more accurate than GPT-6 at 41% of its fee.Data public; reproducibility package described, location not foundB public-data
21:00 UTC · v3 replaced earlier results
B09 arXiv 2609.26758 v2, "Type-Safe Is Not Error-Free"
Academic (affiliation not recorded).
Robustness: swap which option name (0/1, no/yes, A/B, random strings) is bound to which rubric. 1,200 yes/no questions from LocalLLaMA/typed-decisions.
Models: hosted Jev, Laya, Open-Jev; no LLM or classifier.
No/yes names flip 32.5% of Jev's decisions, 30.4 points more than 0/1; AUC falls from .8146 to .5806. Type-error rate 0% throughout. Random-string names return to the neutral regime.
Laya: 76.9% flipped with yes/no names; see Jev vs Laya vs Kev.
Data public; code not linkedB public-datano-baseline
21:00 UTC
B10 MindStudio, "Jev vs BERT and Zero-Shot NLI" (20 Sep)
An AI agent platform's marketing blog. A secondary summary of an unnamed "recent independent benchmark".
Banking77, Yelp stars, an emotion set and a phishing set; n not given.
Baselines: a 22M-parameter encoder with a trained head; zero-shot NLI.
Banking77: trained encoder 93.2% (8 ms on CPU), Jev 80.1% zero-shot, NLI 48.8% and 66.7%. Yelp: Jev 67.2%, trained classifier 51.9%.None; primary study not identifiedD secondary
20:53 UTC
B11 Opper, "Jev vs. Kev" (25 Sep; results 24 Sep)
A gateway that serves Jev and hosts Kev.
Fresh items published after 20 Sep, ground truth from the source: arXiv category (n = 160), Stack Exchange site (120), GitHub bug or feature (82).
Baseline: Kev 4B, an open Jev-like model (not an LLM or classifier).
Accuracy Jev 96.9 / 97.5 / 95.1% vs Kev 95.0 / 98.3 / 93.9%; calibration error Jev lower on all three. "differences under about 5 points are within noise."
In context with Laya and Kev's documented limits: Jev vs Laya vs Kev.
Code and item manifest (Apache-2.0); resultsB COIsmall-nno-baseline
20:53 UTC
B12 Bryo AI email benchmark (17 Sep)
A company evaluating Jev for its own product; no stake in the result stated.
1,565 German and English business emails, 10 categories. Private data.
Baselines: Gemini 3.5 and 3.8.
The CTO's post: "Jev lost to Gemini on our email classification benchmark. I'm still interested in putting it into production." Figures from Prefactor (secondary): Jev 96.4% vs 97.5% and 98.5%, "10 to 20 times cheaper".NoneC single-task
20:53 UTC · only non-English row with an LLM baseline
B13 arXiv 2609.24574 v2, decision models for text annotation
New York University Abu Dhabi, academic.
18 computational-social-science classification tasks, 7,977 items; via OpenRouter.
Baselines: 19 frontier and open-weight LLMs, same zero-shot protocol; the best LLM per task was chosen after the results.
Jev "trails the per-task best LLM on 14 of 15 evaluation tasks, with a median deficit of 11.6 macro-F1 points, at a median 44 times lower measured cost". Confidence better calibrated than the stated confidence of 16 of 19 LLMs; high confidence at near-chance accuracy on one empathy task. Routing low-confidence items to an LLM "matches or exceeds the LLM alone at a quarter to half of its cost".Data public; code repository (MIT) exists, not openedA provisionalpublic-data
21:01–21:02 UTC
B14 Elastic Search Labs, "Using Jev as a search reranker" (30 Sep)
A search vendor; not a Jev reseller.
E-commerce reranking on public ESCI (US English): questions frozen on 50 queries, then 250 test queries (4,754 query–product pairs).
Baselines: BM25, hybrid search, random order, a Jina reranker (row not extracted).
nDCG@10 for four Jev strategies 0.9533–0.9565 vs hybrid 0.9351 (+0.018 to +0.021, confidence intervals above zero); BM25 0.9234; random 0.8595 (high because most judged results are exact matches).Data public (ESCI); notebook at elastic/elasticsearch-labs commit 3e3d688, opened 7 Oct: it runs a 10-query sample only (data/esci-sample-10-queries.json); the 250-query test sample, raw judgments and bootstrap code were not foundB public-datasingle-task
20:57 UTC (30 Sep)
Regraded 7 Oct 2026: A (provisional) → B. The published notebook runs a 10-query sample, not the reported 250-query test, so the reported evaluation cannot be rebuilt from published materials. Prompts and code are published. Read 05:21 UTC.
B28 Red Hat Developer, "Benchmarking AI decision models against traditional guardrails" (2 Oct)
Red Hat. It publishes two of the classifier baselines (RedHatAI/deberta-v3-base-prompt-injection-v2, RedHatAI/granite-guardian-hap-125m) and one LLM baseline build, and runs the evaluation through its EvalHub and NeMo Guardrails work, so it has a stake. No Jev stake stated.
Guardrails: prompt injection, and content safety, toxicity and profanity, from EvalHub's NeMo Guardrails benchmark library, "class-balanced between risky and safe prompts". n not stated. "Jev-1.13.0, via TypeSafe's API", through an experimental NeMo Guardrails branch; a prompt was blocked "If any question returned a noul greater than 0.5".
Baselines: trained classifiers deberta-v3-base-prompt-injection-v2 and granite-guardian-hap-125m; LLM judges Shieldstral-1.0-3B, Nemotron-3.5-Content-Safety (default and custom), Qwen3.6-35B-A3B-FP8, DiffusionGemma-26B; zero-shot or decision models bart-large-mnli and Laya. Classifier baselines in context: Jev vs an LLM vs a classifier; content-safety use: content moderation.
Prompt injection (accuracy): Qwen3.6-35B 89.31%, deberta-v3-base classifier 89.01%, DiffusionGemma 87.72%, Jev 86.35% (4th of 9). Content safety: Jev 86.20% (1st), DiffusionGemma 85.53%, Qwen3.6-35B 85.47%, granite-guardian-hap-125m classifier 80.27%. Median latency: Jev 348.1 ms vs deberta 54.1 ms; Jev 360.4 ms vs granite-guardian 33.2 ms. The article concludes that "decision models like Jev do not reliably outperform LLM-as-a-judge, pre-trained predictive models, or open source decision models in speed or accuracy."Data: named public benchmark sets (n and revision not stated); code linked (NeMo Guardrails branch feat/jev-rail, eval-hub/eval-hub-contrib), not opened; prompts in the appendix; raw outputs not foundB COIprovisionalpublic-data
n not stated · 7 Oct, 05:18 UTC; Tables 1–2 rechecked on a rendered page 05:26 UTC: match
B29 Synthpop, "Does a decision model (like Jev) know when it is guessing?" (Post 1) (1 Oct)
Synthpop, a company with a commercial product; no Jev stake stated. Its repository says "This project is not affiliated with or endorsed by TypeSafe."
Calibration and self-knowledge on 15 public datasets and six generated task families. Main test: Banking77 and CLINC150 intents behind opaque codes, under nine knowledge settings; also TriviaQA, MMLU-Redux and MMLU-CF as four-option multiple choice. "About 575,000 API calls"; requests used jev-latest, "every response reported jev-1.13.0".
Comparators: GLiNER2.5-Decide (an open decision model). Fine-tuned student models (five copies of a 22M-parameter encoder); not a classifier head-to-head. No LLM. Using calibration in practice: confidence thresholds.
TriviaQA: "Keep only the answers Jev was at least 90% sure of, and you keep 82% of the questions, with 99.75% of those answers right". Smooth ECE "0.016 on MMLU-Redux, 0.031 on TriviaQA". Beyond the observed knowledge boundary, accuracy was "51% ... yet its confidence rose to 82%". Banking77 with one example per category: "83% confident but only 68% accurate".Data public (pinned revisions); code yes (Apache-2.0, commit 1867760; metrics/calibration.py opened); prompts rebuilt by the code; raw outputs not publishedA public-data
7 Oct, 05:18–05:20 UTC
B30 Maximum Effort, Minimum Reward (maximumeffort.substack.com), "How Accurately Calibrated is Jev?" (2 Oct)
An individual's blog; no stake stated.
Calibration on physics distributions: 10 distributions × 5 prompt templates × 20 variations, "1,000 settings"; Jev chooses among about 49–50 binned ranges. Also distribution choice (100 settings per family) and a ladder of arithmetic tests. Prompts written by GPT-6 Astra and Claude Opus 5.5. Jev version not stated.
Baseline: none; the reference is the analytic distribution, with a flat guess as the floor. Calibration in practice: confidence thresholds.
Total variation from the true distribution: "If Jev were to completely punt on the answer ... it would score a mean TV of 0.546. But Jev scores 0.518. For Uniform it is 0.77 vs. 0.39 for a flat guess, and for Poisson 0.65 vs. 0.64." Gamma problems were labelled "exponential" in "40/100 settings".None located: no code, data or raw outputs; example prompts as images onlyC no-baselinesingle-task
7 Oct, 05:18 UTC
B31 DoubtBench (Show HN, 4 Oct); the author's write-up "Jev isn't a better judge than Claude" (6 Oct) has the same numbers
The benchmark's author (GitHub deeplearningguy), an individual; no Jev stake stated. The benchmark invites leaderboard submissions.
NVIDIA HelpSteer2 (CC BY 4.0, pinned revision) turned into Score, Choice and Noul questions: attribute scores, acceptability, pairwise preference (asked both ways) and preference strength. Labels are human annotator votes. Validation split, 7,455 questions; Jev answered 7,455 with 0 errors.
Baselines: Claude Haiku 4.5, Sonnet 5.5 and Opus 5.5 with "verbalized" probabilities in JSON; Sonnet and Opus at low effort, and their refusals (30 and 36) count as wrong. A human-replay ceiling and a uniform floor. Judge use: Jev as a judge; calibration: confidence thresholds.
DoubtBench score (mean of accuracy, 1 − ECE and 1 − JS divergence) / accuracy: Jev 1.13 67.8 / 48.2%; Claude Haiku 4.5 69.2 / 45.8%; Opus 5.5 67.8 / 44.0%; Sonnet 5.5 67.4 / 42.5%; human ceiling 90.4 / 82.6%. On accuracy: "paired McNemar test, p < 0.001 against each". The author's caveat: "One dataset, one task".Data yes (data/doubtbench.jsonl, manifest hashes); code yes (MIT, commit d5345ee; adapters/jev.py, metrics/core.py opened); prompts yes; raw outputs yesA public-data
7 Oct, 05:19 UTC; leaderboard rechecked 05:26 UTC: match
B32 system-one-security (GitHub repository, created 4 Oct; runs 3 Oct)
An individual; no stake stated. Coordinated disclosure with a vendor is pending: "some experiments, claims and results are withheld".
Security experiments on a synthetic citation-checker knowledge base: truncation, instruction injection, fact poisoning, prepend eviction and a tripwire question. 12 to 24 claims per experiment; experiment 02 ran 3 times on Jev, the rest once. jev-1.13.0, hosted TypeSafe API.
Comparators: Cloudflare Clef and Clef-flash (decision models); no LLM or classifier baseline. Security limits in context: limitations.
"The instruction-style injections tested never made a wrong claim pass on the choice answer ... 0 of 12 on every injection". But "a 'labels are swapped' note at the end of the state pushed nearly every correct claim below 0.5 on the yes/no answer". Truncation: "Jev 24/24". The repository's 2,048-token truncation finding concerns Clef, not Jev.Data yes (synthetic, kb/); code yes (MIT, commit ecdda36; experiments/02-instruction-injection.ts opened); prompts yes; raw outputs yes, per runA small-nno-baselinesealedsingle-task
7 Oct, 05:20 UTC
B33 Vals AI, "An Independent Evaluation of TypeSafe's Jev" (6 Oct)
Vals AI, an AI evaluation company that publishes benchmarks and reports; no Jev stake stated.
(1) Claim verification: 400 items built from the latest 10-Ks of 48 large US companies, 5 categories × 80. (2) A preregistered 12-subtask slice of LegalBench: 396 questions (33 per subtask). Jev 1.13.0 through the provider API.
Baselines: 11 LLMs, including GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5, Sonnet 5.5, Haiku 4.5, Gemini 4 Argon and DeepSeek V4.1 Flash, each asked for a label and a probability. Judge use: Jev as a judge; error budgets: confidence thresholds.
"On claim verification, all twelve systems score 0.953 to 0.995. Jev ... scores 0.975 at $0.02 per 1,000 cases and ties with GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna." "On LegalBench, Jev finishes last of twelve on raw accuracy and on class-balanced accuracy." On the 1% error-budget test (held-out split of 265 items): "Jev was one of them, automating 95% of cases at 1.6% error". ECE "0.011 on claim verification, the lowest of the twelve, but 0.108 on LegalBench, the highest".Claim-verification data not published; LegalBench public, but the 396-question sample not identified; no code; prompts and raw outputs not publishedC public-data
7 Oct, 05:25 UTC
B34 Oodle AI blog, "Is Jev the right model for your agents?" (22 Sep; on Hacker News 6 Oct)
Oodle AI, an observability and AI-evaluation platform vendor; the post promotes its own playground. No Jev stake stated; the run was self-funded.
Two yes/no agent decisions, 200 items each (100 yes, 100 no): support escalation (public Hugging Face set with human sentiment labels) and model routing (labels from GPT-4's grades of a Mixtral answer). jev-latest through OpenRouter (resolved version not stated); P(yes) above 0.5 counts as yes.
Baselines: GPT-5.4-mini, Claude Haiku 4.5, Gemini 3.5 Flash, same prompt, no tuning. Scored by three LLM judges, averaged. Routing in practice: model routing.
Escalation: GPT-5.4-mini 84.5%, Jev 69.5%, Claude Haiku 4.5 66.0%, Gemini 3.5 Flash 66.0%; Jev p50/p95 163/264 ms. The author says a 0.3 threshold "would score about 82%"; that threshold was chosen after the fact from the first 100 items. Routing: Gemini 3.5 Flash 65.5%, GPT-5.4-mini 62.5%, Jev 59.5%, Haiku 4.5 56.5%. The author's caveat: "Treat this one as a method demonstration, not a verdict."Source datasets public; the 200-item samples not identified; task and judge prompts published; no code; raw outputs not located (promised on the vendor's playground, which needs an account)C small-nLLM-labelspublic-data
7 Oct, 05:28 UTC
B35 ivnle.github.io, "Your Language Model Is Not-So-Secretly a System 1 Model" (4 Oct; on Hacker News 6 Oct)
Two academic researchers on a personal blog; no commercial or Jev stake stated. The post argues a thesis ("matching Jev's overall performance ... does not require a new kind of model"); "A preprint with full details will follow".
25 datasets in three parts, 19,505 decisions: classification (13,053: Banking77 plus 20 BTZSC sets), agent monitoring (5,800: ABCD, AgentRewardBench) and triage (652: two TypeSafe published examples; the authors "replaced 25 of the 202 published labels" in the guardrail set). Separately, the 231 public JevBench questions. jev-1.13.0 through TypeSafe's Python SDK.
Baselines: 23 ranked models, mostly open-weight LLMs read from next-token probabilities in one forward pass, no training, temperature-scaled on a 30% split; closed comparators GPT-6 Luna (read from the letter it writes) and Gemini 2.5 Flash-Lite. No trained task-specific classifier. In context: Jev vs an LLM vs a classifier.
Overall macro-F1, the three parts weighted equally: "Jev scores 0.787 [0.776, 0.797]"; gemma-4-31B-it 0.794 (difference +0.006 [−0.001, +0.013]); GPT-6 Luna 0.782, "at about twice Jev's cost"; Gemini 2.5 Flash-Lite 0.735. "Seven of the 23 open models we ranked match Jev" (pre-set 0.03 margin). Agent monitoring: "six of the seven beat Jev, by 0.025 to 0.052"; triage: "Jev is ahead: six of the seven fall below it". JevBench: "Jev answers 85.7% [81.8%, 89.6%] correctly. gemma-4-31B-it beats it with 91.8%". The authors say equal weighting of the parts "favours Jev".Data public, plus TypeSafe's published examples (the 25 relabelled items not located); prompt template shown; no code repository found (latency points only); raw outputs not locatedB public-data
LLM-labels in part (excluded from its calibration figures) · 7 Oct, 05:28 UTC; rechecked 05:32 UTC: match
B36 SREGym blog, "Jev-Driven SRE Diagnosis: What Worked and What Failed" (6 Oct; on Hacker News 7 Oct)
The SREGym team, which maintains the SREGym SRE-agent benchmark and built the pipeline it tests. No Jev stake stated.
Kubernetes incident root-cause diagnosis: code collects cluster evidence, Jev picks the component, origin or victim role, cause category and key evidence item, and code assembles the diagnosis. "the 21 fault scenarios in the September 4 SREGym-Lite cohort" × 5 runs = 105 diagnoses. jev-1.13.0; 252 Jev calls.
Baseline: none. Scored by gpt-6-astra at high reasoning effort against SREGym's nine-question rubric, pass at 0.70.
"Judged diagnosis passes 80/105 (76.2%)". All five attempts passed on 16 of 21 faults and failed on 5 of 21: "For every fault, either all five attempts passed or all five failed." Median diagnosis time 14.6 s; estimated Jev cost "$0.15".Code yes at the linked commit 1a668ee (it has diverged from main); pipeline and judge oracle opened; judge.py, the rubric file and the 4 Sep cohort list not verified; per-fault scores on the page; raw Jev and judge outputs not locatedB LLM-labelssmall-nno-baselinesingle-task
Graded B (a first grading pass proposed A): rerun materials not all located, and the operator built the benchmark and pipeline. May move to A once those files are read. 7 Oct, 05:28–05:29 UTC
B38 komzweb/jev-vs-decisions-rpg (9 Oct; added 11 Oct)
GitHub account komzweb, an individual with a Japanese article on note.com; no stake stated. COI: none stated.
Choosing NPC moves in a small turn-based role-playing game. Jev jev-1.13.0 (pinned in deciders/jev.py L10) vs OpenAI Decisions API gpt-6-luna (deciders/openai_decisions.py L12). 100 matches; a per-move test of 100 positions × 5 option orders (500 answers per model).
Baselines: the Decisions API; a 4-rule NPC.
The Decisions API won 82 of 100 matches (Jev won 8; 10 draws). Per-move accuracy: Jev 51.0%, Decisions API 59.4% (exact McNemar p = 0.007). Calibration error (ECE, 5 bins): Jev 0.048, Decisions API 0.161. Median latency 168 vs 199 ms.Data yes (data/positions.jsonl, data/positions_candidates.jsonl, data/shopkeeper.jsonl; game engine game/engine.py); metric code experiments/stats.py L9–60 (Wilson interval; exact McNemar L29–31; ECE L34–46); results results/summary.md L9, L52–53, L70 and L100 hold the README numbers; outputs partly (logs_sample/ holds the first 100 calls only)A game taskoperator-designed taskone runfull logs not committed
Read at f81c004, so not provisional. Checked 11 Oct 2026, 05:23:13 UTC (R13-S50 to R13-S54).
B39 mateusfonsek/openai-decisions-vs-jev-vs-laya (8 Oct; added 11 Oct)
GitHub account mateusfonsek, an individual; the README says "Benchmark independente"; no stake stated. COI: none stated.
Portuguese, three tasks with 150 cases each (450 per provider): agent routing (8 tools), prompt-injection detection, LLM-as-judge (1–5 score). Jev (adapter default jev-latest, bench/adapters/jev.py L16; README L102 says jev-1.13.0 is recorded in every saved response) vs OpenAI Decisions API gpt-6-luna vs Laya laya-multilingual 0.3.28 run locally.
Baselines: the Decisions API; Laya zero-shot.
Routing accuracy: Jev 88.0%, Decisions API 86.7%. Injection F1: Jev 96.6%, Decisions API 95.8%. The operator says neither the routing nor the injection difference is statistically significant. Injection precision: Decisions API 100.0%, Jev 97.3% (0.0% vs 4.0% false positives on hard negatives). Judge: Jev ahead on the exact judge score, 80.7% vs 72.0% (McNemar p = 0.011, as stated in README L13); the Decisions API ahead on scores within ±1 point, 100.0% vs 98.7%. Laya zero-shot routing: 30.7%.Data yes (dataset/routing.jsonl, injection.jsonl, judge.jsonl, suites.yaml, SHA256SUMS; CC BY 4.0); metric code bench/metrics.py L4 (headline metric per task: accuracy, F1, exact), L7–41, L94–118; results results/2026-10-07/report/summary.md holds the README numbers (88,0% vs 86,7%; 96,6% vs 95,8%; 80,7% vs 72,0%); outputs yes (per-provider saved responses in results/2026-10-07/*.jsonl). The McNemar computation was not located in the files read.A single languageLLM-generated dataone runjev-latest aliasp-value source not located
Dataset generated with one LLM (Claude), with a quick human review (README L99). Read at 160e112, so not provisional. Checked 11 Oct 2026, 05:23:14 UTC (R13-S55 to R13-S60).
B40 anessbelbati/jev-rerank-bench (25 Sep; added 11 Oct)
Reranking-pipeline study. GitHub account anessbelbati, an individual with a blog write-up on the author's own domain; no stake stated. COI: none stated.
Each query gets a 30-passage BM25 shortlist (candidates/*.jsonl), and each system reorders it (rerankers/jev.py L7–19). 8 English datasets with 1,617 scored questions, plus BRIGHT subsets, NevIR and MIRACL French.
Baselines: Cohere Rerank 4 Pro, ZeroEntropy zerank-2, DeepSeek V4.1 Flash and open models.
Equal-dataset nDCG@10: Jev rubric 0.692, Cohere Pro 0.691, zerank-2 0.682. Jev minus Cohere is +0.001 (95% interval −0.009 to +0.012), and the README says this "establishes neither a winner nor equivalence". With equal weight per query, Cohere scores 0.756 and Jev 0.738 (README L66–76).Data yes (BM25 shortlists in candidates/); metric code eval.py L71–76 (nDCG@10) and L236–258 (macro-average over datasets), significance.py L1–6, L47–54 (paired bootstrap from the cached answers); results results/significance_run_sept25.txt L16–17 ("best = jev-score-batch (0.692)", "vs cohere-pro 0.691 … p=0.906 within noise"); outputs yes (cache/, 945 saved-response files; README L314)A reranking onlyoperator-chosen datasetsorder depends on averagingper-dataset results go both ways
Jev is behind on FiQA, NQ, TREC-COVID and MIRACL-fr ("REAL" gaps in the same file). Read at fecba75, so not provisional. Checked 11 Oct 2026, 05:23:14 UTC (R13-S61 to R13-S66).
B41 kartikanand73/decision-api-benchmark (10 Oct; added 12 Oct)
GitHub account kartikanand73, an individual. README L295: "No affiliation with TypeSafe, OpenAI or Microsoft. All API usage is paid for by the author." The repository "grew out of" the same account's Jev Decision Gateway, an article series (README L23). COI: none stated.
Bank-withdrawal routing: approve, step-up or manual review, plus a takeover probability, after a fixed policy. 500 synthetic cases (seed 505; 320 approve, 76 step-up, 104 review), each in all 6 option orders, plus 50 repeats; two full runs on 10 Oct 2026, about 2 hours apart, from one location. All three models were called through OpenRouter's /api/v1/systemone with byte-identical requests apart from the model ID: Jev typesafe/jev-1.13 (served jev-1.13-20260917), OpenAI Decisions API openai/gpt-6-luna-decisions (served -20261006) and Microsoft-Decision-1 microsoft/microsoft-decision-1 (Microsoft's model, a hosted API from another provider; served -20261009).
Baselines: the OpenAI Decisions API; Microsoft-Decision-1.
On synthetic bank-withdrawal routing (B41, grade A), Jev (89.2% macro-F1) and Microsoft-Decision-1 (87.6%) were not separated and both were ahead of OpenAI's Decisions API (71.4%); Decision-1 kept the same route across all 6 option orders more often (97.2–97.6% vs Jev 91.6–91.8%, Decisions API 64.2%). Pooled over both runs with 95% intervals: Jev 89.2% (86.1–92.1), Decisions API 71.4% (66.7–75.5), Decision-1 87.6% (83.9–90.8); Jev minus Decision-1 +1.7 points (−1.8 to +5.2), "macro-F1 not separated"; Jev minus Decisions API +17.8 points (+14.2 to +21.9). Unnecessary manual reviews per run: Jev 1 and 2, Decisions API 10 and 10, Decision-1 51 and 50. Fraud auto-approved: 0 for all three. Median latency: Jev 233 ms, Decisions API 214 ms, Decision-1 303 ms (README L205–241).Data yes (data/cases.jsonl, 500 lines, with its generator compare/dataset.py; its sha256 prefix 8e337b334f20396d equals the run's meta.json); metric code compare/metrics.py L61–83 (per-class precision, recall, F1, macro-F1, balanced accuracy) and L264–296 (paired case-resampling bootstrap); results runs/compare-20261010-222249Z/comparison.md L26 holds the README's 89.2% (README L222), and the run-2 report.md L37 gives macro-F1 88.0%, 71.4% and 87.4%; outputs yes (calls.jsonl with every call's full response, summary.json and meta.json for both runs). Cost figures not carried.A synthetic single taskthresholds set with Jev-shaped probabilitiesall three via OpenRoutercode version not pinned
Thresholds: "set with Jev-shaped probabilities" is the operator's own note (README L248); it affects the action metrics, not the route metrics, and may favour Jev. OpenRouter adds one hop for each model, so the Decisions API was not called in its native /v1/decisions format; the operator says Microsoft's wire format was unpublished at the time (README L68). Code version: meta.json has git_commit: null ("Code version: not a git checkout"); the committed results files hold the README's numbers, so this is a flag, not a lower grade. Read at 1ffa4d5, so not provisional. Checked 12 Oct 2026, 05:12:40 UTC; spot-check 05:18:03 UTC (R14-S200 to R14-S206, R14-S51).
B42 etsabary/luna-decisions-benchmark (10 Oct; added 13 Oct)
GitHub account etsabary: "Independent community benchmark; no affiliation with OpenAI, OpenRouter or Jev/TypeSafe"; MIT licence. The same account built the Jev benchmark the questions come from (jev-deterministic-benchmark). COI: none stated.
1,000 five-option deterministic logic questions (25 families), copied from the same operator's jev-deterministic-benchmark. OpenAI Decisions API openai/gpt-6-luna-decisions called through OpenRouter's alpha Decisions endpoint (https://openrouter.ai/api/alpha/decisions), 2 attempts per item, 10 Oct 2026; Jev answers reused from a 22 Sep run.
Baselines: the OpenAI Decisions API (Jev and the Decisions API scored against the same withheld key).
On 1,000 five-option logic questions (B42, grade B), Jev answered 817 correctly and OpenAI's Decisions API 612; Jev's answers come from a 22 Sep run, and the answer key is not published. Decisions API 612/1,000 (61.2%; 62.3% of 983 valid answers; 17 unanswered); Jev 817/1,000 (81.7%) (README; analysis/summary.json).Data partly (data/questions_1000.csv, data/results_jev_1000.csv, data/results_luna_1000.csv; the answer key is not committed, only its sha256); metric code analyze_comparison.py (correct = status ok and selected option equals the key's option; scored with an external key, aggregates only); results analysis/summary.json (612 and 817 match the README); run manifest data/luna_run_manifest.json (model, OpenRouter alpha endpoint, 2 attempts per item). Latency and cost not carried.B via OpenRouter alpha endpointJev not re-runanswer key withheld17 refusals, cause unknownoperator built both benchmarksyoung repositorylatency and cost not comparable
Flags: (1) the Decisions API was called via OpenRouter's alpha Decisions endpoint, not OpenAI direct; (2) Jev was not re-run (22 Sep vs 10 Oct; mixed Jev providers, 256 answers without a verified snapshot); (3) the answer key is withheld (only its sha256 is committed), so the scores cannot be rechecked; (4) 17 persistent refusals (HTTP 502), cause unknown; (5) the operator built both benchmarks; (6) young repository (2 commits; HEAD changed after the main publish commit); (7) latency and cost are not comparable across dates and routes. Shown below B38, B39 and B41; not in the key finding. Read at 7a5ea13, so not provisional. Files opened 12 Oct 2026, 05:27:50–05:29:20 UTC (R14-S300 to R14-S305, spot-check R14-S55); pin rechecked unchanged 13 Oct 2026, 05:24 UTC (R15-S53).
B43 kprzybys/snap-vs-jev-swe (12 Oct; added 14 Oct)
GitHub account kprzybys. Its disclosure: "I take part in SubQ's Snap creator challenge (which has prizes); the runs and numbers are my own. Not affiliated with TypeSafe". COI: the operator takes part in Subquadratic's Snap creator challenge, which has prizes; Subquadratic makes SubQ Snap. Code MIT; results and labels CC BY 4.0.
SWE-bench Verified file localization: 200 issues, 10 candidate files each. Also seven public software-engineering classification datasets (3,522 cases) and an option-order test (2,222 questions; 699-question common subset). SubQ Snap subq-snap-preview and Jev jev-1.13.0 through the vendors' APIs (base URLs not opened; the README says "the APIs"); Clef-flash 9B, H2O-Lightning-4B and Intern-Decision-4B run locally on an AMD GPU. One rerun on 11 Oct 2026 (APIs 22:15–22:36 UTC).
Baselines: SubQ Snap; three open decision models run locally.
On 200 SWE-bench Verified issues, SubQ Snap and Jev picked the file to change about equally often (78.5% and 79.0%); asked per file, Snap was ahead but not significantly (B43, grade B; the operator takes part in Subquadratic's prize challenge). Per file: Snap 85.5% vs Jev 81.8%, "+3.7 points, 95% CI −0.5…+8.3 … p = 0.12: not significant"; AUROC 0.981 vs 0.963 (difference CI +0.011…+0.025). Long context: Snap 87.5% with all 10 files in one request (about 71k tokens); not run for Jev, which is "over its ~32k-token limit for ~95% of issues", so this result has no Jev arm. Classification: Snap ahead on 6 of 7 datasets, significant on 3 (uncorrected). Option order: reversed options changed 5.0% of Snap's answers and 7.1% of Jev's. The 79.0% and 78.5% were recomputed from the results and labels files with the repository's own tie rule (FractionalTies, ties.py); both match (200 of 200 Jev answers; 200 Snap answers in 201 lines, one retry).Data yes (labels data/labels/swe-paths.jsonl; public datasets); metric code src/snapbench/evaluation/evaluators.py and src/snapbench/evaluation/ties.py; results results/swe-paths-jev.jsonl and results/swe-paths-snap.jsonl (raw answers); summary reports/summary.md. Latency and cost not carried.B COIsingle runspreview modelpublic-datapost-hoc wordingyoung repository
Grade B: rerunnable from published code, data pins and raw answers, but with a stake (B is "partly rerunnable or everything published with a stake"). Flags: (1) COI: the operator takes part in Subquadratic's Snap creator challenge, which has prizes; (2) single runs (run-to-run variation reported); (3) preview model, with a server change the vendor confirmed on 10 Oct (the published rerun is after it); (4) public data, so contamination is possible; (5) question wording chosen after the fact on one dataset; (6) young repository (1 commit). Not a small-LLM comparison. Read at b754c44, so not provisional. Files opened 14 Oct 2026, 16:52:30 UTC (R16-S66); pin from the commit feed, 16:51:37 UTC (R16-S65).
B44 dante01yoon/decision-model-arena (12 Oct; added 14 Oct)
GitHub account dante01yoon; MIT licence. No stake and no vendor affiliation stated. COI: none stated.
Versus Tetris round robin among four decision models: 8 seeds × 6 pairs = 48 games, scored as games won under the operator's scoring script (measure/tournament.mjs, the winnerOf rule: topping out loses, then more lines, garbage sent, pieces). Also 12 four-player bomb games and image probes (not recounted). Measured 12 Oct 2026, 14:51:37 UTC, "called from the US". Routes: OpenAI Decisions API gpt-6-luna via OpenAI direct; Microsoft-Decision-1 via OpenRouter's alpha decisions endpoint (provider Azure), not Foundry; Jev as jev-latest via TypeSafe direct; Laya run locally (Apple Silicon).
Baselines: the OpenAI Decisions API; Microsoft-Decision-1; the open Laya model.
In a 48-game Tetris round robin, Microsoft-Decision-1 won 15, Jev 12, OpenAI's Decisions API 11 and the open Laya model 8; Jev beat the Decisions API 6–2 head to head (B44, grade A, small sample, one game). Games won of 48: Decision-1 15, Jev 12, Decisions API 11, Laya 8, 2 draws. Head to head (8 games per pair): Jev 6, Decisions API 2; Laya tied Jev 4–4. The 48 matches were recounted from evidence/league.json and match the README.Code yes (game server and scoring measure/tournament.mjs, MIT); seeds yes; results evidence/league.json (48 matches, measured_at 2026-10-12T14:51:37Z); call log evidence/calls.jsonl.gz published (not opened). Rerunning needs the vendors' keys. Latency and cost not carried.A small-nsingle-taskDecision-1 via OpenRouter alpha endpointJev called as jev-latestone date and region
Grade A: rerunnable; server, scoring code, seeds and the call log are published. Flags: (1) small sample (8 games per pair); (2) one task (a game); (3) Decision-1 called through OpenRouter's alpha endpoint, not Foundry; (4) Jev called by the alias jev-latest, not a pinned model; (5) one date and one region ("called from the US"). One game with 8 games per pair, so it does not rank the models outside this game. Read at 285c803, so not provisional. Files opened 14 Oct 2026, 16:55:30 UTC (R16-S67); pin from the commit feed, 16:51:37 UTC (R16-S65).
B45 gbolask24/decision-bench (18 Oct; added 24 Oct)
GitHub account gbolask24, an individual (articles and a video). No stake found in the opened files: not a vendor, gateway or alternative-model author. COI: none found.
Five public-data classification tests: T1 Bitext support intent (1,000 items), T2 Amazon electrical product category from text (2,000), T3 the same with the main product photo (1,999), T4 CLINC150 (1,000), T5 Decision Bench bench-v4 (1,071); 7,070 items per full arm, seed 1017, the same option order for every arm. Jev jev-1.14.0 (pinned in dbench/run.py L21–24) and OpenAI's Decisions API gpt-6-luna (/v1/decisions) on every item; Claude Opus 5.5 (low effort) on a seeded one-in-ten sample (707 items). Run 17 Oct 2026, 16:22–16:50 UTC, while OpenAI's Decisions API was in public beta (generally available from 22 Oct 2026).
Baselines: the OpenAI Decisions API; Claude Opus 5.5 on a sample.
Accuracy, Jev 1.14 vs the Decisions API: "no detectable difference" on T1–T3; "Jev ahead" on T4 (88.2% vs 84.2%) and T5 (94.5% vs 90.6%). Brier: "Jev better" on all five (not corrected for multiple comparisons). On the Opus sample, Opus was ahead of Jev 1.14 on T2 (+5.0 points) and T3 (+4.5) and ahead of the Decisions API on T2–T5; none ahead on T1. Jev's T1 accuracy, 0.922 in results/summary.json L8, matches the operator's report (92.2%, results/data-pack.md L21).Data public (seeded samples of public datasets); metric code dbench/score.py L72–116 (accuracy L87; McNemar test with Holm correction and the claim rule L115–116; Brier L23–28); results results/summary.json; report results/data-pack.md; prices as published in dbench/prices.py (16 Oct: Jev $0.042 and Decisions $0.10 per 1M input tokens, output free; Opus 5.5 $4 input and $20 output). Per-call logs referenced in the README are not in the repository (.gitignore excludes logs/). No licence file in the tree. Rerunning needs the vendors' keys.A public-datasmall-n (Opus arm)
Grade A: data, code and configuration are published and no commercial stake was found. Flags: (1) public data, so contamination is possible; (2) small sample for the Opus arm only (a one-in-ten sample of each test). Note: per-call logs referenced in the README are not in the repository. Read at e5b7202 (commit 18 Oct 2026, 13:58 UTC), so not provisional. Files opened 24 Oct 2026, 05:15–05:17 UTC; the T1 figure spot-checked at 05:21 UTC (R23-S200 to R23-S211, R23-S49).
B46 jcarlosrodicio/subq-vs-jev (13 Oct; added 24 Oct)
GitHub account jcarlosrodicio, an individual: "Independent test, not affiliated with Subquadratic or TypeSafe". SubQ access was free during its preview weekend (stated; not a commercial stake). COI: none found. A different operator, repository and task from B43.
Question answering over a seeded synthetic store event log, 2K to 360K tokens (8 lengths × 20 logs), with Choice, Noul and Score questions; ground truth from the generator, re-derived by tests/test_truth.py. 3,840 graded answers; head to head up to 30K tokens, n = 960 per model. SubQ Snap subq-snap-preview vs Jev called as jev-latest at api.typesafe.ai, which served jev-1.13.0 (per results.json; bench/client.py L12–13). Above Jev's context limit the operator split the log into 29K chunks. Run 13 Oct 2026, 14:53–15:05 UTC, during SubQ Snap's public preview and before Jev 1.14.
Baseline: SubQ Snap.
Accuracy up to 30K tokens: Snap 93.9%, Jev 94.7% (Wilson intervals overlap). At 360K, with the log split into 29K chunks: Snap 58.8%, Jev 77.1%. ECE up to 30K: Snap 0.028, Jev 0.038; at 64K and above: Snap 0.163, Jev 0.117 (chunked). Same answer on repeat: Snap 90.0%, Jev 50.8%. Median latency at 8K: Snap 0.75 s, Jev 0.31 s. No overall winner stated. Jev up to 30K: 909 of 960 correct, 0.946875 in site/data/results.json L100–104, matches the README (94.7%).Generator, seeds and tests; metric code bench/analyze.py L30–40 (grading), L115–118 (accuracy with Wilson intervals), L131–134 (head to head up to 30K), L143–153 (ECE and Brier); results site/data/results.json. Listed prices in the README (13 Oct): Jev $0.042 and SubQ $0.05 per 1M input tokens. The 360K figure was not matched line by line and the raw answers were not opened.A single-task
Grade A: rerunnable; the generator, metric code and results are published, with no commercial stake. Flag: one synthetic task. Jev was called by the alias jev-latest, which then served 1.13. Read at c037905 (commit 13 Oct 2026, 15:09 UTC), so not provisional. Files opened 24 Oct 2026, 05:16 UTC; the head-to-head figure spot-checked at 05:21 UTC (R23-S217 to R23-S223, R23-S49).
B49 dengzhe-hou/S1CogBench (commit 8 Nov 2026; added 9 Nov)
GitHub account dengzhe-hou, the benchmark's author (an arXiv paper, 2611.00911, is linked; not opened). The author's own benchmark; it also evaluates the open Laya and Intern-Decision models. COI: no commercial stake found in the files opened.
A multi-task cognitive benchmark for decision models. One task read here: a Bayesian urn task, where the model gives a probability and the metric is its absolute distance from the Bayesian posterior (mean absolute error, lower is better), n = 400 for Jev. Jev 1.14 via the API snapshot typesafe/jev-1.14-20261014.
Baselines in the same table row: a uniform guess, Laya, Intern-Decision, Claude Opus, Sonnet and Haiku.
Bayesian urn, posterior MAE (lower is better), as listed: Jev 1.14 0.068; Claude Opus, Sonnet and Haiku 0.001; Intern-Decision 0.235; Laya 0.291; uniform guess 0.349. The README states no figures; the other tasks were not read. Jev 0.068 in results/aggregate/table2.csv L2 matches 0.0682 (n = 400) in results/aggregate/results.csv L1848.Code and aggregate results; metric code src/paper_probabilistic.py L187 (per-row distance) and L200–212 (means). The raw Jev response file named in results.csv is not in the repository, and eight modes need gated access to a test set (README L95–96).B one task matched
Grade B: partly rerunnable; the raw Jev responses are not published and some modes need gated data. Flag: a multi-task benchmark of which one task row was matched. Read at 9cd68df, files opened 9 Nov 2026, 11:09 UTC; the Jev figure re-read at the pin at 11:17 UTC (R29-S112, R29-S113, R29-S38).
B50 GopalAlag/FrontierLLMvsJEV (commit 6 Nov 2026; added 9 Nov)
GitHub account GopalAlag, an individual; slide decks in the repository suggest a business evaluation (not opened). COI: no stake in Jev or the LLMs found.
IVR intent routing: map a caller's request to one exact intent; threshold fitted on a calibration set only (src/ivr_intent/metrics.py L37–49). 250 evaluation items per iteration, 2 iterations, seed 42 (run manifest). Jev through langchain-typesafe 0.0.1a3.
Baselines: GPT-6 Luna, Claude Sonnet 5.5, Gemini 3.1 Flash-Lite, Claude Haiku 5.5 and a TF-IDF logistic-regression classifier.
Exact-intent accuracy, as listed: GPT-6 Luna 0.652, Claude Sonnet 5.5 0.644, Jev 0.636, Gemini 3.1 Flash-Lite 0.620, Claude Haiku 5.5 0.608, TF-IDF classifier 0.304. Jev 0.636 in results_ivr_final/summary_by_system.csv L3 matches results.md L26. A second project in the same repository (fraud detection) reports Jev F1 0.5965 against 0.9503 for gradient boosting; not graded here.Code, data, run manifest and raw event logs published; metric code src/ivr_intent/metrics.py L52–88. The README states no result figures; they are in the results files. Seed data were transcribed from screenshots (README L56).A small-n run command differs
Grade A: rerunnable; code, data and results are published, with no stake found. Flags: a small sample (250 per iteration); the README's run command (3 iterations, n 400, L63) differs from the manifest of the reported run (2 iterations, n 250). Read at 27a73b7, files opened 9 Nov 2026, 11:09 UTC; the Jev figure re-read at the pin at 11:20 UTC (R29-S112, R29-S114, R29-S40).
B51 tukka25/graph-reflex (commit 8 Nov 2026; added 9 Nov)
GitHub account tukka25. COI: the system under test, "Neo4j measures, Jev decides", is the author's own harness.
Product search on STaRK-Amazon (public data), 100 questions, head to head: Hit@1, Hit@5 and reciprocal rank against STaRK's answer IDs. STaRK labels "miss some valid products" (README L34).
Baselines: a Claude Opus 5.5 agent writing Cypher queries, a Claude Haiku 5.5 decider in the same harness, a cross-encoder and vector search.
Hit@1, as listed: Opus 5.5 agent 0.69, harness with Jev 0.58, harness with a Haiku 5.5 decider 0.50, cross-encoder 0.46, vector search 0.33. The README's "84% of the agent's accuracy at 3% of its cost" is the author's own derivation, not checked. 0.58 in README L42 matches 0.580 (n = 100) in docs/results.md L13.Code and a generated results page; metric code src/lib/eval/metrics.ts L14–29. Per-question outputs are not committed (data/ is git-ignored).B COI small-n public-data
Grade B: everything needed to rerun is published except the per-question outputs, and the author has a stake in the harness. Read at 47859bd, files opened 9 Nov 2026, 11:09 UTC; the Jev figure re-read at the pin at 11:24 UTC (R29-S112, R29-S115, R29-S44).

Using Jev as a judge: rows B02, B06 and B08 are summarised, unchanged, on Jev as a judge, next to what four judge tools record when a Jev call fails (4 Oct 2026). Judge-style studies added on 7 Oct: B31 (grade A) and B33 (grade C); see Jev as a judge.

Tier 2: shorter reads

Reported About a minute per item on 30 Sep 2026 (around 20:57 UTC); "claim only" unless materials were seen. B26 is carried from earlier research. B47 and B48 (blog posts and their Hacker News threads) read 24 Oct 2026, 05:16–05:17 UTC.

Study and operatorTask, n and baselineResult as reportedGrade and flags
B15 punk2898 benchmark in awesome-jev-verified
Awesome-list maintainer; no stake stated.
2,390 labelled questions (true/false, multiple choice, consistency, injection). Baselines: GPT-4.1-mini, GPT-5.6 Sol.True/false 92.5% vs 87.8% (4.1-mini) and 92.1% (Sol); multiple choice 77.3% vs 74.3% and 82.5%; yes/no calibration error 0.048 (Sol 0.043); multiple choice "overconfident, same as Sol"; "445× cheaper did not reproduce" (88× vs Sol).A provisional
B16 alexmolas.com (individual blog), "Jev can't be calibrated" (23 Sep)
Individual.
Argument with one small example; no baseline."Jev says a fair coin lands heads with probability 0.92"; yes/no questions better calibrated than Choice in a linked experiment.D
B17 AnthusAI/Jev-Calibration (19 Sep)
An AI firm; no Jev stake stated.
Held-out calibration of a yes/no question on a sentiment-style set; recalibration methods compared.Calibration error 0.117 raw, 0.052 after Platt scaling, 0.008 after isotonic; accuracy and AUC "barely move (0.872 → 0.874)".B no-baseline
B18 Rene-1 31B model card (27 Sep)
Alternative-model author.
"Decision Index 0.2, 37 of 40 benchmarks, evaluated locally"; compared with Jev.Balanced skill 64.18, marked verified: false; "+9 over Jev" per the pool's alternatives notes.D COI
B19 Ollaya README
Local-runtime author.
LocalLLaMA/typed-decisions, n not stated; compared with Jev.winnow:e4b "0.722 accuracy on typed decisions (TypeSafe's Jev: 0.738)".D COI
B20 firelex/jeff README (created 28 Sep)
firelex, the alternative model's author.
Several benchmarks; compared with Jev."On benchmarks they approach, and sometimes beat, Jev" (figures not extracted).D COI
B21 "A DIY Jev" (kgluszczyk, DEV.to) (28 Sep)
An engineer building an in-house alternative.
6,762 public examples, 13 datasets. Baselines: prompted JSON and a logprob adapter on gpt-6-luna.Average accuracy: JSON 70.1%, DIY logprob 73.8%, Jev 76.5%; calibration error 0.175 / 0.211 raw (≤0.10 calibrated) / 0.110; DIY "nearly matched Jev on most category tasks".C public-data
B22 "Jev After Eight Days of Independent Tests" (DEV.to) (24 Sep)
Individual; a secondary review with its own grades.
Synthesis of other studies, not a study."Jev sits level with mid-price LLMs and 6.5 to 11.5 points behind the frontier in the cleanest comparison". Led this site to B13.Not graded
B23 Casco CVSS benchmark (28 Sep)
A security product company.
2,449 security findings scored with CVSS 3.1; 8 models including GPT-6 Astra and Claude.Mean absolute error vs Casco's scores: Astra 1.92 (first), Jev 3.40 (seventh of eight), Haiku 3.94. Jev scored 537 of 655 informational findings above zero.C
B24 jev-scifact-eval (29 Sep, "pre-registered")
Individual.
SciFact claim verification. Baselines: keyword rule; VeriSci (2020); no LLM.Label+Rationale F1 0.625 [0.567, 0.684] vs 0.303 and 0.485.A provisionalpublic-datano-baseline
B26 Vercel fx command safety (CEO post, 16 Sep)
Vercel, a gateway selling Jev. Production claim on Products.
Not published. Baseline: GPT Luna."up to 18x faster (p95) and more accurate".D COI
B37 mugunthank7/jev-pulse (README dated 7 Oct; added 10 Oct)
Reranking-pipeline study. An individual developer's demo (GitHub account mugunthank7); no stake stated. COI: none stated.
Amazon ESCI search reranking: 40 queries, 653 products. Jev (typesafe/jev-1.13 via OpenRouter) vs BM25.
Baseline: BM25 only; no LLM or trained reranker.
nDCG@10 BM25 0.660 vs Jev 0.772 ("+0.112, standard error ~0.037"). README only.A small-nBM25-only baselinesingle-taskoutputs not committed
Code, data and metric at a1e796f: run.ts, apps/api/src/rerank/metrics.ts L5–11, the data (data/esci-search.json) and the README opened 8 Oct, so the repository was read and the grade is not provisional. Graded A on 10 Oct 2026 (R12-S70, R11-S209, R11-S210).
B47 kapa.ai, "Jev is NOT a state-of-the-art reranker (but it's close)" (15 Oct 2026; added 24 Oct)
Reranking-pipeline study. kapa.ai, a RAG and company-knowledge-agent vendor, on its own benchmark (Company Knowledge Bench); it uses Voyage in production. COI. Hacker News 50115347.
Rerank the top 50 hybrid-search candidates and score the top 10 against labelled criteria (partial credit); 1,000 cases from production queries, each with its own knowledge base. Jev "1.13 and 1.14" in Noul and Score modes (model IDs and route not stated). Run date not stated.
Baselines: Voyage rerank-2.5 and rerank-3; Cohere rerank-v3.5, v4.0-fast and v4.0-pro; Qwen3-Reranker-8B (through OpenRouter).
Retrieval score: Voyage rerank-3 0.535 ("No other reranker beat it on any of the three"); Jev 1.14 (Noul) 0.532 ("just 0.003 behind"); Voyage rerank-2.5 0.526; Cohere rerank-v4.0-pro 0.506; Jev 1.13 (Noul) 0.519. The post's cost figures are per 1,000 questions and are not quoted here.C COIsealedsingle-task
Grade C: task, n and baselines stated; no materials published. Read 24 Oct 2026, 05:16 UTC (R23-S224, R23-S226).
B48 Sayora, Jev vs embeddings for picking a page (Many Hands blog, 12 Oct 2026; added 24 Oct)
Relevance-judgment study. Sayora, a UK AI-receptionist company, evaluating page picking for its own product (a product decision; no COI flag). Hacker News 50070393.
Pick the right page for a caller's question from page descriptions (a choice question that includes "none"), scored against relevance labels judged by an LLM. 168 questions (128 answerable) over four small-business sites (22, 74, 75 and 232 pages). Jev jev-1.13.0. Questions were written by Claude Opus 5.5, page descriptions by Claude Haiku 4.5, and relevance judged by Claude Opus 5.5 (5 checked by hand); Haiku is a writing tool here, not a compared model, so this row is not counted among the small-LLM comparisons. Run "at the start of October".
Baselines: text-embedding-3-large (production chunk search, and over the same descriptions); BM25; GPT-6 Luna through Azure's chat API.
Right page first (of 128): Jev 88% vs production embeddings 70% ("18 points, with a 95% interval of 11 to 26"); embeddings over the same descriptions 73%; BM25 53–57%; GPT-6 Luna 86% ("Luna tied Jev on picking"). OpenAI's Decisions API was not tested ("We don't have access to it yet").C LLM-labelssmall-nsealedsingle-task
Grade C: task, n and baselines stated; no materials published. Read 24 Oct 2026, 05:17 UTC (R23-S225, R23-S227).

Leads, not graded (treated as D until read): calfram-bench (B25, calibration audit, not read); Vercel engineer's Banking77 sample, n = 385, 80.3% as quoted by Jevals (B27, not opened); LanceDB reranker comparison (now in Reranking studies); Halv's SWE-rebench cost claim; the CLM-8B "9x faster than Jev" headline; PostHog's Jeeves; the YouTube video "Jev's Founder Explains: Why You Can't Trust AI Benchmarks"; on the videos page, M37 and two YouTube creator tests. Unverified

Leads opened 10 Oct 2026 (three graded on 11 Oct)

Reported README text and top-level file lists at pinned commits, 10 Oct 2026, 05:20–05:22 UTC (the GitHub API rate limit ran out at 05:20 UTC, so no other file was opened). Results are in each operator's own units; nothing was rerun or downloaded. Operators are named by GitHub account or organisation. The first three were graded on 11 Oct 2026 as B40, B39 and B38; the lines below point to their rows.

  • anessbelbati/jev-rerank-bench at fecba75: now graded: see B40 (grade A, reranking-pipeline study; metric, results and saved responses opened 11 Oct 2026).

  • mateusfonsek/openai-decisions-vs-jev-vs-laya at 160e112: now graded: see B39 (grade A; metric, results and saved responses opened 11 Oct 2026).

  • komzweb/jev-vs-decisions-rpg at f81c004: now graded: see B38 (grade A; metric and results opened 11 Oct 2026).

  • alpersonalwebsite/classifier-bakeoff at 4a1c4a0 (9 Oct). Operator: GitHub account alpersonalwebsite, an individual with a site report. Lead, not graded: its public report (reports/public/20261007T235443Z-da1571.html) was opened on 11 Oct 2026, 05:25 UTC; its metric code (bakeoff/) was not opened. Task: synthetic lead triage; Jev vs OpenAI Decisions API vs Claude through OpenRouter, six variants. The report states n: 200 synthetic messages, "written by anthropic/claude-opus-5.5", an LLM from a vendor whose models are in the comparison. Reported in the report: "OpenAI's Decisions API has the highest share fully right at 88.0%, Sonnet 5.5 has 86.0%, and Jev 84.0%, all within noise"; Jev ranks first on the report's cost tie-break rule. To carry when graded: LLM-generated data from a vendor in the comparison; one run. The README says results/ is not committed (R13-S67, R12-S61).

  • KizitoNaanma/decisionbench at 196e424 (7 Oct). Operator: GitHub account KizitoNaanma, an individual. Lead, not graded: where its results data sits was not identified (a dashboard; db/), and no metric file was opened. Task: invoice processing, 100 cases × 5 questions (500 decisions per model); Jev vs Claude Sonnet 5, which is not a substitute comparison. Reported in the README: accuracy 82.1% (Jev) vs 62.2% (Claude); per question, Claude is higher on duplicate detection (95.0% vs 89.9%). Data from a public Hugging Face dataset (R12-S63). Not done on 11 Oct 2026: the results location is still not identified, so it stays a lead, not graded.

  • TypeSafe blog, Jack & Jill case study (typesafe.ai, 7 Oct 2026; still online with its text unchanged on 14 Oct 2026, 16:46 UTC, though the blog index no longer shows its card). Vendor lead: D if graded (vendor and customer; no code, data or test set published). One candidate-scoring stage of a hiring marketplace moved from Gemini 3.1 Flash Lite to Jev, with "88% lower cost". Reported. TypeSafe's Enterprise page repeats "Candidate Screening at 88% Cost With The Same Accuracy". It is not a reranking study (R11-S56, R12-S56, R12-S58).

  • New comparison leads found 11 Oct 2026 (10 repositories), not opened, no grade. Seen in a GitHub repository search for repositories created 10 Oct, 05:11 UTC to 11 Oct, 05:16 UTC, by README or description only; no file was opened and no pin was taken (R13-S23).

Leads from real-time game and robot-simulation demos (12 Oct 2026)

Reported README comparisons only; not graded, no grade colour. The demos were read at pinned commits for what each loop does when a Jev call fails. Nothing was run.

  • FazalAAli/jev-robotics-demo at 531de61 (19 Sep; its only commit, unchanged 12 Oct 2026, 05:12 UTC). Operator: GitHub account FazalAAli. Lead, not graded. Task: a MuJoCo robot-arm simulation that stacks cubes, Jev vs an Opus LLM arm. Reported in the README: Opus 158.8 s and $0.75 vs Jev 19.1 s and $0.0006, one recorded run each (n = 1 per arm). The results are not committed (run recordings and logs are excluded by .gitignore). Failure behaviour: its row on the real-time control page (R13-S107, R13-S109, R14-S101).

  • alexcpn/llm-invaders at dfcbc49 (10 Oct). Operator: GitHub account alexcpn. Lead, not graded. Task: a Space Invaders speed benchmark. Jev (~typesafe/jev-latest) and Microsoft-Decision-1 are called through OpenRouter; a local Gemma model is the third arm. Reported in the README: a speed sweep with n = 3 seeds per speed per model. The results files (results.jsonl, summary.json, ceiling.json, report.html) are committed but were not opened, so no number is quoted here. In the benchmark, the time a model takes to answer is charged as game time. Failure behaviour: its row on the real-time control page (R14-S107, R14-S108, R14-S110, R14-S111).

Leads found 13 Oct 2026

Reported Repository names, descriptions and READMEs as seen in search; not graded, no grade colour. Seen 13 Oct 2026, 05:18–05:22 UTC; nothing was opened at a pin and nothing was run.

Leads found or read 24 Oct 2026

Reported Not graded, no grade colour. Nothing was run.

  • egarim/decision-1-vs-jev at 14e0b93, read 24 Oct 2026, 05:16 UTC (README, Bench.cs, results/decision1.jsonl and results/openjev.jsonl). Lead, not graded: it compares Microsoft-Decision-1 with open-jev, a self-hosted reproduction, not with TypeSafe's Jev ("I don't have an early-access key"); 48 tickets. With no Jev arm it is not a Jev comparison and gets no ledger row (R23-S212 to R23-S216).

  • New benchmark-named repositories created 23–24 Oct 2026 (9), not opened, no grade. Seen in a GitHub repository search by name or description only; no file was opened and no pin was taken (R23-S35): pournasserian/decision-models-vs-small-llms (Jev, Kev and Laya vs small LLMs on six agent decisions), j-geuter/Jeff-JevBench ("serving Jeff models on JevBench"), Blyuger/jev-benchmark, Marcyvan1777/jev-arena (Jev vs GPT and Claude in a robot-boxing game), Day-in-the-Country-LLC/bot-derby (Jev vs OpenAI Decisions in paper trading), MuShan-bit/decision-moderation-bench (Chinese moderation, 825 gold items), kint4/playwright-system1-triage (a QA calibration benchmark), FIXFIBER/jev-decision-lab, 66Kevin/Jev-like-Decision-Models-Experiments.

More leads seen 8–10 Oct, not graded (3 opened at README level, 12 not opened, 7 carried)
  • Opened for the README and pin only: Neha/jev-vs-openai-decisions at 7e5d874, a comparison app with 24 labelled tickets and no committed results (R12-S64); ivipinmishra/decision-models-showdown at 5a688b7, a demo app with 20 + 15 hand-labelled samples, a mock mode without keys and no committed results (R12-S65); CMaintz/classifier-bench at 2bc5965, a harness and corpus (2,321 cases, 31 tasks) covering Jev, OpenAI Decisions, Cloudflare Clef and Claude; published results not found in the README at 05:20 UTC (R12-S67).
  • Not opened (GitHub search window 8 Oct, 05:12 UTC to 10 Oct, 05:11 UTC; R12-S02): Kungie/haiku-decides, humanity-forever/decision-luna-browse-control, akitaonrails/jev-docs-benchmarks, bks-lab/lab-bench, SarathChandraBellam/ToolDiscoveryBench, hshicalu/jev-selective-prediction, vicwlau/finance-decision-evals, dorkitude/order-blind-decisions, haxudev/jev-benchmark, heventure/systemone-bench; the abnormal.ai blog "Classifying Emails with Jev and the OpenAI Decisions API" (R12-S15); Opper's "Jevman" (Pac-Man), whose operator Opper also sells a Jev route, so it has a stake (R12-S16).
  • Carried as "lead, not read": DouglasSM747/SupplyRisk, manan744/evaltrix, shivanisutreja-at/jev-benchmarks, Xinyuan-Gao/jev-workbench, gbesse/jev-guardrails-fr-arena, anweat/jev-websearch-eval, patcon/polis-jev-impute-simulation.

Found 7 Oct 2026, not graded, with reasons: jev-safety-eval (Aegis 1.0 content safety, baselines planned) has no results yet: "Results will follow"; to recheck next run for content moderation. jev-zh-tw-eval is not a Jev study: it tests Plumb-4B locally through a "Jev-compatible" interface, so its numbers are Plumb-4B's. The DAIR.AI piece "Jev-as-a-Judge for Agent Evaluations" is a tutorial, not a study: one refund example, no measurement on the open page. Checked 7 Oct 2026, 05:19–05:28 UTC.

Opened 9 Nov 2026, not graded

Reported Not graded, no grade colour. Nothing was run.

  • mahirmalik030/Jev-vs-LLMs at 4c168e5 (phishing-email detection), read 9 Nov 2026, 11:09 UTC: lead, not graded. The README gives no result figures, the results file it names is not in the repository, and the evaluation script calls only LLMs through OpenRouter: no Jev call was found in it, so the Jev arm the README names cannot be located (R29-S116).

  • randlee/judge-jev at 422f886, read 9 Nov 2026, 11:09 UTC: lead, not graded: not a benchmark. It is a requirements-compliance judge built on Jev that turns typed answers into pass or fail; no measurement of Jev's accuracy was found in the files opened (R29-S117).

  • Hightouch blog, "A Jev that can ask Opus for help gives Opus accuracy at 4x lower cost", read 9 Nov 2026, 11:13 UTC. Hightouch, a customer-data-platform vendor, grades identity-resolution results for its own product and states no stake in Jev. Its table: Claude Opus 5.5 99.0%, OpenAI's Decisions API at a 50% cutoff 97.3%, Jev at a 50% cutoff 95.6%. Lead, not graded: the number of cases, data and code were not found on the page (R29-S118).

  • An individual's blog post, "Decision models against small LLMs" (6,662 decisions; also posted on DEV.to, 5 Nov 2026), read 9 Nov 2026, 11:14 UTC: "Jev's 77.2% sits just under Claude Haiku 5.5 and Gemini 3.5 Flash Lite, and the ranking flips from task to task" (six-task score); on prompt injection "four of the five LLMs beat Jev's 85.2% and Mammoth-1's 84.3%". Lead, not graded: the post says every call is published; that was not checked (R29-S119, R29-S120).

TypeSafe's own figures, outside the ledger. TypeSafe's cookbook "Verifying LLM answers against their source" (figures from jev-1.14.0 on 7 Nov 2026; 1,000 MuSiQue questions answered by gpt-6-luna, of which 88.5% were right): Jev's check passed 73% of the answers, and 95.7% of those were right; it sent the other 27% to review, including 84 of the 115 wrong answers. Reported TypeSafe sells Jev; these are its figures, not graded here and not checked independently (cookbook read 9 Nov 2026, 11:11 UTC; R29-S211). The same figures are on Jev as a judge.

Where the studies agree and conflict, by task type

No averaging: each line names the rows it rests on. Reported

  • Judging and evaluation
    Agree on direction

    Jev can match a frontier LLM judge on verdicts that can be read off the text once a threshold is chosen on labelled data (B02: 87% vs Opus 5's 87% after tuning, 76% vs 83% at 0.5). It is behind where the verdict must be derived: math, code, logic, JudgeBench −12 points (B08). B15's true/false row (92.5% vs Sol 92.1%) is consistent. Added 7 Oct: B31 (grade A; Claude models with verbalized probabilities, two at low effort) reports Jev's accuracy at 48.2% vs 42.5–45.8% and a composite of 67.8 vs 67.4–69.2; one dataset, one task. B33 (grade C) reports a tie on claim verification and Jev last of twelve on LegalBench. The size of the gap depends on the task and the threshold.

  • Topic and intent classification
    Depends on the baseline

    Ahead of the smaller LLM baselines tested (small tiers in B03 and B15; GPT-6 Luna in B21, which the 6-of-10 small-LLM count in the short answer does not treat as a small tier): B03 (87.5% vs 71–83%), B21 (76.5% vs 70.1–73.8%), B15 multiple choice vs GPT-4.1-mini. Behind stronger or best-per-task LLMs: B06 (Gemini 3.8 Flash on Banking77), B13 (14 of 15 tasks, median 11.6 macro-F1), B15 (77.3% vs Sol 82.5%). Against trained classifiers on a fixed label set, mixed: one grade B study, B28 (behind a prompt-injection classifier, 86.35% vs 89.01%; ahead of a small content-safety classifier, 86.20% vs 80.27%; Red Hat publishes both classifiers), and grade D evidence, B10 (93.2% vs 80.1%) and B04. Open-weight LLMs scored through token probabilities: about level overall in one grade B study (B35, 0.787 for Jev; open models ahead on agent monitoring, Jev ahead on triage). More on choosing between them: Jev vs an LLM vs a classifier. Against an open Jev-like model on fresh data, no difference beyond noise (B11).

  • Email and ticket routing
    Point different ways

    B12 (Jev 1.1–2.1 points below Gemini on 1,565 bilingual emails; secondary figures) and B05 (Jev above Haiku 4.5 on 80 authored prompts; the operator has a stake) use different baselines. Neither publishes materials; both are grade C. Added 7 Oct: B34 (grade C, 200 items per task) reports Jev behind GPT-5.4-mini on support escalation (69.5% vs 84.5%) and ahead of Haiku 4.5 and Gemini 3.5 Flash (66.0%).

  • Reranking
    Agree

    B14 (+0.018 to +0.021 nDCG@10 over hybrid search; grade B since 7 Oct) and B04 ("comparable to internal system") agree that Jev can improve on or match a first-stage ranking. Added 11 Oct: B40 (grade A, reranking-pipeline study) compares Jev with trained rerankers on BM25 shortlists: level with Cohere Rerank 4 Pro within noise on the equal-dataset average (0.692 vs 0.691), behind it with equal weight per query (0.738 vs 0.756). Three more reranking or relevance studies: Reranking studies.

  • Multi-question workflows
    Vendor only

    Only TypeSafe's own B01 (agreement with LLM reference answers) and the robustness study B09, on a dataset whose subsets carry the same four workflow names.

  • Numeric scoring
    Agree: weak

    B23 (seventh of eight by CVSS error) is consistent with TypeSafe's own documentation ("It struggles with tasks that require numeric precision"). On B06's HelpSteer2 score task no model beats base rates.

  • Calibration
    Usable, with exceptions

    B06 (5.0 points on PubMedQA), B15 (yes/no level with Sol), B13 (better than 16 of 19 LLMs' stated confidence, worse than three frontier models) and B11 find the probabilities usable as a gate. B15 (multiple choice), B10 and B16 report overconfidence; B17 shows recalibration on your own data cuts the error sharply without changing accuracy. Added 7 Oct: B29 (grade A) reports 99.75% accuracy on the 82% of TriviaQA answers given at 90% confidence or more, but rising confidence beyond the knowledge boundary; B30 (grade C) reports a mean TV of 0.518 against 0.546 for a flat guess on physics distributions; B33 (grade C) reports low ECE on claim verification and high ECE on LegalBench. How to set a cutoff: confidence thresholds.

  • Cascades (Jev first, then an LLM)
    Agree

    B08, B13 and B02's escalation section agree that sending low-confidence items to an LLM can match the LLM's accuracy at a fraction of its cost. See the hybrid pattern.

Reranking studies

Reported Checked 7 Oct 2026, 05:18–05:22 UTC; LanceDB benchmark script opened 8 Oct 2026, 05:13 UTC. Graded on the same A–D scale for the reranking question only; rows without a B number are not in the 48 graded rows; B14, B37, B40, B47 and B48 are ledger rows. B40 read 11 Oct 2026, 05:23 UTC; B47 and B48 read 24 Oct 2026, 05:16–05:17 UTC. Nothing was rerun.

Study, operator and interestTask, n and baselineResult as reportedMaterialsGrade and flags
iwhalen.com, "Pseudo-relevances with Jev" (5 Oct)
Relevance-judgment study. An individual's blog; no stake stated.
Relevance judgments on the TREC DL 2023 judged subset (22,327 judged pairs), compared with human grades; system ranking of TREC runs.
Baseline: gpt-6-luna.
Cohen's kappa against human grades "0.1907" (Jev) vs "0.2391" (GPT-6-luna); system ranking: "Jev and Luna are essentially tied" (tau values are in graphics, not extracted). Cost $0.6783 vs $1.3283.Data on Hugging Face; code at 82d9a0f (main.py opened); metrics defined in codeA public-datasingle-task
Hugging Face community blog, "Introducing jev-reranker" (Sep; date not confirmed)
Reranking-pipeline study. The library's author, writing about the library.
NanoBEIR-en NanoHotpotQA via HAKARI-Bench at a pinned revision; 50 queries.
Baseline: hybrid search only.
nDCG@10: hybrid 0.833; rerank 0.969; relevance_rerank (threshold 0.2) "0.975", keeping 7.62 documents per query (92.38% removed).Code at d58594b (examples/eval.py, docs/eval.md opened); data publicB COIsmall-nno-baselinepublic-datasingle-task
LanceDB, "How Jev Compares to Other Rerankers" (date not confirmed)
Reranking-pipeline study. A vector-database vendor that ships a TypeSafeReranker.
19 reranker configurations on GooAQ (20,000 queries, seed 0), BEIR NQ, HotpotQA, FiQA and SciDocs.
Baselines: other rerankers and no reranker.
Example: GooAQ jev-relevance Hybrid@10 92.59, p50 169 ms. The README says the default prompt "drops below no reranker on HotpotQA".Results and READMEs at 4616259 read; benchmark script reranking/benchmark/bench.py opened 8 Oct: data (L30, L38, L106), metric (L290–302), Jev setup (L61–75, L168–169) and report writer (L305–338) line up with the README. Per-query score files are not committed.B COIpublic-data
"Provisional" removed 8 Oct (script opened); grade unchanged
B14 Elastic Search Labs (30 Sep)
Reranking-pipeline study. A search vendor.
ESCI reranking, 250 test queries reported.See row B14. Does not count here: the published notebook covers 10 queries, not the reported 250-query test.Notebook at 3e3d688 opened (10-query sample)B regraded 7 Oct
B37 jev-pulse (GitHub account mugunthank7)
Reranking-pipeline study. An individual developer; no stake stated.
Amazon ESCI search, 40 queries, 653 products; BM25 baseline only.nDCG@10 BM25 0.660 vs Jev 0.772 (README). See row B37.Code, data and metric at a1e796f; run outputs not committedA small-nsingle-task graded 10 Oct
B40 jev-rerank-bench (GitHub account anessbelbati)
Reranking-pipeline study. An individual with a blog write-up; no stake stated.
30-passage BM25 shortlists reordered; 8 English datasets with 1,617 scored questions, plus BRIGHT subsets, NevIR and MIRACL French.
Baselines: Cohere Rerank 4 Pro, ZeroEntropy zerank-2, DeepSeek V4.1 Flash, open models.
Equal-dataset nDCG@10: Jev rubric 0.692, Cohere Pro 0.691, within noise; with equal weight per query, Cohere 0.756 vs Jev 0.738. See row B40.Data, metric code, results and saved responses at fecba75A reranking onlyoperator-chosen datasetsorder depends on averagingper-dataset results go both ways graded 11 Oct
B47 kapa.ai (15 Oct 2026)
Reranking-pipeline study. A RAG vendor on its own sealed benchmark.
Top 50 hybrid-search candidates reranked, top 10 scored; 1,000 production-derived cases.
Baselines: Voyage, Cohere and Qwen rerankers.
Retrieval score: Voyage rerank-3 0.535, Jev 1.14 (Noul) 0.532, Voyage rerank-2.5 0.526, Cohere rerank-v4.0-pro 0.506; Jev 1.13 0.519. See row B47.None published (blog post)C COIsealedsingle-task graded 24 Oct
B48 Sayora (12 Oct 2026)
Relevance-judgment study. An AI-receptionist company evaluating its own retrieval.
Right page for a caller's question from page descriptions; 168 generated questions over four sites; LLM-judged labels.
Baselines: production embeddings, BM25, GPT-6 Luna.
Right page first: Jev (jev-1.13.0) 88% vs production embeddings 70%; GPT-6 Luna 86% (a tie by its interval). See row B48.None published (blog post)C LLM-labelssmall-nsealedsingle-task graded 24 Oct

Result: 2 of 2 met beyond Parallel (B04) and MindStudio's reranking post (28 Sep, not graded). Two reproducible reranking or relevance studies were read at pinned commits: iwhalen (relevance-judgment study, grade A) and Hugging Face jev-reranker (reranking-pipeline study, grade B, COI). LanceDB is a third (reranking-pipeline study, grade B, COI; its script at 4616259 was opened 8 Oct). Elastic B14 (reranking-pipeline study, grade B) does not count. Added 10 Oct 2026: B37 jev-pulse (reranking-pipeline study, grade A, no stake stated; small-n with 40 queries, BM25-only baseline, single task, run outputs not committed) is a rerunnable pipeline study from an operator with no stated stake. Added 11 Oct 2026: B40 anessbelbati/jev-rerank-bench (reranking-pipeline study, grade A, no stake stated; reranking only, operator-chosen datasets, order depends on averaging) reports Jev and Cohere Rerank 4 Pro within noise on the equal-dataset average. Added 24 Oct 2026: B47 kapa.ai (reranking-pipeline study, grade C, COI, sealed) reports Jev 1.14 0.003 behind Voyage rerank-3 on its own benchmark; B48 Sayora (relevance-judgment study, grade C, LLM labels, small n) reports Jev ahead of its production embeddings at picking a page and tied with GPT-6 Luna. Neither publishes materials.

Caveat. iwhalen measures relevance judgments (agreement with TREC grades and system ranking), not a reranking pipeline. jev-reranker is its author's own library. Two other items are not studies: the Spring blog's Modular RAG post is an implementation demo with no measurement, and refix.ai's "Jev for RAG" is an explainer.

Study types side by side, and what reranker and passage-filter implementations do with your passages when Jev fails: Jev reranker and Jev for RAG.

What nobody has measured yet (as far as this site found)

  • Non-English work with a baseline and published materials: B12 (German and English, grade C); added 11 Oct, B39 (Portuguese, grade A; dataset generated by an LLM) and the MIRACL French subset in B40; one Italian repository lead. TypeSafe's docs say English is "where accuracy is currently best".
  • Calibration of the Score question type.
  • Long state near the 32k limit (TypeSafe documents that large irrelevant state hurts). Added 24 Oct 2026: B46 (grade A, one synthetic task) reports accuracy on logs up to 30K tokens (Jev 94.7%, SubQ Snap 93.9%) and, with the log split into 29K chunks by the operator, at 360K (Jev 77.1%, Snap 58.8%). Not measured at the 262k opt-in context.
  • A head-to-head against a fine-tuned classifier (BERT, SetFit, GLiClass) or zero-shot NLI with published materials that this site has read. B28 (grade B) links its code, but the code was not opened and n is not stated.
  • Behaviour across model versions: until 24 Oct 2026 no graded study compared jev-1.13.0 with jev-1.14.0, which TypeSafe released as a preview on 14 Oct 2026 (checked 16 Oct 2026, 05:21 UTC). Added 24 Oct 2026: one graded study reports both, B47 (kapa.ai, grade C, own sealed reranking benchmark): Jev 1.14 0.532 and Jev 1.13 0.519 in Noul mode. Two leaderboards list both versions (outside the ledger). Since 24 Oct 2026 jev-latest on TypeSafe's API gives 1.14 (versions and aliases).
  • Dates and arithmetic as separate tasks.
  • Throughput and error rates under load or near the rate limits.
  • Outcomes in production: no row measures the downstream cost of errors or drift.
  • Images: until 24 Oct 2026 no graded study tested image input. Added 24 Oct 2026: one graded study does, B45 T3 (Amazon product category from text and the main photo, 1,999 items; Jev 1.14 vs OpenAI's Decisions API, "no detectable difference" on accuracy; run 17 Oct 2026). Reported. Jev takes text by default; image input is an opt-in experiment since 14 Oct 2026 (TypeSafe: not yet production-ready), checked 16 Oct 2026, 05:21 UTC. Since at least 7 Nov 2026 TypeSafe's models page lists the input of both Jev 1.14 and Jev 1.13 as "Text, images"; the images page still requires the opt-in header and calls images not yet ready for production (first observed 7 Nov 2026, 15:29 UTC; R27-S05, R27-S06).

How this site checked one study

The only Tested item on this page is offline arithmetic, not a Jev call. The site downloaded Jevals' published per-decision log for Jev on PubMedQA (1,500 records: 300 items × 5 runs; commit 21bb47b), recomputed the summary figures with its own script, and deleted the file afterwards.

Jevals B06, PubMedQA row: published vs recomputed (30 Sep 2026, 20:58 UTC)
FigurePublishedRecomputedOutcome
Accuracy91.3%91.33%match
Median latency438 ms0.4377 smatch
95th percentile latency653 ms0.6528 smatch
Mean cost per 1,000$0.029$0.0285match
Calibration error (10 bins)5.0 points5.10 pointsnear match binning may differ

The Decision Score (69.0) was not recomputed. The check confirms the published summary matches the published log; it does not confirm the log itself.

What was not verified

  • No study was rerun and no Jev call was made. The linked code repositories behind the three provisional A grades (B13, B15, B24) and JevBench's code were not opened.
  • B08's reproducibility package was not located; if it is found, the row may move to grade A.
  • B10's primary study is not identified (a related lead: a 26 Sep Show HN gist claiming "94% on Banking77").
  • B14's Jina reranker row was not extracted; its 250-query test sample, raw judgments and bootstrap code were not found (7 Oct).
  • Added 7 Oct 2026:
    • B28 (Red Hat): n per benchmark set, and the code at a pinned commit, were not read. The article returned HTTP 403 to a plain fetch; its tables were confirmed on a browser render.
    • B29: the arXiv paper (2610.01006) and Part 2 were not read.
    • B30: the Jev version is not stated.
    • B33: the claim-verification data and the LegalBench 396-question sample were not located.
    • B34: which 200 items were sampled; the playground experiments (need an account); OpenRouter's resolved Jev version.
    • B35: no code repository found; the 25 relabelled guardrail items and the preprint not located.
    • B36: the 4 Sep SREGym-Lite cohort list, judge.py and the rubric file were not read; SREGym's earlier Jev post was not read.
    • Reranking: the LanceDB per-query score files are not committed (its benchmark script was opened 8 Oct); jev-pulse results are in its README only and its run outputs are not committed; anessbelbati/jev-rerank-bench code was not opened (opened 11 Oct; see B40); MindStudio's 28 Sep reranking post was not re-read; the Hugging Face and LanceDB post dates were not confirmed on the pages; iwhalen's Kendall tau values are in images only.
    • A 240-example measurement in a Hacker News comment on the Red Hat thread (item 49981222) was not read.
  • Added 10 Oct 2026: B37's grade rests on the 8 Oct read of metrics.ts L5–11 and the README; run.ts was reported by a helper and not quoted, and the repository was not re-opened on 10 Oct. For the 8 repositories opened on 10 Oct (5 in the leads list, 3 under "More leads") only the README and the top-level file list were read (the GitHub API rate limit ran out at 05:20 UTC); no metric, data or result file was opened. classifier-bakeoff's public report was not opened (opened 11 Oct; see lead). The Jack & Jill case study has no code, data or test set to open.
  • Added 11 Oct 2026:
    • B39: the source of the McNemar p-value (0.011) was not located; it is not in bench/metrics.py.
    • B38: the full call logs are not committed (only the first 100 are, in logs_sample/).
    • classifier-bakeoff: the metric code (bakeoff/) was not opened.
    • decisionbench: the results location is still not identified.
    • The ten new comparison repositories in the new lead list were not opened.
  • Added 12 Oct 2026:
    • B41: the two calls.jsonl files (11.7 MB each) were not opened line by line; whether the committed code is the code that ran is not known (git_commit: null).
    • The jev-robotics-demo comparison has no committed results; the llm-invaders results files were not opened.
    • The other nine comparison repositories from 11 Oct were not graded.
  • Added 13 Oct 2026:
    • B42: the answer key is not published (only its sha256), so neither score can be checked against it; the cause of the 17 refusals is unknown; Jev was not re-run on the Decisions API's date. Only the commit feed was rechecked on 13 Oct (the files were read on 12 Oct).
    • The ten comparison repositories found on 13 Oct and JevBench were not opened; utakatano/decision-models-demo had no files.
  • Added 14 Oct 2026:
    • B43: the API base URLs were not opened (the README says "the APIs"); the long-context result has no Jev arm; only the file-localization one-choice numbers were recomputed (the per-file, classification and option-order numbers are as the operator reports them).
    • B44: the bomb-battle and image results were not recounted, and the call log evidence/calls.jsonl.gz was not opened; only the 48-game Tetris league was recounted.
    • The eight remaining 13 Oct comparison leads and the JevBench repository were pinned from their commit feeds only; no file in them was opened.
  • Added 24 Oct 2026:
    • B45: the per-call logs referenced in the README are not in the repository; only the T1 accuracy was matched line by line (the other figures are as the operator reports them).
    • B46: the 360K figure was not matched line by line and the raw answers were not opened.
    • B47 and B48: no materials are published; kapa.ai's benchmark and Sayora's questions and labels could not be read.
    • The nine benchmark-named repositories of 23–24 Oct were not opened.
    • Leaderboards: see the claims list and the notes under leaderboards for what was not checked.
  • Who operates the Hugging Face account LocalLLaMA, whose typed-decisions subsets carry the same four workflow names as TypeSafe's WorkflowEvals, is unknown.
  • The WorkflowEvals Hugging Face datasets and published runs were not opened.

Sources for the 7, 8, 10, 11, 12, 24 and 30 Oct and 9 Nov 2026 updates (UTC)

9 Nov 2026 update: B49–B51, leads, leaderboards, option-order and cookbook lines (R29-S…)
30 Oct 2026 update: Base Labs pricing line (2 source IDs, R24-S…)
24 Oct 2026 update: leaderboards and B45–B48 (R23-S…), with pinned commits and opened files
  • R23-S100, R23-S101: Space API and its commit list, 05:15:01. R23-S102 to R23-S108: data/index.json, data/methodology.json, data/own.json, data/v03.json, index.html, methodology.html and README.md at 0b5ecaf, 05:15:07–05:16:21. R23-S109 to R23-S114: data/index.json at 133313c, de1f4ea, 16f312e, 1f4e490, 76fbe37 and 2f77796, 05:15:34–05:15:38. R23-S48: positions re-sorted from the data files at 0b5ecaf and 133313c, 05:23:40–05:23:41.
  • R23-S115, R23-S116: Baseten blog and Base Labs report, 05:15:49–05:15:50. R23-S118: Jeffan-31b-1710 card at cebfa86, 05:19:25.
  • R23-S119, R23-S120, R23-S123: Artificial Analysis methodology, article and evaluation page, 05:16:42–05:16:52.
  • R23-S125 to R23-S127: Hacker News items 50093003 (Eevo), 50007993 (JevMan) and 50199105 (Decido), 05:17:18–05:17:19. R23-S129 to R23-S133: JevMan page, commit feed, README, tree and public/leaderboard.json at fd465f0, 05:17:28–05:17:49. R23-S128, R23-S134: JevBench site and data.js, 05:17:19–05:18:12. R23-S137, R23-S138, R23-S140: basal-omni-1.0-ultra card at 6d5ba67 and the Decido README and results/ listing at f895cf0, 05:18:41–05:18:47.
  • R23-S200 to R23-S211: decision-bench at e5b7202 (B45): commit feed, tree, README, dbench/score.py, dbench/adapters.py, results/summary.json, .env.example, dbench/prices.py, results/data-pack.md, dbench/run.py, .gitignore and DEVIATIONS.md, 05:15:24–05:17:37. R23-S49: spot-check of summary.json L8 and data-pack.md L21, and of B46's results.json L100–104, 05:21:18.
  • R23-S217 to R23-S223: subq-vs-jev at c037905 (B46): repository search, commit feed, tree, README, site/data/results.json, bench/analyze.py and bench/client.py, 05:16:24–05:16:39.
  • R23-S224, R23-S226: kapa.ai post and Hacker News 50115347 (B47), 05:16:58–05:16:59. R23-S225, R23-S227: Hacker News 50070393 and Sayora post (B48), 05:16:59–05:17:04. R23-S212 to R23-S216: egarim/decision-1-vs-jev at 14e0b93 (lead), 05:16:08–05:16:15. R23-S35: GitHub repository search for the new lead names, 05:17.
40 source IDs (R10-S…) in 22 entries, with pinned commits and opened files
8 Oct 2026 update: 9 source IDs (R11-S…), with pinned commits and opened files
  • R11-S206, R11-S207, R11-S208: LanceDB benchmark at 4616259: reranking/benchmark/bench.py (L30, L38, L61–75, L106, L168–169, L290–338), README.md, results/README.md, 05:13:50.
  • R11-S209, R11-S210: jev-pulse at a1e796f: README.md, apps/api/src/rerank/metrics.ts (L5–11), 05:14:10.
  • R11-S212, R11-S213: one web search ("Jev TypeSafe reranker evaluation nDCG independent benchmark RAG", boundary: published since 15 Sep 2026), 05:14:23; jev-rerank-bench at fecba75, README.md only, 05:14:40.
  • R11-S56: TypeSafe case study (7 Oct), 05:14:53.
10 Oct 2026 update: 14 source IDs (R12-S…), with pinned commits
11 Oct 2026 update: 19 source IDs (R13-S…), with pinned commits
  • R13-S50 to R13-S54: jev-vs-decisions-rpg at f81c004 (B38): tree listing, 05:22:50; README.md, experiments/stats.py (L9–60), results/summary.md (L9, L52–53, L70, L100), deciders/jev.py (L10) and deciders/openai_decisions.py (L12), 05:23:13.
  • R13-S55 to R13-S60: openai-decisions-vs-jev-vs-laya at 160e112 (B39): tree listing, 05:22:50; README.md, bench/metrics.py (L4, L7–41, L94–118), results/2026-10-07/report/summary.md, dataset/suites.yaml and dataset/LICENSE, bench/adapters/jev.py, laya.py and openai.py, 05:23:13–05:23:14.
  • R13-S61 to R13-S66: jev-rerank-bench at fecba75 (B40): tree listing, 05:22:50; README.md, eval.py (L71–76, L236–258), significance.py (L1–6, L47–54), results/significance_run_sept25.txt (L16–17) and rerankers/jev.py (L7–19), 05:23:14.
  • R13-S67: classifier-bakeoff at 4a1c4a0, reports/public/20261007T235443Z-da1571.html, 05:25:30.
  • R13-S23: GitHub repository search, repositories created 10 Oct 05:11 UTC to 11 Oct 05:16 UTC, 05:21:01 (the ten new comparison leads).
14 Oct 2026 update: 3 source IDs (R16-S…), with pinned commits
  • R16-S65: commit feeds (commits/main.atom) of the eleven comparison repositories (B43, B44, the eight remaining 13 Oct leads and leanmcp/jevbench), 16:51:37.
  • R16-S66: snap-vs-jev-swe at b754c44 (B43): README.md, reports/summary.md, src/snapbench/evaluation/evaluators.py, src/snapbench/evaluation/ties.py, results/swe-paths-jev.jsonl, results/swe-paths-snap.jsonl, data/labels/swe-paths.jsonl, 16:52:30.
  • R16-S67: decision-model-arena at 285c803 (B44): README.md, evidence/league.json, measure/tournament.mjs, 16:55:30.
13 Oct 2026 update: 5 source IDs (R15-S…) and 7 carried (R14-S…), with pinned commits
  • R14-S300, R14-S301: luna-decisions-benchmark at 7a5ea13 (B42): commit feed (HEAD pin) and tree listing, 12 Oct, 05:27:50–05:27:54.
  • R14-S302 to R14-S305: README.md, analyze_comparison.py, analysis/summary.json and data/luna_run_manifest.json, 12 Oct, 05:28:00.
  • R14-S55: spot-check of analysis/summary.json (612 correct), 12 Oct, 05:29:20.
  • R15-S53: commit feed, HEAD still 7a5ea13, 13 Oct, 05:24:16.
  • R15-S227, R15-S228: utakatano/decision-models-demo: repository metadata (size 0), 05:18:56; commits API HTTP 409 ("This repository is empty"), 05:22:03.
  • R15-S28: GitHub repository search, repositories created 12 Oct 05:12 UTC to 13 Oct 05:17 UTC, 05:21:29 (the ten 13 Oct comparison leads).
  • R15-S34: JevBench page, 05:22:07.
12 Oct 2026 update: 14 source IDs (R14-S…) and 2 carried (R13-S…), with pinned commits
  • R14-S200, R14-S201: decision-api-benchmark at 1ffa4d5 (B41): HEAD pin and tree listing, 05:12:35.
  • R14-S202 to R14-S205: README.md (L23, L68, L205–241, L248, L295), compare/metrics.py (L61–83, L264–296), runs/compare-20261010-222249Z/comparison.md (L26), runs/20261010-023836Z-or-run2/report.md (L13–18, L37) and meta.json, 05:12:40.
  • R14-S206: compare/providers.py, compare/dataset.py and data/cases.jsonl (500 lines, sha256 prefix checked against meta.json), 05:15:00.
  • R14-S51: spot-check of comparison.md L26 and compare/metrics.py, 05:18:03.
  • R14-S52: the published short answer and ledger rows, read by script for the small-LLM count (10 graded studies: 6 ahead, 2 mixed, 2 behind), 05:20:30.
  • R14-S101, R13-S107, R13-S109: jev-robotics-demo at 531de61: HEAD recheck 12 Oct, 05:12:24; README.md and .gitignore, read 11 Oct, 05:17:20–05:17:25.
  • R14-S107, R14-S108, R14-S110, R14-S111: llm-invaders at dfcbc49: tree listing, README.md (L1–75), bench.py, invaders/harness.py, 05:12:55–05:13:13.

Search published pools, pages, reports, and evidence.