Shaduf.Research preview
CompareLedger rows and docs checked · Decisions API graded comparisons updated · B42 added · B44 added · Jev input and context lines rechecked · Decisions API status · B45 added and hosted-API list recounted

Jev (TypeSafe AI) vs an LLM with JSON output vs a trained classifier: when to use which

For a classification, routing, gating or scoring task you can call Jev, ask a general LLM for structured output, train a classifier on your labels, or use a zero-shot NLI model. This page gives ten criteria for the choice, the published head-to-head results behind them, and a combined pattern several studies support.

Short answer

Use Jev when you have few labels or the labels change, and you want a probability you can threshold. Use an LLM when you need text, long inputs, arithmetic or the best accuracy on hard judgments. Use a trained classifier (or an open local model) when you have thousands of labels for a fixed set, need CPU or on-device inference, or cannot send data out. Use Jev first, then an LLM for low-confidence items when you want LLM accuracy at lower cost.

All benchmark evidence here is Reported by the operators named on the benchmarks ledger; this site ran no test. There are no prices on this page: compare costs on cost per decision and with the worksheet there.

How strong is the classifier comparison?

Until 7 Oct 2026 the comparison between Jev and trained classifiers or zero-shot NLI rested only on grade D rows: B04 (Parallel: "Internal 'wins'" on topic and freshness, no n) and B10 (MindStudio's secondary summary of an unnamed study). No graded study with published materials compares Jev with a fine-tuned classifier such as BERT, SetFit or GLiClass, or with zero-shot NLI (B28 below uses its publisher's own guardrail classifiers; its code was not read). Criterion 1's "thousands of labels" advice is therefore a reasoned expectation plus weak evidence, not a measured result.

Added 7 Oct 2026: the ledger now has one grade B study with trained-classifier baselines, B28 (Red Hat, which publishes both classifiers; code not read, n not stated). Its result is mixed: Jev was behind a prompt-injection classifier and ahead of a small content-safety classifier.

Ten selection criteria

Each row says which approach it favours, why, and the evidence with its label. Rows are ordered by how early the question usually comes up, not by weight. Documented rows cite TypeSafe's docs (read 30 Sep 2026, 20:47–20:48 UTC); Reported rows cite ledger studies.

CriterionFavoursWhyEvidenceStill unknown
1. Labelled data: none, some or thousandsNone or some: Jevor LLMThousands, fixed labels: classifierJev and LLMs need no training. A few hundred labels make a threshold reliable. With thousands of labels for a fixed set, a trained model learns the label boundary directly and is likely to win on accuracy, speed and cost.B02 thresholds tuned on half the labels; B17 recalibration Reported. B10 encoder 93.2% vs Jev 80.1% and B04, both grade D ReportedNo graded study with materials against a fine-tuned classifier
2. Labels or options change at run timeJevor LLMOptions and criteria are sent with each request; a trained classifier must be retrained. Caution: renaming options can change Jev's answers.TypeSafe primitives docs Documented; B04 "specify the question and labels at request time"; B09: no/yes names flip 32.5% of decisions vs 0/1 ReportedHow much accuracy moves when options are edited in practice
3. You need a free-text explanation or open outputLLMJev returns no text. TypeSafe lists "Generation: Use a generative model" among Jev 1.13's failure modes.Jaggedness page, failure mode 9 Documented; limitations—
4. Latency budget; CPU-only or on-deviceTight hosted budget: JevCPU or on-device: classifier, NLI or open modelStudies report Jev medians of about 0.13–0.6 s per call, several times faster than LLM calls. Jev has no public weights, so it cannot run on-device.B05, B06, B08, B15 Reported; B10 encoder 8 ms on CPU (grade D); no weights DocumentedLatency under load and near the rate limits
5. Data residency or offline requirementClassifier, NLI or open local modelJev runs only as a hosted API, from TypeSafe or gateways that route to TypeSafe. TypeSafe offers zero data retention directly "for enterprise customers"; some gateways label the TypeSafe provider zero-retention. Data still leaves your system on every route.TypeSafe legal index; route terms quoted on data handling by route (1 Oct) DocumentedTypeSafe subprocessors; EU residency not offered at the TypeSafe layer
6. You need a probability you can thresholdJevor LLM logprobs calibrated on your dataJev returns a probability per option. Most studies find it better calibrated than LLMs' stated confidence, but not everywhere.B06, B13, B15; B21 calibrated logprob adapter ≤0.10; B13 empathy task and B16 overconfidence Reported; see confidence thresholdsCalibration of Score; drift across versions
7. Input length and contextLong inputs: LLMBy default Jev takes 32k tokens for state plus the longest question and 64k per request; an opt-in long-context header raises both to 262k, experimental and not production-ready per TypeSafe (16 Oct 2026). TypeSafe says large state full of irrelevant detail hurts accuracy.Models page and jaggedness page Documented; limitationsNo independent long-state measurement
8. Arithmetic, dates, other languages, imagesCode plus an LLMnot Jev aloneTypeSafe documents weakness on math, counting and date comparison; its models page said English is the primary training language until at least 2 Nov 2026 (that section is gone on 7 Nov 2026, and TypeSafe's changelog lists "Better multilingual support" for Jev 1.14; no multilingual accuracy figures are published); images only as an experimental opt-in since 14 Oct 2026, not production-ready per TypeSafe (no image input before that; 16 Oct 2026, 05:22 UTC). Since at least 7 Nov 2026 (first observed 7 Nov, 15:29 UTC; the change happened after 2 Nov, 11:13 UTC) TypeSafe's models page lists the input of both Jev 1.14 and Jev 1.13 as "Text, images" and says Jev "also reads PNG and JPEG images anywhere in the request"; the images page still calls images and Box "not yet ... ready for production" and requires the TypeSafe-Danger-Zone-Images: true header (R27-S05, R27-S06).Jaggedness and models pages Documented; B08 math/code/logic, B23 numeric severity, B12 the only bilingual row ReportedGraded non-English and date studies
9. Volume and costHigh volume: Jev over a frontier LLMcheapest LLM setups or self-hosted classifiers can be cheaperPer-call cost differences are large against frontier LLMs, small against the cheapest LLM setups, and self-hosting changes the picture at scale.B03, B04 Reported. Figures: cost per decisionYour own token counts; Jev's fixed per-request token overhead is on the cost page
10. Vendor dependence and access riskClassifier or open modelOne vendor. Signups were paused 22–27 Sep; new-user credit is disabled; rate limits "can change without notice" and changed this week; every gateway routes to TypeSafe.Access status (30 Sep) Documented. A hosted "decision API" from OpenAI was reported on 29 Sep (The New Stack headline) Reported; OpenAI's own guide was read on 8 and 10 Oct 2026 (Decisions API, then a public beta; R11-S300, R12-S134); OpenAI's Decisions API became generally available on 22 Oct 2026 (OpenAI API changelog; guide read 23 Oct 2026, 05:23 UTC; R22-S32, R22-S47) DocumentedHow the Decisions API compares with Jev on your task: five graded comparisons (B38, B39, B41, B44, grade A; B42, grade B) cover other tasks and differ by task (14 Oct 2026); on 24 Oct 2026 six, with B45 (grade A, five public classification tests) (see "What was not verified" below; side by side)

A combined option: Jev first, then an LLM

Cascade pattern supported by three studies

  1. 1. Every item goes to JevJev returns an answer and a probability for each option.
  2. 2. Compare with a thresholdChosen in advance on your own labelled data. Above it, your code acts on Jev's answer.
  3. 3. Low confidence goes to an LLMOr to a person. Only this share pays LLM prices.
The threshold must be chosen on your own labelled data before use; confidence thresholds gives a procedure.
  • B08 (CMU): with a threshold frozen in advance, the cascade escalated 31% of 1,610 held-out pairs and was 0.9 points more accurate than GPT-6 at 41% of its fee.
  • B13 (NYU Abu Dhabi): routing low-confidence items to an LLM "matches or exceeds the LLM alone at a quarter to half of its cost".
  • B02 (Arize): an escalation recipe in its judging study.

Reported Each figure is the operator's own; the escalation share and savings depend on the task and threshold.

Head-to-head results from the ledger

Rows where Jev is compared with an LLM or a classifier. Each row shows who won on the operator's own metric. This is not a ranking, and the rows must not be summed. Reported

Row and taskWho won, on which metricOperatorGrade
B01 Four multi-question business workflowsOpus 5 and Sol above Jev on agreement with LLM reference answers; Jev level with Sonnet 5 and Terra; Jev lowest cost and time per caseTypeSafe (vendor)B COILLM-labels
B02 Hallucination detection; summary qualityTie with Opus 5 after threshold tuning (Opus ahead at 0.5); Jev slightly ahead on SummEval correlation; Terra behindArizeB
B03 AG News topics (n = 200)Jev ahead of gpt-4.1-nano, gpt-4o-mini and Luna on accuracy; nano with one token cheaper per decisionNavyaAIC
B04 Reranking; topic; freshnessTie with an internal reranker (NDCG@10); internal classifiers win on topic and freshnessParallelD
B05 Routing tiers (80 authored prompts)Jev ahead of Haiku 4.5 on match rate, latency and costLiteLLM (gateway)C COI
B06 PubMedQA, Banking77, HelpSteer2PubMedQA tie with Gemini 3.8 Flash; Banking77 Gemini ahead; HelpSteer2 nobody above base rateJevals (independent)B
B08 LLM-judge benchmarksWithin 3 points of GPT-6 on text-readable verdicts; GPT-6 ahead on math, code, logic; cascade ahead of GPT-6CMU (academic)B
B10 Banking77, Yelp and two moreTrained encoder ahead on Banking77; Jev ahead on Yelp; zero-shot NLI behind JevMindStudio (secondary)D
B12 Business email, 10 categories, German and EnglishGemini 3.5 and 3.8 ahead on accuracy (secondary figures)Bryo AIC
B13 18 social-science annotation tasksBest LLM per task ahead on 14 of 15 (macro-F1); Jev's confidence better than most LLMs' stated confidenceNYU Abu Dhabi (academic)A provisional
B15 True/false, multiple choice, injectionTrue/false: Jev level with Sol, ahead of GPT-4.1-mini. Multiple choice: Sol aheadpunk2898A provisional
B21 13 public datasetsJev ahead of a DIY logprob adapter and prompted JSON on average; DIY close on category taskskgluszczykC
B23 CVSS severity scoringGPT-6 Astra first; Jev seventh of eight by errorCascoC
Carried: general LLM with structured outputNo score. OpenAI's Structured Outputs guarantees schema conformance, not decision semantics or probabilitiesOpenAI docs (22 Sep report)Documented
Carried: GLiClassNo score. A local zero-shot or multi-label classifier; no Jev head-to-head recordedAlternatives hubNo score

Named open Jev-like models (Kev, Laya, CLM and others) are on the alternatives hub; hosted Jev, Laya and Kev are compared on the same rows on Jev vs Laya vs Kev; the one operator comparison of Jev and Kev is B11.

Other providers' hosted decision APIs are a separate layer: on 12 Oct 2026 the alternatives page listed 3 (OpenAI's Decisions API, then a public beta; Microsoft-Decision-1; Cloudflare's Clef family); on 24 Oct 2026 it lists 8: OpenAI's Decisions API (generally available since 22 Oct 2026), Microsoft-Decision-1, Cloudflare's Clef family, SubQ Snap, Liquid AI d1-ultra, Mouse-1, Eevo and Drex (hosted API from another provider).

What was not verified

  • No approach was tested by this site on any task. The criteria come from TypeSafe's documentation and from other operators' published results.
  • No graded comparison with a fine-tuned classifier or zero-shot NLI exists with published materials (see the caveat above).
  • OpenAI's Decisions API was read in OpenAI's own documentation (10 Oct 2026, 05:12 UTC) but not called. Update, 11 Oct 2026: two comparisons with Jev are now graded A from files opened at pinned commits, and they differ by task: the Decisions API ahead on RPG move choices (B38, komzweb); Jev ahead on exact judge score in Portuguese, level on routing and injection (B39, mateusfonsek). Results Reported; no overall parity is stated. On 10 Oct they were leads with metric code not yet opened. Update, 12 Oct 2026: a third graded comparison. On synthetic bank-withdrawal routing (B41, grade A), Jev (89.2% macro-F1) and Microsoft-Decision-1 (87.6%) were not separated and both were ahead of OpenAI's Decisions API (71.4%); Decision-1 kept the same route across all 6 option orders more often (97.2–97.6% vs Jev 91.6–91.8%, Decisions API 64.2%). (B41, A, kartikanand73/decision-api-benchmark at 1ffa4d5; flags: synthetic single task; thresholds set with Jev-shaped probabilities, the operator's own note; all three models called via OpenRouter; code version not pinned; checked 12 Oct 2026, 05:12 UTC; R14-S201–S206.) Reported The three results differ by task; no overall direction is stated. Update, 13 Oct 2026: a fourth graded comparison. On 1,000 five-option logic questions (B42, grade B), Jev answered 817 correctly and OpenAI's Decisions API 612; Jev's answers come from a 22 Sep run, and the answer key is not published. (B42, B, GitHub account etsabary, etsabary/luna-decisions-benchmark at 7a5ea13, 1,000 five-option deterministic logic questions (25 families); Decisions API called as openai/gpt-6-luna-decisions via OpenRouter; flags: via OpenRouter's alpha Decisions endpoint, not OpenAI direct; Jev not re-run (22 Sep vs 10 Oct; mixed Jev providers, 256 answers without a verified snapshot); answer key withheld (only its sha256 committed); 17 persistent refusals (HTTP 502), cause unknown; the operator built both benchmarks; young repository (2 commits; HEAD changed after the main publish commit); latency and cost not comparable across dates and routes; re-pin checked 13 Oct 2026, 05:24 UTC; R14-S300 to R14-S305, R15-S53). Reported Update, 14 Oct 2026: a fifth graded comparison. In a 48-game Tetris round robin (metric: games won; 8 games per pair), Microsoft-Decision-1 won 15, Jev 12, OpenAI's Decisions API 11 and the open Laya model 8, with 2 draws; Jev beat the Decisions API 6–2 head to head (B44, A, dante01yoon/decision-model-arena at 285c803; flags: small-n (8 games per pair); single task (games); Decision-1 through OpenRouter's alpha endpoint, not Foundry; Jev called as the jev-latest alias; one date and region (called from the US); files opened 14 Oct 2026, 16:55 UTC; R16-S67). Reported The five results differ by task; no overall direction is stated. Update, 24 Oct 2026: a sixth graded comparison. B45 (gbolask24, grade A; Reported): on five public-data tests run 17 Oct 2026, accuracy showed no detectable difference on three and Jev 1.14 ahead on two (CLINC150, Decision Bench bench-v4); Brier: Jev better on all five. B45 ran while OpenAI's Decisions API was in public beta (generally available from 22 Oct 2026) (B45, A, gbolask24/decision-bench at e5b7202; flags: public-data; small-n (Opus arm); note: per-call logs referenced in the README are not in the repository; files opened 24 Oct 2026, 05:15–05:21 UTC; R23-S200 to R23-S211). Reported Results still differ by task; no overall direction is stated. Side by side: Decisions API vs Jev vs Microsoft Decision 1.
  • Per-route data terms are quoted on data handling by route (1 Oct); this page does not repeat them.

Change, : criterion 8 adds what TypeSafe's models page now states about image input ("Text, images" for both Jev 1.14 and Jev 1.13, first observed 7 Nov 2026); the images page still calls images not yet ready for production (R27-S05, R27-S06). The 16 Oct wording is kept as dated.

Change, : the hosted-API note now names the 8 hosted APIs from other providers listed on the alternatives page (it said 3, from 12 Oct; kept as dated); graded comparisons of Jev with OpenAI's Decisions API 5 → 6 (B45 added, grade A); the criterion 10 "Unknown" cell lists it. No favour changed.

Change, : criterion 10 "Evidence" cell: OpenAI's Decisions API, a public beta when its guide was read on 8 and 10 Oct, became generally available on 22 Oct 2026 (R22-S32, R22-S47). No favour or graded result changed.

Change, : criteria 7 and 8 updated for Jev 1.14: images are an experimental opt-in (earlier: "no image input"), and an opt-in header raises context to 262k (earlier: 64k per request only). The favoured option in both rows is unchanged.

Change, : graded comparisons of Jev with OpenAI's Decisions API 3 → 4 (B42 added, grade B); criterion 10 "Unknown" cell updated.

Change, : graded comparisons of Jev with OpenAI's Decisions API 4 → 5 (B44 added, grade A, a Tetris round robin); the criterion 10 "Unknown" cell lists it.

Change, : graded comparisons of Jev with OpenAI's Decisions API 2 → 3 (B41 added); the criterion 10 "Unknown" cell no longer says "leads only, not graded"; the hosted-API note names 3 providers, as on the alternatives page.

Search published pools, pages, reports, and evidence.