Can Jev replace RAG or just improve retrieval?

A search tool can find the right incident without a vector database. That is interesting. It is not the same as replacing retrieval-augmented generation.

We investigated that distinction by inspecting Chat Seek's source and running a separate, bounded comparison of four retrieval methods. Hosted Jev selected a relevant first result for every answerable question at every tested corpus size. The embedding baseline found all labeled evidence within its top three, too. Neither result tells us whether a generated answer would be correct.

The useful question is not whether RAG is dead. It is which part of your evidence-finding workflow needs to change.

What the demo actually replaces

Start with the implementation, not the slogan. In the inspected Chat Seek v0.1.2 source, search begins with a lexical scan: it looks for query terms in saved conversation records. It retains up to 80 candidates, then scores a smaller subset locally with Laya. Selected scores are blended with lexical scores; unselected candidates keep their lexical ranking signal.[1][2]

That is still retrieval. More importantly, this path does not call hosted Jev. Laya's inspected ONNX export combines a ModernBERT encoder with a Laya decision head. Similar-looking interfaces do not make Laya and Jev the same model.[2][3]

Chat Seek returns existing conversation excerpts and lets someone open the surrounding messages. It does not generate a new answer from those excerpts.[2] By contrast, the original RAG paper combines retrieval with a language-generation model.[4]

Our architectural conclusion is straightforward: replacing vector search changes the retrieval stage. If another model then writes an answer using the retrieved evidence, the overall application remains retrieval-augmented. If the application only finds old conversations, calling it a replacement for a complete RAG system overstates what it does.

This distinction also prevents a misleading benchmark. Our hosted Jev measurements below are not measurements of Chat Seek or local Laya.

How we tested four retrieval methods

Imagine a QA team searching incident notes. Someone remembers a PDF download failure but not whether the cause was browser policy or a storage outage. Another person needs both a reproduction log and the release that fixed it.

We wrote synthetic records for that kind of work, then froze the documents, questions and relevance labels before the run. The same 24 questions were tested against nested collections of 24, 96 and 240 documents. Twenty questions had labeled evidence, including four requiring two records. Four deliberately had no answer.

The methods were:

• BM25: a local keyword-ranking baseline, using lowercase word tokens.

• Embeddings: local all-MiniLM-L6-v2 document and query vectors, ranked by exact cosine similarity. No hosted vector database was involved.

• BM25 plus Jev: the top eight keyword results, scored in one request with a separate Jev Noul relevance question for each candidate. Noul returns a yes probability.[6]

• Direct Jev: all documents supplied together, with a Choice selecting a document or NONE, plus a separate question checking whether any evidence existed.

The remote model was pinned to jev-1.13.0, and every saved successful response returned that identifier. There were 144 real Jev requests and 288 method/query/scale rows. No answer-generation model ran.

We measured whether the first result was relevant, whether the top three contained the labeled evidence, and whether methods declined no-answer questions. Latency covered the online retrieval step, not initial document embedding.

This was a small diagnostic, not a production leaderboard. The extra documents were mostly templated distractors. Labels were authored, not independently adjudicated. There was one run per condition, and abstention thresholds were not calibrated. Those limits are part of the result, not fine print.

What the measurements showed

At 240 documents, direct Jev had the strongest first-result performance in this run:

The first two columns measure ranking on answerable questions, ignoring abstention. The third tests abstention on the four negative questions. These are different jobs.

At 240 documents, first-result relevance was 16/20 (80%) for BM25, 17/20 (85%) for local MiniLM embeddings, 19/20 (95%) for BM25 top 8 + Jev, and 20/20 (100%) for direct Jev. All labeled evidence appeared in the top three on 16/20 (80%), 20/20 (100%), 18/20 (90%) and 20/20 (100%) questions, respectively. These ranking metrics ignore abstention. Small synthetic sample; no generated answers or end-to-end RAG were evaluated.

Screenshot of our local experiment viewer, rendered from saved measured outputs. Synthetic data, not a provider dashboard. Timings combine different execution environments and are not model-only speed comparisons.

Direct Jev reached 20/20 relevant first results at all three sizes. The embedding baseline remained at 17/20, but included every labeled evidence record within its top three. That makes “vectors failed” an inaccurate reading. For an application that passes several passages to a reader, evidence coverage may matter more than which passage comes first.

One example shows the difference. Q05 asked about a PDF incident with a successful HTTP status whose response the browser would not expose. Embeddings ranked the storage-outage record first and the browser-policy record second. Both Jev methods put the labeled browser-policy record first. This is a useful selection improvement, not evidence that semantic retrieval found nothing.

Across all remote calls, reported input usage totaled 850,704 tokens. At the documented September 21 price of $0.042 per million input tokens, that implies about $0.0357 in input charges, with free output.[5] This is a calculated estimate, not an invoice, and excludes local compute, engineering and any downstream generation.

Where retrieval still fails

The clearest failure happened before Jev could inspect the relevant record.

Q07 asked why keyboard navigation falls out of a contacts scroller when rows disappear. At 240 documents, BM25 placed the labeled incident, D006, at rank 45. Our hybrid method sent only the top eight candidates to Jev. D006 therefore never arrived.

Jev assigned low relevance scores to the supplied candidates and abstained. That avoided accepting a weak match, but still left an answerable question unanswered. Both embeddings and direct Jev ranked D006 first and accepted it.

An illustrative failure, not a representative sample. The screenshot shows an actual synthetic corpus record and measured rankings from the saved run.

The practical lesson is to inspect candidate coverage before changing the reranker. A reranker cannot recover a record excluded from its input. Combining candidate sources or widening the shortlist is worth testing, but we did not run those variants here.

Abstention deserves its own caution. Our embedding cutoff was an arbitrary cosine similarity of 0.5. The hybrid used a Noul cutoff of 0.5. Numerically identical thresholds do not mean equivalent decisions. TypeSafe also distinguishes relative Choice probabilities from separate yes/no judgments and warns against transferring thresholds between them.[7][8]

On just four no-answer questions at 240 documents, BM25 rejected 0/4, MiniLM embeddings 2/4, and both Jev pipelines 4/4. Thresholds were untuned: BM25 top score ≤ 0; cosine < 0.5; hybrid maximum Noul < 0.5; direct NONE or existence < 0.5. These different score spaces are not comparably calibrated. Four successes do not establish production reliability; this chart also does not measure rejection of answerable questions.

Likewise, a relevant first result is not a complete answer. The Atlas question needed both a failure record and a fix record. Direct Choice asks for one best option; interpreting its remaining probabilities as a multi-document evidence ranking is only a diagnostic. We did not test whether a writer would use both records faithfully or cite them correctly.

What changes as the corpus grows

Direct selection's median HTTP roundtrip rose from 172.41 ms at 24 documents to 202.07 ms at 96 and 297.41 ms at 240. Its reported input usage across the 24 questions rose from 52,685 to 206,573 to 510,893 tokens.

Median online retrieval latency across 24 queries at 24, 96 and 240 documents: BM25 0.67, 1.11, 1.86 ms; local MiniLM embeddings 64.42, 63.44, 61.26 ms; BM25 top 8 + Jev 174.71, 163.83, 161.59 ms; direct Jev 172.41, 202.07, 297.41 ms. All four panels share a linear 0–350 ms scale. Local CPU timings are not model-only comparisons with hosted HTTP roundtrips. Query encoding is included; offline document encoding is excluded. One run per condition, no repeated timing trials or confidence intervals.

The eight-candidate hybrid used 26,350, 27,188 and 27,015 input tokens respectively. Keeping the shortlist small bounded what reached the hosted model, but candidate coverage suffered at the larger sizes. The keyboard-focus record was excluded, as was one required record in a separate two-evidence question.

Total reported Jev input tokens over all 24 queries rose from 52,685 to 206,573 to 510,893 for direct Jev as the corpus grew from 24 to 96 to 240 documents. The eight-candidate hybrid used 26,350, 27,188 and 27,015 tokens. These are hosted input units, not total cost: the local methods use no hosted Jev tokens but still require compute and indexing. Mostly templated distractor growth in one synthetic study is not a production scaling law.

This is a trade-off, not a universal scaling law. We expanded mostly templated distractors around the same questions. We did not test millions of documents, long manuals, live updates, concurrent traffic or repeated timing trials.

There are also hard request boundaries. The documentation inspected on September 21 specifies 64k tokens for the total request and 32k for state plus the longest question. Choice supports up to 255 options; our largest direct request used 240 document options plus NONE.[5][6]

An all-corpus prompt is therefore a bounded technique, not an unlimited index substitute. TypeSafe's own Jev limitations page recommends retrieving and filtering in code before sending irrelevant material to the model.[7] Larger collections force another design decision: which evidence should the model see?

When to use Jev in your retrieval stack

Our recommendation is to choose the smallest change that addresses an observed failure.

Consider direct Jev for a small, bounded collection when choosing among known records is the job. This experiment supports evaluating that pattern. It does not establish a safe universal corpus size or production reliability.

Consider Jev after retrieval when candidate coverage is already good. If the right record arrives but ranks below a plausible near-match, relevance scoring may help. If it never arrives, improve the first stage before celebrating reranker accuracy.

Keep the generator evaluation separate. A system that must explain an incident, reconcile records or produce citations still needs tests of the resulting answer. TypeSafe's own passage-classification cookbook combines embedding retrieval, relevance filtering and a separate generator.[9] Those components can coexist.

Before adopting either pattern, build a held-out set from approved data, include no-answer and multi-record questions, and compare against the retrieval system you actually operate. Tune abstention separately from ranking. Keep permissions and exact date or numeric rules in application code. Our hosted experiment used synthetic records; it does not validate sending private conversations to that endpoint.

The takeaway is narrower than “Jev replaces RAG” and more useful: Jev improved first-result selection in this small experiment, while embeddings preserved strong evidence coverage and keyword shortlists introduced recoverable design failures. Measure where evidence disappears, then change that stage.

Experiment evidence

All numerical findings and examples above come from the frozen local experiment: results and limitations, method/query rows, queries, corpus, independent validation and editorial review. These links provide public downloads of the frozen experiment evidence. The two screenshots are saved-output views, not new model executions.

Sources [1] Chat Seek search implementation [2] Chat Seek extension implementation [3] Laya model card [4] Original RAG paper [5] Jev models and pricing [6] Jev API reference [7] Jev limitations [8] Confidence guidance [9] RAG passage classification