Same embedding model. Higher on every dataset.

* Capped recall, as BEIR reports it for TREC-COVID, whose queries have up to ~500 relevant documents. Sources: BM25 from the BEIR paper, Tables 2 and 9; bge-base-en-v1.5 from the MTEB results repository (FiQA from the model authors' own run). Hosted APIs in the hero chart: MTEB published scores for OpenAI text-embedding-3-small and -3-large and Cohere embed-english-v3.0.

On this question-and-answer set the reranker helps, as the BEIR results predict for Q&A content. Each answer's section heading reaches the model with the chunk, so "Which release added offline checkout?" is answerable when the version number lives only in the heading.

By language

Passed on English, Hindi and Hinglish, then everything measured on Hindi and Hinglish together. Naturalness and register come from a judge model (openai/gpt-4.1-mini) and are graded for Hindi and Hinglish only.

A model meets the bar when it answers every item, keeps thinking as stated, passes at least 80% of the items and declines at least 95% of the questions its passages cannot answer. Recommended is the highest pass rate among those, then the lowest cost. The Hindi pick also needs, on the Hindi and Hinglish items together, 85% passed, 95% declined correctly and the right register in 95% of the judged replies; among those the highest Hindi pass rate wins, then the fastest first word. The Hindi budget pick is the cheapest model that meets the same bars (register 90%) and matches the Hindi pick's pass rate.

Best Hindi tone: anthropic/claude-haiku-4.5 (4.4 of 5 from the judge), not recommended: it declines only 69% of the unanswerable questions overall (50% on Hindi and Hinglish) and invents an answer for the rest.

94 frozen items (54 English, 20 Hindi, 20 Hinglish; 16 cannot be answered from the passages, 10 ask for a comparison table), the same passages for every model, measured 2026-10-03. Costs are what the provider billed, in rupees at 84 to the dollar. The benchmark cost $0.88 in all (answers $0.72, the Hindi and Hinglish judge $0.16).

Default fast model: openai/gpt-6-luna, the model with the highest business-question recall (98.6%) among those that pass every bar, then the highest overall accuracy (98.9%): p90 1.94 s, $0.162 per 1,000 labels. Budget option (the cheapest passing model): google/gemma-4-26b-a4b-it (64% cheaper than the default, $0.058 per 1,000, 97.7% overall, p90 1.91 s). Fastest passing: openai/gpt-4.1-nano (p90 1.27 s, 94.0% overall). Most accurate: openai/gpt-6-luna (98.9% overall, p90 1.94 s). Bars: overall 90%, each language 85%, business-question recall 97%, reply language 95%, label fallbacks at most 5%. Updated 2026-10-03; thinking control not verified for 0 of 15 rows.

Intel i7-14700K, RTX 5060 Ti 16 GB, Docker Desktop limited to 8 CPUs and 9.4 GB. One API process; throughput scales out with RAG_API_WORKERS. On CPU the embedding model sets the import ceiling; use a hosted embedding model or a GPU for large imports.

# GPU is optional; this overlay embeds in full precision, as published
docker compose -f docker-compose.yml -f docker-compose.gpu.yml \
  -f benchmarks/compose.local-embeddings.yml up -d --build

# BEIR: downloads ~330 MB, imports 272k documents, evaluates
python benchmarks/beir.py --datasets scifact,nfcorpus,arguana,scidocs,fiqa,trec-covid \
  --configs dense,hybrid,"hybrid(rrf)",hybrid+rerank --label mine
python benchmarks/fusion_study.py

# Real documents, security and operations, small policy set
python benchmarks/documents.py --label mine
python benchmarks/ops.py --label mine
python benchmarks/run.py --label mine

Retrieval benchmarks measure whether the right passage is found, not how an LLM words the answer. BEIR is English. Reference numbers for other models are their published results, not re-runs. Method and raw results: the benchmarks/ folder in the repository, available to partners.