{"updated_at": "2026-10-26T10:12:47Z", "read": "2026-10-24", "report": "https://aiforlawyers.app/report/legal-ai-benchmarks", "pdf": "https://aiforlawyers.app/report/legal-ai-benchmarks.pdf", "title": "Legal AI benchmarks", "note": "Other publishers' figures, copied as they state them with their date, limit and primary link. Not our score, not a ranking. evidence = verbatim phrases from the source read on the read date.", "license": "https://creativecommons.org/licenses/by/4.0/", "attribution": "aiforlawyers.app (AI Vortex), Legal AI benchmarks, read Oct 24, 2026, https://aiforlawyers.app/report/legal-ai-benchmarks", "benchmarks": [{"id": "vlair-2025-02", "name": "Vals Legal AI Report (VLAIR), first edition", "measured_by": "Vals AI, with a lawyer control group (the Lawyer Baseline)", "date": "2025-02-27", "date_label": "report page, \"Updated: 02/27/2025\"", "kind": "legal products, 7 tasks", "tools": "CoCounsel (Thomson Reuters), Vincent AI (vLex), Harvey Assistant (Harvey), Oliver (Vecflow). Lexis+AI (LexisNexis) withdrew.", "metric": "Accuracy or performance score against predefined criteria on seven tasks (data extraction, document Q&A, summarization, redlining, transcript analysis, chronology generation, EDGAR research).", "headline": "The report states that Harvey opted into six of the seven tasks, received the top score among the participating AI tools on five and placed second on one. CoCounsel scored from 73.2% to 89.6% on its four tasks. The Lawyer Baseline outperformed the AI tools on two tasks and matched the best tool on one; one of those was redlining (79.7%).", "limit": "Vendors could withdraw from any task before publication, so the tools were not all measured on the same tasks (Harvey withdrew from EDGAR research, vLex from chronology generation, LexisNexis from every skill except legal research). Scoring is automated (LLM as judge). Products as of the 2025 study.", "url": "https://www.vals.ai/industry-reports/vlair", "evidence": ["Updated: 02/27/2025", "Harvey opted into six out of seven tasks. They received the top scores of the participating AI tools on five tasks and placed second on one task.", "with scores ranging from 73.2% to 89.6%", "The Lawyer Baseline outperformed the AI tools on two tasks and matched the best-performing tool on one task.", "Redlining (79.7%) was the only other skill in which the Lawyer Baseline outperformed the AI tools.", "we allowed each vendor to withdraw its product from evaluation for any single task or for the study in totality prior to publication", "LexisNexis chose to withdraw from all skills except legal research", "Harvey withdrew from the EDGAR Research task. vLex withdrew from the Chronology Generation task.", "This LLM-as-judge method"], "read_sha256": "df3a48aa89841468", "read": "2026-10-24"}, {"id": "vlair-research-2025-10", "name": "Vals Legal AI Report: legal research (VLAIR legal research)", "measured_by": "Vals AI, with a lawyer control group and six law firms that wrote the questions", "date": "2025-10-14", "date_label": "report page, \"Updated: 10/14/2025\"", "kind": "legal products, research", "tools": "Alexi, Counsel Stack, Midpage and ChatGPT (OpenAI), plus the Lawyer Baseline.", "metric": "200 U.S. legal research questions scored on three weighted criteria: accuracy (50%), authoritativeness (40%) and appropriateness (10%).", "headline": "Average weighted scores were 74% to 78% for the four AI products and 69% for the Lawyer Baseline. The page reports 78% to 81% accuracy for the legal AI products and 80% for the generalist product. It says Counsel Stack received the highest score across all criteria.", "limit": "Only vendors that agreed to take part were evaluated, four in all. The lawyers wrote shorter answers by default, which the report says may have lowered their scores. Reference answers came from the contributing firms, and evaluators could use discretion where a reference answer was incomplete.", "url": "https://www.vals.ai/industry-reports/vlair-10-14-25", "evidence": ["Updated: 10/14/2025", "The evaluated products were Alexi, Counsel Stack, Midpage and ChatGPT.", "respond to 200 U.S. legal research questions", "Accuracy : whether the response is substantively correct with no incorrect elements (50% of average weighted score)", "Authoritativeness : whether the response is supported by citations to relevant and valid primary and/or secondary law sources (40% of average weighted score)", "Appropriateness : whether the response is easy to understand and could be immediately shareable with colleagues or clients (10% of average weighted score)", "all AI products scored within 4 points of each other (74%-78%) and within 9 points of the Lawyer Baseline (69%)", "78-81% accuracy, 80% averaged", "generalist AI product (80% accuracy)", "Counsel Stack received the highest score across all criteria", "the lawyers’ shorter responses may not have included all elements", "Our evaluators were allowed to exercise discretion where they felt the reference answer had minor errors or was less complete than ideal"], "read_sha256": "9496f3c01c053723", "read": "2026-10-24"}, {"id": "stanford-2024-05", "name": "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (Stanford)", "measured_by": "Magesh, Surani, Dahl, Suzgun, Manning and Ho (arXiv preprint)", "date": "2024-05-30", "date_label": "arXiv v1, submitted 30 May 2024", "kind": "legal research products, hallucination", "tools": "Lexis+ AI (LexisNexis), Westlaw AI-Assisted Research and Ask Practical Law AI (Thomson Reuters), with GPT-4 as the general-purpose comparison.", "metric": "Share of responses that hallucinate, from a preregistered set of legal research queries and a typology that separates hallucinated from accurate responses.", "headline": "The authors report that the Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI tools each hallucinate between 17% and 33% of the time, and that hallucinations are reduced relative to general-purpose chatbots (GPT-4).", "limit": "The abstract gives one range for the three tools, not a figure per tool. A May 2024 snapshot of products that have changed since. A hallucination rate depends on the question set and on each product being used for what it is designed for.", "url": "https://arxiv.org/abs/2405.20362", "evidence": ["Submitted on 30 May 2024", "first preregistered empirical evaluation of AI-driven legal research tools", "each hallucinate between 17% and 33% of the time", "hallucinations are reduced relative to general-purpose chatbots (GPT-4)", "Westlaw AI-Assisted Research and Ask Practical Law AI", "proposes a clear typology for differentiating between hallucinations and accurate legal responses"], "read_sha256": "bf7cffab5b566afe", "read": "2026-10-24"}, {"id": "legalbench-2023-08", "name": "LegalBench (paper and open task set)", "measured_by": "Neel Guha and 39 other authors (arXiv preprint), built with legal professionals", "date": "2023-08-20", "date_label": "arXiv v1, submitted 20 Aug 2023", "kind": "language models, legal reasoning", "tools": "20 open-source and commercial language models in the paper; no legal software products.", "metric": "Performance on 162 tasks covering six types of legal reasoning, written or hand-crafted by legal professionals.", "headline": "The paper presents 162 tasks in six types of legal reasoning and an empirical evaluation of 20 open-source and commercial language models.", "limit": "It measures language models, not legal products. The 20 models are those of 2023. It is a task set and a method; current scores live on leaderboards that run it, such as the Vals page in the next row.", "url": "https://arxiv.org/abs/2308.11462", "evidence": ["Submitted on 20 Aug 2023", "consisting of 162 tasks covering six different types of legal reasoning", "tasks designed and hand-crafted by legal professionals", "empirical evaluation of 20 open-source and commercial LLMs"], "read_sha256": "532dd342187fe232", "read": "2026-10-24"}, {"id": "vals-legalbench-live", "name": "LegalBench on Vals AI (live scores)", "measured_by": "Vals AI (runs the LegalBench task set on current models)", "date": "2026-10-13", "date_label": "leaderboard page, \"Updated 10/13/2026\"", "kind": "language models, live scores", "tools": "Frontier and open models from many labs; the page shows 23 latest, top and frontier models in its chart.", "metric": "Accuracy on open-source legal reasoning tasks.", "headline": "As of October 13, 2026 the Vals page lists Claude Fable 5 first with 88.56%, followed by Claude Fable 5.1 (88.51%) and Gemini 4 Argon (88.30%).", "limit": "The first three scores are 0.26 points apart and the page text we read shows no interval, so the order among them is not a clear difference. Scores are for raw models, not legal products. The page changes with every new model, so read the date.", "url": "https://www.vals.ai/benchmarks/legalbench", "evidence": ["Updated 10/13/2026", "As of October 13, 2026, Claude Fable 5 ranks first on LegalBench with 88.56%, followed by Claude Fable 5.1 (88.51%) and Gemini 4 Argon (88.30%).", "Evaluating language models on a wide range of open source legal reasoning tasks."], "read_sha256": "e8cfe3dcb18353d6", "read": "2026-10-24"}, {"id": "vals-lrb-live", "name": "Legal Research Bench on Vals AI (live scores)", "measured_by": "Vals AI (research paper linked from the page)", "date": "2026-10-22", "date_label": "leaderboard page, \"Updated 10/22/2026\"", "kind": "language models in an agent harness, US legal research", "tools": "Frontier and open models from many labs, run as agents with case-law search, web search and document retrieval.", "metric": "All-pass accuracy: a question counts only if the answer satisfies every item of a lawyer-written rubric (0% otherwise), graded by an LLM judge. A partial-credit score is also reported.", "headline": "As of October 22, 2026 the Vals page reports Muse Spark 1.3 Max, Claude Opus 5 and Claude Fable 5.1 tied for first at 55.29%, with Gemini 4 Argon next at 54.81%.", "limit": "Models in an agent harness, not legal products such as Harvey or CoCounsel. The test set is private, so outsiders cannot rerun it. Grading is by an LLM judge. All-pass is a strict bar, so scores look low by design.", "url": "https://www.vals.ai/benchmarks/legal_research", "evidence": ["Updated 10/22/2026", "As of October 22, 2026, Muse Spark 1.3 Max , Claude Opus 5 , and Claude Fable 5.1 tie for first on Legal Research Bench with 55.29%.", "4 Gemini 4 Argon 54.81%", "a question counts as correct only if the response satisfies every rubric item: 100% if all checks pass, 0% otherwise", "All grading is performed by an LLM judge", "The Test set will remain private.", "Agents are evaluated on a shared harness with access to five tools."], "read_sha256": "06fa8073a67b15f9", "read": "2026-10-24"}]}