drafting-specific independent test · Legal Benchmarks (Anna Guo and co-authors)
Legal Benchmarks, Phase 2: Benchmarking Humans & AI in Contract Workflows (contract drafting)
published 2025-09 (preliminary findings; tools run July to August 2025) · read 2026-10-23
30 contract-drafting tasks (basic clauses, template adaptation, bespoke clauses; junior to mid level) written by practising lawyers, run on 13 AI tools plus in-house commercial lawyers (about 10 years' experience) as a baseline, 450 task outputs in all. A draft passed only if it met every key element from the contributing lawyer's answer key plus instruction compliance, factual accuracy and legal adequacy. Pass rate averaged 57.3% across the AI tools; the human lawyers reached 56.7%. Top individual tool: Gemini 2.5 Pro at 73.3%, against 70% for the best individual lawyer. The authors report that legal-specific tools did not clearly beat general assistants on draft quality, and that legal tools raised explicit risk warnings more often (83% of high-risk outputs versus 55% for general tools).
Caveat. Run by an independent project, not by a vendor, but the site also sells benchmarking and some tool descriptions carry affiliate-tagged links; 30 tasks and one prompt per tool with default features, so a snapshot of 2025 versions that is over a year old; an LLM jury graded reliability with expert review of borderline cases, and usefulness and workflow scores are expert opinion; one tool is anonymized; the authors say their 2025 figures are not comparable with the current leaderboard pass rates; read from the live page 2026-10-23.
Names: Gemini 2.5 Pro, GPT-5, GC AI, Brackets, August, SimpleDocs.
https://www.legalbenchmarks.ai/research/phase-2-research