AI chatbots can explain the law to people with civil legal problems, but they often fail to point them to local help, a new study found. Vals AI and NextLadder Ventures released the Access to Justice Benchmark on Nov. 10.
The test covered 23 AI models and 252 legal scenarios. Legal aid experts in six states wrote the scenarios, which reflect problems often faced by low-income Americans.
Every model met 78% to 87% of the criteria for general explanations of the law. But the same models met only 30% to 50% of the criteria for naming specific resources, such as local legal aid programs or court self-help centers.
Free Models Keep Pace With Paid Ones
The study found little difference between free and paid chatbot models. Free-tier models averaged 69.7%, while premium models averaged 70.4%.
Anthropic’s free-tier Claude Sonnet 5.5 scored 1.2 percentage points higher than its premium Claude Opus 5.5. Only one gap was larger than the margin of error, with xAI’s premium Grok 4.7 beating its free version by 2.3 percentage points.
Alibaba’s Qwen 3.8 Max ranked first with a score of 76.4%. Even so, it missed about a quarter of what experts expected, such as asking questions to understand a person’s situation before giving guidance.
The top three models were within the margin of error of each other. The two highest-ranked OpenAI models trailed at about 66%.
Results also varied by topic. Housing and landlord-tenant questions scored best at 75.4% across all models, while public benefits questions scored worst at 57.3%.
Family law scores depended on the subtopic. Visitation scored 61.3% and kinship caregiving with child protective services scored 63.0%, while guardianship, divorce finances, family adoptions and domestic violence fell between 66% and 68%.
Where the Chatbots Fell Short
In one test, a New Orleans parent asked about suing the child’s other parent in small claims court over a verbal loan. Most models named the correct court and cited Louisiana’s 10-year prescriptive period.
Only 8 of 23 models asked whether the debt involved real estate, which decides whether a verbal deal can be enforced. Just 4 warned the claim is lost after 10 years, and only 3 pointed to the court clerk’s website for forms.
In another test, a New York City tenant who moved out after a constructive eviction asked how to recover an unreturned deposit. Nearly every model cited the landlord’s 14-day deadline and suggested a demand letter followed by a small claims filing.
Fourteen models lost points for pushing that out-of-court step, which the expert rubric warned against. Only 4 of 23 referred the tenant to the city’s Right to Counsel program, while most offered general hotlines like 311.
According to the Legal Services Corporation’s 2022 Justice Gap study, low-income Americans did not get any or enough legal help for 92% of their civil legal problems. Of those who did not seek help, 46% cited cost as a reason.
“Most legal AI benchmarks measure work done for law firms and corporate clients,” said Meghana Kotcherlakota, a research scientist at Vals AI. She said the benchmark was built to show how well models serve the people who need help most.
NextLadder Ventures CEO Ryan Rippel said the results show “exactly where AI models need to get better.” He said he hopes independent builders and frontier labs will take up the challenge.
Vals AI and NextLadder Ventures are now working on a full report and an additional brief. The materials are meant to help frontline practitioners understand the gaps these tools still need to close.
