AI Achieves a Perfect LSAT Score

arXiv cs.AI / 4/14/2026

📰 NewsSignals & Early TrendsIdeas & Deep AnalysisModels & Research

共有:

Key Points

The paper (arXiv:2604.10034v1) claims the first documented case of a language model scoring a perfect 180 on an officially disclosed LSAT in controlled experiments.
It finds that prompt changes, shuffling answer choices, and sampling multiple responses do not materially affect performance, suggesting the model’s results are robust to common evaluation-time perturbations.
Removing the model’s generated “thinking” phase reduces frontier accuracy by up to 8 percentage points, mostly impacting logical reasoning.
Distillation that reproduces full thinking traces still underperforms frontier systems, implying that trace format alone is insufficient for top performance.
A pilot reward-model approach fine-tuned with QLoRA on official LSAT explanations using best-of-5 selection narrows the gap, again with gains concentrated in logical reasoning.

Abstract

This paper reports the first documented instance of a language model achieving a perfect score on an officially disclosed Law School Admission Test (LSAT). Controlled experiments on eight reasoning models show that varying the prompt, shuffling answer choices, and sampling multiple responses have no meaningful effect as drivers of performance. Ablating the thinking phase that models generate before answering, however, lowers frontier accuracy by up to 8 percentage points, predominantly in logical reasoning. Distilled models produce full thinking traces in the same format yet plateau far below frontier performance. A pilot process reward model fine-tuned via QLoRA on official LSAT explanations narrows this gap through Best-of-5 selection, with gains again predominantly in logical reasoning. The gatekeeper of elite legal education since 1948, the LSAT has not merely been passed but answered without a single error by models that reason. The upper bound of the cognitive capacities it has tested is no longer exclusive to human cognition.