In an unexpected twist, the bar exam for prospective lawyers in California has been revealed to have been partially crafted with the help of a large language model, according to a report by The Los Angeles Times. The State Bar of California made this admission on Monday, following a wave of complaints regarding the quality of the exam questions and several glitches reported during the February testing period.
The organization announced that 23 of the multiple-choice questions were created by ACS Ventures with AI assistance, while 48 others were borrowed from a prior exam designed for first-year law students. This disclosure has ignited outrage among legal professionals, with Mary Basick from UC Irvine Law School expressing her astonishment at the involvement of non-lawyers and AI in question formulation.
Katie Moran from the University of San Francisco School of Law characterized the revelation as a “staggering admission,” particularly noting that the same company responsible for generating the questions was also in charge of approving them for the exam. Test-takers reported being abruptly removed from the online testing platform, as well as encountering typographical errors, inaccuracies, and nonsensical questions throughout the exam.
Despite the acknowledged issues and the use of AI, a spokesperson for the State Bar insisted that the questions were reviewed by validation panels and experts in the field. The Supreme Court of California, which supervises the State Bar, claimed to have been unaware of the AI involvement until recently, despite prior suggestions to explore AI for potential enhancements to testing.
The hurried development of the exam questions, along with the last-minute publication of flawed practice questions, has raised alarms about the quick decision-making processes involved. The State Bar’s efforts to cut costs—including moving to a hybrid testing model and outsourcing certain services—have resulted in a $22 million deficit and prompted legal action from students who experienced the flawed exams.
The State Bar has pledged to consider adjustments for test scores and engage in discussions regarding possible remedies. However, it remains unclear whether the exam questions will be disclosed to the public or if the National Conference of Bar Examiners will be reinstated. This legal situation serves as a cautionary example of the dangers of favoring cost savings over a thorough development process for examination materials.
