the main points
- ChatGPT was given the same test as the students, consisting of 95 multiple-choice questions and 12 essay questions.
- The robot passed the law school exam and got a C+.
- “ChatGPT has demonstrated a strong understanding of basic legal rules and has had a consistently strong organization and composition.”
Powered by large amounts of data from the Internet, the chatbot passed exams at a US law school after writing essays on topics ranging from constitutional law to taxes and torts.
ChatGPT from OpenAI, a US company that this week acquired a huge cash infusion from Microsoft, uses artificial intelligence (AI) to generate text streams from simple prompts.
The results were so good that teachers warned it could lead to widespread cheating and even signal the end of traditional classroom teaching methods.
Jonathan Choi, a professor at the University of Minnesota Law School, gave ChatGPT the same test the students face, which consists of 95 multiple-choice questions and 12 essay questions.
In a white paper titled “ChatGPT Goes to Law School” published Monday, he and his colleagues report that the bot gets a C+ grade overall.
While this was enough to pass, the bot was near the bottom of the class in most subjects and “bombed” on multiple choice questions involving math.
“In writing the articles, ChatGPT demonstrated a solid understanding of basic legal rules and had well-established organization and composition,” the authors write.
But the robot “often struggled to identify problems when given an open prompt, which is an essential skill on law school exams.”
Officials in New York and other jurisdictions have banned the use of ChatGPT in schools, but Choi suggested it could be a valuable educational aid.
“Overall, ChatGPT was not a great law student acting alone,” he wrote on Twitter.
“But we expect that in collaboration with humans, language models like ChatGPT will be very useful for law students taking exams and for practicing lawyers.”
Downplaying the possibility of cheating, he wrote in response to another Twitter user that two out of three brands had spotted the paper written by the bot.
Choi wrote: “[They had]a hunch and their hunch was right, because ChatGPT had elaborate grammar and was fairly repetitive.