The advent of ChatGPT over eighteen months ago has changed the way we access information. Although still in a nascent stage, the artificial intelligence (AI) tool's influence has invariably caught the eye of law school students in India.
Law faculty have been constantly adapting to ensure integrity of online legal research done by students. The use of technology like Turnitin has aided them in checking whether projects are plagiarised, albeit with its own shortcomings of falsely reflecting original content as AI-generated.
So how are they now adapting to the latest technological change in the research field? Have students become lackadaisical with their assignments or has this tool been used to fairly aid them in projects assigned to them?
We caught up with law students and legal academicians to understand the influence that this tool can have on legal education, and how professors have tactfully integrated ChatGPT and re-imagined the way assignments are disbursed and assessed.
In a candid chat with about 30 law students across law schools on their experience of ChatGPT for and what exactly they utilise the tool for, it came to light that 29% students used the tool for summarising concepts or notes. Students opined that the responses can pass off as human.
54% used the tool for studying and found it particularly helpful for conceptual clarifications. 50% students used the tool for research and 47% used it for creating content while writing e-mails, blogs and projects.
Of these, 82% students found the tool useful, while 18% felt otherwise. In terms of the reliability of the output, 47% reported positively, while 24% responded negatively given its tendency to hallucinate and produce fake case laws.
Another 21% students mentioned that the reliability of the output depends from instance to instance. In the case of complex prompts, the accuracy of the output needs to be cross-verified, and sometimes the content gets repeated after the first 500 words of response to a prompt.
While running a prompt of a research project undertaken during my time at law school through ChatGPT, the tool did a decent job in dividing the content into chapters, headings and sub-headings. On asking the tool to produce content under each heading, there is a tendency for the tool to stray away from the main topic unless the prompt is very specific to what the user is expecting to address. Further, given that the output by the tool is a product of recombining existing information, the tool lacks the ability to critically analyse and provide innovative solutions to issues posed before it. The tool also falls short of creativity in presenting information and responding soundly to application-based questions.
On cross-checking the responses produced by it, most times it seems to be able to circumvent basic plagiarism tools like Grammarly. However, similar tools like Turnitin also have their shortcomings by virtue of it being an AI tool in itself. There may be a tendency for it to detect false positives which may tend to create unnecessary doubts and apprehension in the minds of facutly. Further, the lack of academic sourcing to the content generated by ChatGPT also mandates its users to ensure that the output generated by the AI tool is cross-verified.
Krishna Deo Singh Chauhan, Associate Professor at Jindal Global Law School, opines that introducing ChatGPT formally as part of the law school curriculum may not be feasible at the moment, given the complexity of the application. However, he admits that law schools cannot afford to ignore the presence of the AI tool. He added,
"Possibly it could be part of the basic training on library resources that students undergo. That too can only be done under supervision of how students must develop a critical faculty during its use. Do the students test and verify the responses they get from it? How do they do it? Does that ensure that work is scientific and trustworthy?"
On whether ChatGPT can be used as a legitimate research aid Chauhan said,
“This can happen only if we get students on board with the idea that this is a tool for better work, not a justification for less work...I have personally interacted with ChatGPT extensively and while awe-striking at first, it is nearly not as good as a student who reads diligently, thinks critically and wants to create an impact with her writing.
The rise of ChatGPT is no doubt a watershed event. All stakeholders must engage with it and use it to improve their work. Used mindlessly and unethically, it would stunt the intellectual development of the user, and the end product would be uncritical and sub-optimal.”
Deo shared that one of the ways he ensures students have not blindly used ChatGPT for projects is to have viva voce to find out whether the student has engaged with the work they have submitted.
Aman Gupta, Assistant Professor of Law and Technology at West Bengal National University of Juridical Sciences (WBNUJS), admitted that large language models (LLMs) have become a concern when students submit written assignments.
“These concerns and solutions are not new and have been implemented in various forms over the years. Even before ChatGPT became popular, there were concerns regarding plagiarism and the use of paraphrasing tools. However, the popularity of ChatGPT has certainly resulted in a greater focus on the problem in the last two years.
The initial response involved asking students to submit projects on issues that have emerged over the last three years (as September 2021 was the initial cut-off point for ChatGPT’s knowledge)."
On some of the other methods incorporated to circumvent the reliance on AI tools like ChatGPT by students, Gupta said,
“A few approaches have emerged. One has been to require students to undertake empirical research, which can involve statistical analysis of legal trends that have emerged over the last few years. Another option has been to retain the written components (either in the form of a position paper or a moot court memo) and link them to a viva component where the knowledge of the students regarding their written submission is tested.
One option, which I find interesting, has been to use AI to generate information using prompts and then require the students to find faults and improvements within the information using knowledge gained in class and/or research. Several of these solutions have been implemented at WBNUJS.”
Gupta admitted that it is crucial for law teachers to thoroughly read the project submitted by students.
“Personally speaking, the primary test for judging GPT use is reading the project as the AI tends to hallucinate. I have seen it produce legislation and sections within legislation that do not exist, which is often a clear giveaway of GPT output. The second level of scrutiny comes from the Turnitin AI Writing Detection Tool. Having seen it in action, it does yield fairly accurate results on whether something is written by AI or is an original work of the student. However, as Turnitin itself warns its users, it is not completely accurate and may yield false positives.”
Commenting on the integration of ChatGPT in law school curriculum, Gupta noted that there is not much difference between ChatGPT and other LLMs from any internet-based resources.
“I do not think we should be averse to integrating it into our law school curriculum in the future. Even law firms appear to be using AI (including LLMs) to automate various tasks, which include creating the first draft of agreements/clauses.
However, we must ensure that students understand that the LLMs represent only research starting point, and not its end. The emphasis must be on two elements: (i) originality in writing; and (ii) critical analysis. Students’ primary use of ChatGPT appears to be to finish assignments without putting in requisite effort or to meet deadlines.”
As an ardent user of ChatGPT, Nikhil Naren, Assistant Professor at JGLS, firmly believes that the tool is here to stay and that incorporating the AI tool into law school curriculum is crucial.
“Prior to such incorporation, we will need to make the understanding of GPT (General Purpose Technology) such as AI; and Generative Pre-trained Transformer Models such as ChatGPT clear. Due to its extensive database, ChatGPT may become an important tool. Assessing the STAR - Safety, Transparency, Accountability, and Reliability of AI-enabled systems is a must. Users must be sensitised to societal issues raised by AI such as bias and privacy challenges as much as ethical issues behind using ChatGPT.”
Elaborating on the techniques he adopts to integrate ChatGPT as part of assignments, Naren said,
“I give a legal factual situation like a moot problem. I ask the students to feed the prompt to ChatGPT and to critically analyse the prompt given by the tool. I ask students to review privacy policies of oft-used apps like Instagram and notice dark patterns. This makes them go through policies in real time and make observations. I also make them present their findings through PPTs. For any take-home assignment, it is a given that students are going to take the assistance of an AI tool, so to circumvent the complete reliance on the tool, I started disseminating these kinds of assignments. This makes the students put their mind to assessing the GPT model as well. So it is both a critical analysis and an encouragement to use ChatGPT without compromising on the effort required to complete a task.”
On the overall sentiment around the influence of technology on academia and jobs, Naren drew an analogy to the time when computers were newly introduced.
“The same fear is surfacing today. But the purpose of technology is not to replace, but to use it as a tool to become more efficient and for better work.”
Prof Dr Shashikala Gurpur, Director of Symbiosis Law School, Pune, opined that AI tools like ChatGPT can make students future-ready. However, they cannot replace the crucial skills required by a lawyer.
“In legal education, the legal acumen of problem solving can not be fully addressed by ready data, as human operations are ingenuous. People are unpredictable. It has limits to address human transformation and innovation. Merit of public service, client commitment, designing society or activity in regulated fashion is a synthesis of intuition, information and thinking. A machine cannot substitute the wonders of multilayered human possibility. It can support meritocracy and caution should be exercised not to let it destroy meritocracy.”
At the University of Minnesota earlier last year, ChatGPT was made to respond to 95 multiple choice questions and 12 essay-based questions and its responses were shuffled with the responses of other law students. The responses generated by the tool secured passing grades, although it scored a below average with a C+. This experiment was done across subjects and the results were more or less the same throughout. It also mentioned that if its performance was consistent throughout, it would possibly even pass the JD exam.
A professor at the University of Pennsylvania Carey Law School opined that banning the tool is not a solution and that students need to become aware of when the tool can be relied on. Another teacher from the University of Michigan is of the view that integrating ChatGPT in law schools would prove as a nudge for students to enhance their legal writing skills. To circumvent the use of the tool in exams, he asks questions which are more focused on the teachings in class and avoids generic questions which can be easily answered by the tool.
In Australia, a specialised AI tool called Australia Law School SuperGPT has been developed. It assists students on a range of tasks from reviewing, and writing drafts, and understanding and applying complex legal principles and doctrines. While schools in Australia had initially restricted the use of ChatGPT, the government developed a framework for the use of emerging technologies in consultation with a national AI schools taskforce, education unions, experts and other key stakeholders.