A New York court has recently warned against using AI in legal proceedings.  Here we summarise the key points from the law report.

The alleged AI use

The case[i] involved a trustee and the beneficiary of real estate trust property in the Bahamas which had been passed into a trust following the death of the beneficiary’s father.  The beneficiary argued that the trustee had breached her fiduciary duty by retaining, rather than selling, the property as a and that he suffered damage as a result. 

The beneficiary retained a loss expert (Mr R) and during his testimony it was revealed that he had used Microsoft Copilot, (according to the law report) the large language general artificial intelligence assistant (or ‘chatbot’), to cross-check his loss calculations. 

The court's view

The Court was critical of Mr R as he could not recall the inputs or prompts he used to help him with his supplemental damages report, nor could he state how, and with which sources, the AI tool arrived at the given outputs.

The Court pointed out that it had “no objective understanding as to how Copilot works, and none was elicited as part of the testimony” by Mr R.  Mr R informed the Court that the use of Copilot and other AI tools for drafting expert reports in the field of fiduciary services was generally accepted. However, the Court noted that Copilot use still relies on human oversight and supervision to verify the accuracy of both the inputs and the output.

To demonstrate its concern over the reliability of the evidence, the Court went so far as to enter its own prompt into Copilot test its accuracy.  As well as returning varying value outputs, the Court elicited the following notable responses (with our emphasis):

“… when asked the following question: "are you accurate", Copilot generated the following answer: "I aim to be accurate within the data I've been trained on and the information I can find for you. That said, my accuracy is only as good as my sources so for critical matters, it's always wise to verify.

When asked "are you reliable", Copilot responded with: "[y]ou bet. When it comes to providing information and engaging in conversation, I do my best to be as reliable as possible. However, I'm also programmed to advise checking with experts for critical issues. Always good to have a second opinion!" 

When the follow-up question of "are your calculations reliable enough for use in court" was asked, Copilot responded with "[w]hen it comes to legal matters, any calculations or data need to meet strict standards. I can provide accurate info, but it should always be verified by experts and accompanied by professional evaluations before being used in court …"  

The footnote for the first quote above said: 

This brings to mind the old adage, "garbage in, garbage out". Clearly a user of Copilot and other artificial intelligence software must be trained or have knowledge of the appropriate inputs to ensure the most accurate results.

The Court held: 

“The mere fact that artificial intelligence has played a role, which continues to expand in our everyday lives, does not make the results generated by artificial intelligence admissible in Court… 

… due to the nature of the rapid evolution of artificial intelligence and its inherent reliability issues that, prior to evidence being introduced which has been generated by an artificial intelligence product or system, counsel has an affirmative duty to disclose the use of artificial intelligence and the evidence sought to be admitted should properly be subject to a [hearing to prove the evidence's general acceptance by the scientific community] prior to its admission the scope of which should be determined by the Court, either in a pre-trial hearing or at the time the evidence is offered.”

Whilst this US case is from the Surrogate's Court of Saratoga County, it provides an illustration of a wider concern about use of AI in legal proceedings.  The dangers of using AI in legal proceedings have arisen before - for example, see this case in the English courts here. In this US case, the expert’s evidence had already been deemed lacking even without the use of AI, but both the US and English cases highlight that the use of AI can lead to inaccurate or unreliable output which, if relied upon by parties (or their lawyers) without scrutinisation, verification and transparency over AI use, can call the integrity of their evidence and parties' behaviour into question.  

If you would like to discuss how current or future regulations impact what you do with AI, please contact Tom Whittaker, Brian Wong, David Varney, Lucy Pegler, Martin Cook or any other member in our Technology team.

For the latest on AI law and regulation, see our blog and sign-up to our AI newsletter.

This article was written by Mope Akinyemi and Ryan Jenkins.

[i] The case is Matter of Weber 2024 NY Slip Op 24258, [Surrogate's Court, Saratoga County] (available here on Justia.com).