ISTANBUL

A Chinese court has recently admonished a lawyer for using fabricated artificial intelligence (AI) case citations in a legal proceeding, according to a report by the state-run news agency Xinhua on Friday.

During a recent commercial dispute hearing, Beijing Tongzhou District People's Court Judge Zheng Jizhe discovered that two of the case references submitted by the plaintiff's lawyer were generated by AI.

Although both citations -- one allegedly from the Supreme People's Court, and another labeled as (2022) Hu 01 Min Zhong No. 12345 from the Shanghai No. 1 Intermediate People's Court -- appeared legitimate, with facts, legal arguments, and reasoning closely linked to the relevant case, verification revealed that they did not correspond to any actual cases.

When questioned in court, the lawyer admitted that the “reference cases” were produced by an AI model. He explained that he had submitted the case details to a large language model and copied the generated outcomes directly into his filing.

According to a law professor at Tsinghua University, Chen Hangping, the fabricated reference cases are typical examples of AI "hallucination," which are false or misleading information produced by AI models that appear believable.

The court ultimately dismissed the AI-generated references and issued a formal warning in its written judgment.

The judgment stressed that fabricated information produced by AI must not be allowed to disrupt judicial order.

In another recent case, a judge in a local court in east China discovered an AI written lawsuit, including fabricated white papers and incorrect case numbers.

China's top legislature amended the Cybersecurity Law earlier this week to improve oversight, ethical standards, and risk assessment for AI.

While such cases of AI misuse raise concerns, legal sectors worldwide are also exploring ways to responsibly integrate AI-assistance in their proceedings.