- Officials and regulators warn that non-consensual sexual deepfakes associated with the AI chatbot may cause psychological, social and reputational harm
- Indonesia and Malaysia have blocked access to Grok, while regulators and prosecutors in Europe and Asia have launched investigations or issued formal warnings
ISTANBUL
Governments worldwide are stepping up scrutiny of xAI’s Grok, an artificial intelligence chatbot developed by Elon Musk, amid growing concerns that the tool can be used to generate non-consensual, sexually explicit and manipulated images.
The controversy centers on Grok’s image-generation feature, introduced last year, which allows users to modify real photographs and create AI-generated images. In recent weeks, the chatbot has been shown to respond to prompts that digitally remove clothing from images of people or otherwise alter their appearance into sexualized content, often without the consent of those depicted.
Authorities in multiple countries say the misuse highlights shortcomings in existing safeguards and moderation systems, raising concerns over privacy violations, human dignity and the protection of children online. Regulators have warned that such content may cause psychological, social and reputational harm, particularly when real individuals are targeted.
The issue has also intensified broader debates over the responsibilities of AI developers and online platforms, as governments assess whether current digital and criminal laws are adequate to address risks posed by rapidly evolving generative AI technologies.
From temporary access blocks to criminal investigations, regulatory probes and formal warnings, authorities across Southeast Asia, Europe and Australia have begun taking action in response to concerns surrounding Grok.
Indonesia
Indonesia on Saturday became the first country to temporarily block access to Grok, citing its alleged use in generating non-consensual, sexually explicit and manipulated images based on real photographs.
The Communication and Digital Affairs Ministry said the restriction was aimed at protecting women, children and the broader public from harmful AI-generated content.
“The government sees non-consensual sexual deepfakes as a serious violation of human rights, dignity and the safety of citizens in the digital space,” Communications and Digital Affairs Minister Meutya Hafid said in a statement.
Indonesia’s National Police described the move as a preventive measure while regulators assess whether the platform violates existing laws.
Alexander Sabar, director general of Digital Space Supervision, said: "Initial findings indicate that there are no specific regulations within Grok AI to prevent the use of this technology in the creation and distribution of pornographic content based on personal photos.”
“This risks serious violations of citizens' privacy and self-image rights,” he added.
Malaysia
Malaysia imposed a temporary restriction on Grok following misuse of the chatbot to generate obscene, sexually explicit and non-consensual manipulated images, including content involving women and minors, according to the Malaysian Communications and Multimedia Commission (MCMC).
In a statement posted on X, the regulator said notices sent this month to X Corp. and xAI requesting stronger safeguards resulted largely in responses that relied on user-reporting mechanisms and failed to address the inherent risks posed by the AI tool.
The restriction will remain in place until effective protections are implemented, the commission said, describing the measure as preventive and proportionate while legal and regulatory processes continue.
European Union
The European Commission on Tuesday called on X to adopt effective measures to shield EU citizens from sexual deepfakes and other harmful AI-generated content on its platform.
"We're still in that window where … now the ball is in their (X’s) court, and … we strongly encourage them to … come up with appropriate measures to settle the case, because we want to settle cases for the benefit of our citizens," commission spokesperson Thomas Regnier told reporters in Brussels.
Regnier said X has between 60 and 90 days to address the issue, warning that under the EU’s digital and AI rulebooks, platforms could face significant penalties or enforcement measures if they fail to prevent the spread of illegal content.
European Commission President Ursula von der Leyen also said Monday in an interview with the Italian newspaper Corriere della Sera that she was “outraged that a technology platform allows users to digitally strip people online,” adding that the bloc would act if companies fail to do so.
United Kingdom
Last week, UK Technology Secretary Liz Kendall urged X to urgently address the issue.
Britain’s online safety regulator Ofcom later opened a formal investigation into X over concerns that the AI tool was being used to generate sexualized imagery.
Ofcom said such material could amount to non-consensual intimate image abuse, pornography or child sexual abuse material, all of which are illegal under UK law.
“There have been deeply concerning reports of the Grok AI chatbot account on X being used to create and share undressed images of people – which may amount to intimate image abuse or pornography – and sexualised images of children that may amount to child sexual abuse material,” the regulator said.
Ofcom said it contacted X urgently on Jan. 5, setting a Jan. 9 deadline for the company to explain what steps it had taken to protect UK users. After reviewing the response, the regulator said it carried out an “expedited assessment” and subsequently escalated the case to a formal investigation.
France
The Paris Prosecutor’s Office expanded an ongoing investigation into X to include Grok in early January after three government ministers and two members of parliament accused the chatbot of generating fake sexually explicit videos, including content involving minors, French newspaper Le Parisien reported.
The investigation, open since July, was initially launched following complaints accusing the platform and its executives of manipulating algorithms for foreign interference.
Sarah El Hairy, France’s high commissioner for children, said earlier this month that she was “scandalised by requests made to Grok AI to digitally undress girls and women,” adding that “even if the image is artificial, the harm is very real.”
Italy
Italy’s Data Protection Authority (Garante) warned on Jan. 8 that the use or distribution of AI-generated images that digitally remove people’s clothing without consent could result in criminal liability.
The regulator said such practices may constitute serious violations of fundamental rights and privacy, stressing that AI providers must ensure compliance with data protection laws.
Italian authorities said they are coordinating with Ireland’s Data Protection Commission, the EU’s lead privacy regulator for X, and noted that national legislation has recently been strengthened to address the spread of AI-generated sexual deepfakes.
In September, Italy amended its criminal code to introduce penalties for disseminating AI-generated sexual deepfakes as part of broader efforts to combat digital abuse.
Germany
German authorities say they are preparing stronger legal measures to combat “digital violence,” including the misuse of AI tools to manipulate images into sexualized content without consent.
Anna-Lena Beckfeld, spokesperson for the Federal Ministry of Justice, said the government is working on legislation aimed at making it easier for victims to pursue legal action.
"It is unacceptable that manipulation is being used on a large scale for systemic violations of personal rights." Beckfeld said at a press conference on Jan. 9. "We, therefore, want to see stronger measures taken against this through criminal law."
In a post on X on Jan. 6, Wolfram Weimer, Germany’s media minister, urged the European Commission to take legal action to stop what he described as the “industrialization of sexual harassment.”
Australia
Australia’s eSafety Commissioner said authorities have requested information from X regarding the Grok’s operation.
The regulator said it has received a small but increasing number of complaints related to AI-generated sexual imagery and warned it could use enforcement powers, including content removal notices, if material breaches the Online Safety Act.
It also reminded online platforms and AI providers that under updated rules coming into force on March 9, services must prevent children from accessing sexual, violent or otherwise harmful content.
Canada
Canada has not imposed a ban or formal restriction on Grok, but officials have raised concerns over the issue.
Canada’s Minister of Artificial Intelligence Evan Solomon said on Jan. 8 that deepfake sexual abuse constitutes violence and that platforms and AI developers must act to prevent such harm.
“Deepfake sexual abuse is violence. We must protect Canadians, especially women and young people, from exploitation,” Solomon said in a post on X, adding that technology companies have a responsibility to stop the spread of such content.
Solomon said the government is advancing responsible AI policies, including Bill C-16, the Protecting Victims Act, which would amend the Criminal Code to classify deepfakes as intimate images for the offence of publishing intimate images without consent.
He added that Canada would “keep Canadians safe by amending the Criminal Code and holding abusers accountable.”
South Korea
South Korea’s media watchdog has asked X to come up with measures aimed at protecting minors from sexually explicit content generated by Grok, according to the Yonhap News Agency.
“We have asked the operator of X to prevent potential illegal activities on Grok and submit measures to protect teenagers from harmful content, including limiting or managing their access,” Korea Media and Communications Commission said in a statement.
