AI Tool 'Grook' Faces Scrutiny Over Creation of Fake Sexual Content
In a recent surge of controversies surrounding the use of Artificial Intelligence, international calls for an investigation have intensified following the dissemination of fake "sexual" videos and images created by the AI tool Grook, which is integrated into the platform X. These materials often depicted minors and adult women in explicit contexts.
Grook acknowledged on Friday that there were "vulnerabilities" allowing users to obtain digitally altered images and videos showing full or partial nudity of real individuals, leading to significant backlash and prompting French authorities to expand their legal investigations against the platform.
* "We are urgently addressing the vulnerabilities"
The Grook account on platform X responded to a user complaint, stating:
"We have identified vulnerabilities in our security measures and are working urgently to address them. Child sexual exploitation is illegal and prohibited."
Reports indicated that some users had sent images or videos of children and teenagers, requesting the algorithm to edit them to display full or partial nudity. However, the company X AI, which developed the program, did not provide an official comment on the ongoing controversy, instead issuing automated responses to some inquiries, claiming that traditional media "lies."
Nonetheless, Grook warned users that any company in the United States "exposes itself to civil or criminal liability if it knowingly facilitates the production of child pornography or fails to prevent it."
* Vulnerabilities also affect adult women
The issue was not limited to minors, as some internet users exploited Grook to manipulate images of adult women on social media for digital humiliation.
The Indian Ministry of Electronics and Information Technology responded by issuing an official notice to platform X, demanding a detailed report within 72 hours on the measures taken to remove "obscene, naked, inappropriate, and explicit sexual content" generated by the application without the consent of the affected women.
* Expanded legal investigations in France
In France, the Paris prosecutor's office has broadened its investigations to include the Grook tool following complaints from three ministers and lawmakers who stated that it generates and publishes fake sexual videos featuring minors without their consent.
The ministers and lawmakers confirmed that the videos were based on deepfake technology and demanded their immediate removal from the platform. The initial investigations were launched last July due to complaints regarding the platform's algorithm manipulation.
This latest crisis raises significant questions about the safety of AI tools and the responsibility of developing companies to prevent abuse, particularly concerning the rights of minors and women.
