Key Takeaways: AI-powered nudify tools have generated an estimated 3 million sexualized images since January, with children and public figures as primary targets. Schools lack specific deepfake policies, leaving administrators to rely on outdated cyberbullying rules. The Take It Down Act now criminalizes nonconsensual AI-generated intimate images, but enforcement remains uneven across jurisdictions.
AI-powered nudify tools have generated millions of nonconsensual deepfake images since the start of the year, creating a new category of digital abuse that schools, parents, and regulators are racing to contain.
The proliferation of generative AI systems capable of stripping clothing from photos or creating explicit content from scratch has turned a fringe internet problem into a mainstream crisis. The Center for Countering Digital Hate estimated with high confidence that xAI's Grok alone created 3 million sexualized images in January, including more than 20,000 depicting children. While xAI has since introduced safeguards, a WIRED analysis published this week found dozens of explicit deepfakes still hosted on Grok.com, including images of celebrities and at least one US politician.
"Elon Musk knowingly added a perverse feature to his platform that helps users undress women and children at the click of a button, with no regard for the predictable damage it would cause," Imran Ahmed, CEO and founder of the Center for Countering Digital Hate, said.
The abuse has migrated into schools, where students are using publicly available nudify apps to target classmates and even teachers. Jennifer Duer, executive vice president of product at Lightspeed Systems, which provides content filtering software to K-12 schools, said the tools are driving a rise in sextortion cases. "Kids are sending out a picture to somebody, and that picture gets converted into something it wasn't," Duer said. "Now they're saying, 'Hey, pay this or I'm going to tell all your friends and family.'"
The Legal Landscape Remains Fragmented
The federal Take It Down Act, enacted this year, criminalizes the nonconsensual online publication of intimate visual depictions, including AI-generated deepfakes. Most states have passed similar legislation. But school districts have been slow to update their internal policies to match the new legal reality.
Teddy Hartman, senior director of privacy and data policy at GoGuardian, said most school systems lack specific deepfake policies. "At the school level, there is usually no specific deepfake policy," Hartman said. "But most school systems do have responsible use and cyberbullying policies in place. Those still work, even with this very extreme and graphic form of cyberbullying."
The gap between federal law and school-level enforcement creates liability risk. Duer warned that parents may sue districts when deepfake bullying occurs on school property or using school devices. Schools are also grappling with students creating fake images of staff members, triggering investigations and suspensions before the content is verified as fabricated.
Detection Tools Struggle to Keep Pace
Off-the-shelf deepfake detection tools can flag manipulated images with roughly 90 percent accuracy, but the technology is losing ground as generative AI improves. "AI is getting better at tricking those," Duer said. "It used to have telltale signs, where somebody had an extra finger or weird-looking teeth. But now they look really realistic."
Content filtering companies are adapting. GoGuardian's visual nudity detection system can block nude images dynamically, whether real or AI-generated, preventing them from being accessed on school-issued devices. Lightspeed Systems offers anonymous reporting tools that let students flag incidents without fear of retaliation.
Regulatory Pressure Mounts on AI Platforms
The issue extends beyond schools. Canada's Privacy Commissioner published preliminary findings this week alleging that xAI violated federal privacy law by failing to implement appropriate safeguards from the outset. The investigation noted that xAI has introduced new protections but said the company has not demonstrated their effectiveness.
SpaceX, xAI's parent company, disclosed in a May filing that it has set aside $530 million to handle ongoing legal complaints, including those linked to Grok. The filing warned that Grok's "Unhinged" and "Spicy" modes present "heightened risks, including reputational harm, the generation of potentially explicit content and misinformation or deceptive outputs."
Henry Ajder, a deepfake expert who has tracked explicit AI content for nearly a decade, said Grok's safeguards still lag behind mainstream competitors. "While Grok and X may have made some amendments to their model, particularly following the backlash around nudification at the beginning of the year, they still have not done a sufficient job to bring it up to the standard of the other mainstream tools that are available," Ajder said.
For investors, the regulatory trajectory is clear. The $530 million reserve at SpaceX signals that legal costs tied to AI-generated content are becoming material. Companies that fail to implement robust content moderation — particularly those serving users under 18 — face escalating liability as federal and state enforcement ramps up. The Take It Down Act provides a private right of action for victims, opening the door to civil suits that could dwarf regulatory fines.
This article is for informational purposes only and does not constitute investment advice.
