Apple and Google Ordered to Remove Deepfake Nudify Apps and Stop Profiting From Sexual Abuse Technology

The City of San Francisco has issued a formal legal challenge to the world’s two largest mobile platform operators, demanding an immediate cessation of the distribution and monetization of "nudify" applications. San Francisco City Attorney David Chiu sent cease-and-desist letters to Apple and Google on Thursday, identifying 13 specific applications that utilize artificial intelligence to generate nonconsensual nude images. The legal notices demand that these tech giants not only remove the offending software from the App Store and Google Play Store but also terminate their financial relationships with the developers responsible for these tools.

The move marks a significant escalation in the regulatory battle against generative AI abuse. According to the letters, which were first reviewed by WIRED, the City Attorney’s office alleges that Apple and Google have been "aiding and abetting" the sale of explicit deepfake content. By facilitating in-app purchases for these services—and taking a standard 15 to 30 percent commission on those transactions—the companies are accused of profiting from technology that facilitates sexual harassment and digital abuse.

The Legal Mandate: Holding Platform Gatekeepers Accountable

City Attorney David Chiu’s office has taken an aggressive stance against the proliferation of deepfake technology, previously targeting 16 major websites dedicated to creating nonconsensual sexual imagery. This latest action, however, pivots the focus from the creators of the content to the distributors. Chiu argues that as the primary gatekeepers of the mobile ecosystem, Apple and Google have a statutory and ethical responsibility to vet the software they host.

"Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu stated. He further emphasized that these companies have likely generated millions of dollars in revenue from fees associated with apps that offer nudification features. The legal basis for the demand rests partly on California’s robust privacy and anti-harassment laws, which prohibit the support of services that create deepfake pornography.

The 13 apps identified in the investigation—eight on Apple’s App Store and five on Google’s Play Store—often hide their true purpose behind the guise of benign "face-swapping" or "photo editing" tools. Once a user downloads the app, the "undressing" or "nudifying" capabilities are typically unlocked through subscriptions or one-time payments. This "dual-use" nature allows developers to bypass standard automated moderation filters during the initial app submission process.

A Growing Crisis: The Evolution of Nudification Technology

The emergence of AI-powered "nudify" technology has transformed from a niche corner of the internet into a mainstream crisis over the last five years. Unlike early iterations of deepfakes, which required significant technical expertise and high-performance computing power, modern applications allow nearly anyone to create realistic sexualized images in seconds.

The technology generally relies on generative adversarial networks (GANs) or diffusion models trained on vast datasets of human bodies. By uploading a single reference photo of a person—usually a woman or girl—the AI can digitally "remove" clothing or transpose the person’s likeness onto graphic sexual scenarios. The results have become increasingly realistic as the underlying generative AI technology has improved, making it difficult for victims to prove the images are fabricated.

The scale of the problem reached a tipping point earlier this year when xAI’s Grok and other mainstream AI tools were reportedly used to create millions of sexualized images of high-profile celebrities. However, the harm is most acutely felt by private individuals. Reports have surfaced of deepfake sexual abuse incidents in at least 90 schools globally, where students use these apps to bully, humiliate, and extort their peers.

Chronology of Warnings and Research Findings

The legal action taken by San Francisco is the culmination of months of warnings from researchers and digital safety advocates. The timeline of discovery suggests that the issues were well-known to the tech giants long before the cease-and-desist letters were issued.

• January 2024: The Tech Transparency Project (TTP), an independent watchdog, released a report uncovering dozens of "nudify" apps available on both major platforms. Some of these apps were even being served as sponsored advertisements to users searching for photo editing tools.
• April 2024: A follow-up report by TTP revealed that despite some removals, the problem had persisted and, in some cases, worsened. The research estimated that apps identified in the study had been collectively downloaded approximately 480 million times, generating an estimated $120 million in combined revenue.
• May 2024: Researchers from Cornell University and Georgetown University published a preprint paper identifying 420 apps with face-swapping capabilities. Upon testing 155 of them, researchers found that 70 percent could be used to create nude images without any safety interventions or content blocks.
• August 2024: San Francisco’s City Attorney issues the formal cease-and-desist, citing California’s Civil Code and the potential for "millions of dollars" in illicit profit-sharing.

Katie Paul, Director of the Tech Transparency Project, noted that the marketing of these app stores as "safe and trusted" environments stands in stark contrast to the reality of the content available. "Apple and Google make a lot of promises… and that is just not what is playing out in reality," Paul said.

Corporate Responses and the Moderation Gap

In response to the allegations, Google has defended its record, asserting that it takes proactive steps to secure the Play Store. Google spokesperson Dan Jackson confirmed that the company has deleted "hundreds" of apps with nudifying features for policy violations, including the five specifically flagged by Chiu’s office.

"Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson said. He added that the company also restricts search terms like "nudify" to prevent users from discovering such software. However, critics argue that the sheer volume of downloads and the revenue generated suggest that Google’s "proactive" measures are insufficient to keep pace with the rapid deployment of AI tools.

Apple, which frequently highlights its "walled garden" approach as a superior security and safety feature, did not provide a comment prior to the publication of the legal notices. The silence from Cupertino has drawn criticism from advocates who point out that Apple’s strict control over its ecosystem should, in theory, make it easier to prevent such apps from reaching the public.

The Dual-Use Dilemma and the Future of Regulation

One of the primary challenges in regulating these apps is their "dual-use" capability. As highlighted by the Cornell and Georgetown research, many apps present as benign image editors. They offer filters, background removals, and face-swaps that are commonly used for harmless entertainment. The "nudify" feature is often an undocumented capability or a result of the AI’s lack of "guardrails."

This creates a moderation paradox: if a platform bans all AI image generators, they stifle innovation and remove popular tools used by millions. If they allow them, they risk hosting tools that can be weaponized. The San Francisco City Attorney’s office argues that the solution lies in more rigorous manual testing and the implementation of "safety-by-design" principles, where developers must prove their AI cannot be used for nonconsensual sexual content before being granted access to the store.

Broader Implications and Potential Legal Outcomes

The outcome of this legal confrontation could set a precedent for how platform liability is interpreted in the age of AI. For decades, Section 230 of the Communications Decency Act has shielded tech companies from liability for content created by third parties. However, Chiu’s office is targeting the sale and distribution of the software itself, as well as the financial gain derived from it, which may fall outside the traditional protections of Section 230.

If Apple and Google do not comply with the demands, they could face civil litigation from the City of San Francisco. Furthermore, this action may embolden other state attorneys general or federal regulators to pursue similar charges. There is also growing momentum for federal legislation, such as the DEFIANCE Act, which aims to provide victims of nonconsensual AI-generated pornography with a clear legal path to sue both the creators and those who distribute the content.

The human cost remains the central focus for advocates. "These images are used to bully, humiliate, and threaten women and girls," Chiu said. "This industry has a horrific impact on one’s reputation, mental health, loss of autonomy. There have been victims who’ve been suicidal."

As the technology continues to evolve, the pressure on Apple and Google to move beyond reactive moderation and toward systemic prevention is likely to intensify. The legal notices from San Francisco serve as a clear signal that the era of "passive hosting" for AI-powered abuse tools may be coming to an end. For the victims of these applications, the hope is that the financial and legal risks will finally outweigh the commissions earned by the tech giants, leading to a safer digital environment for all users.