Safety by Design: One year of progress protecting children in the age of AI

Generative AI is moving fast. It’s reshaping how we work, connect and live- and it carries real risks for children. At Thorn, we believe this is a pivotal moment: safety must be built in now. 

That’s why, in April 2024 we partnered with All Tech is Human to launch our Safety by Design for Generative AI initiative. Together, we brought some of the most influential AI companies to the table, securing commitments to prevent the creation and spread of AI-generated child sexual abuse material (AIG- CSAM) and other sexual harms against children. 

Today, we’re proud to share the first annual progress update. This update highlights real wins achieved by this initiative*, progress from industry leaders, new standards shaping the field, and the global policy milestones that are beginning to shift expectations.

This is worth celebrating. For the first time, leading voices in the AI ecosystem have committed not only to tackling the risk of AI-generated child sexual abuse material (AIG-CSAM) and other sexual harms against children, but also to documenting and sharing their progress publicly.

What we’re learning together

Just as important as the progress itself are the collective learnings from this first year. Across companies and the AI ecosystem, we’ve seen where safeguards work, where they fall short, and where new risks are emerging. These insights are helping to shape a clearer picture of what it will take to keep children safe in the age of generative AI.

The following sections celebrate the progress we’ve seen, and also point to the lessons that will guide the next stage of this work.

Progress worth celebrating: What’s changed this year

• Companies detected and blocked hundreds of thousands of attempts to generate harmful content.
• Hundreds of models that were built to create child abuse imagery were removed from platform access.
• Multiple training datasets at leading AI companies were filtered for CSAM and other abusive material.
• Millions of prevention and deterrence messages surfaced in response to violative prompts.
• Hundreds of thousands of reports of suspected child sexual abuse material—covering AI-generated and non-AI content—were filed to NCMEC’s CyberTipline.
• Companies invested significant resources in new protective technologies and research.

These are real wins. They demonstrate that, when implemented with intention and rigor, Safety by Design is effective. But we’re also seeing clear patterns in what’s missing.

What we’re learning

• NSFW ≠ CSAM. Generic adult content filters used to clean training data may miss necessary signals specific to CSAM and CSEM, including low-production attempts to obscure origin. CSAM-specific tooling and expert review are necessary.
• Removing CSAM from training datasets isn’t enough. Compositional generalization continues to be a real risk. If models learn harmful associations, like benign images of children + adult sexual content, they may produce abusive outputs even without explicit training data. More companies need mitigation strategies here—across all modalities, not just audiovisual.
• Independent evaluation matters. Third-party assessments broaden coverage and surface blind spots that internal teams may miss. This is especially true given legal constraints on handling CSAM. We need structured public-private partnerships to ensure testing can be conducted lawfully and safely.
• Provenance must be adversary-aware. Some approaches (like simple metadata) have value but are too easy to strip. Adoption across open-source releases is nearly zero. Without durable, interoperable, and proven provenance for openly shared models, we’ll continue to play catch-up.
• Open weight ≠ open season. When companies release open-weight models, they must carry forward the same safeguards they apply to closed systems—e.g. training data cleaning, user reporting, and documented misuse testing. They also need to invest in researching solutions that are robust to the unique risks of open weight (e.g. adversarial manipulation downstream).
• Hosting and search are untapped chokepoints to prevent harm. Third-party model hubs and search engines are not consistently pre-evaluating uploads or de-indexing abusive tools (e.g., nudifiers, or chatbots with adult personas that sexually engage with child users). That makes harmful capabilities too easy to find and use.
• Agentic and code-gen systems need a plan. Agentic AI allows LLM’s to control workflows and interact with external environments and systems; code-generation allows developers to generate code by prompting LLM’s. We’re not yet seeing meaningful mitigations to prevent misuse of these tools to e.g. build nudifiers or to automate sextortion schemes.
• Reporting needs more granularity. Hotlines and companies should clearly flag (when they have this information) whether content is AI-generated, AI-manipulated, or unaltered (as well as information like prompts and model information). This detail helps law enforcement triage and prioritize real-world victim identification.

Setting the standard for safety

Changing how individual companies act is critical, but it’s not enough. To truly protect children, we need shared rules of the road that raise the baseline for the whole industry. That’s why standard-setting and policy engagement are such a big deal: they make child safety a requirement, not just a choice.

This year, we were proud to see Thorn’s perspective included in major standards and policy milestones around the world—from global technical bodies to landmark regulatory guidance. These inclusions reflect real steps toward making child safety a foundation of how AI is built and governed.

Standards

IEEE Recommended Practice

IEEE develops many of the standards people rely on daily, like Wi-Fi and Bluetooth. 

In 2024, Thorn led the effort to establish an IEEE working group, to draft the first international standard embedding child protection across the AI lifecycle. This Recommended Practice will formalize Safety by Design as a global best practice, drawing on input from technologists worldwide. 

In September 2025, the IEEE working group advancing this standard voted to send the draft Recommended Practice to ballot.  Embedding Safety by Design as an IEEE standard will provide a solid foundation for safer development, deployment, and maintenance of generative AI models.

NIST AI 100-4

The National Institute of Standards and Technology (NIST) is part of the U.S. Department of Commerce. Its mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve quality of life. NIST’s standards are highly respected and globally influential. 

NIST’s AI 100-4 standard is focused on reducing risks from synthetic content, including AIG-CSAM. Thorn was invited to provide early feedback and deep guidance into this groundbreaking AI standard that helped shape new U.S. best practice on reducing risks from synthetic content.

Policy engagement

• EU AI Act (2025): Thorn was one of the civil society organizations contributing to the EU AI Act Code of Practice, EU’s first major regulatory tool for AI. Our input helped secure key wins—like requiring companies to document how they remove CSAM from training data and treat CSAM as a systemic risk.
• Global briefings: We’ve hosted webinars for policymakers in the U.S., U.K., and EU, and our work was recognized by the White House as part of efforts to combat image-based sexual abuse.

These advances don’t solve the problem on their own, but they set clear expectations and create accountability, ensuring child safety is built into the foundation of AI.

What the ecosystem must do next

Protecting children in this digital age is everyone’s job. Technology companies have a critical role, but they can’t do it alone. Here are some steps we recommend for those in the broader child protection ecosystem:

• Builders: Use specialized CSAM detection tools to clean training data, and incorporate safeguards to both closed and open models
• Platforms & search: Remove nudifiers, tutorials and abusive models from results, evaluate third-party models and technology before providing access, and make it easy to report and enforce against harmful content.
• Policy & governance: Give law enforcement the resources they need, update laws to keep pace with AI, and allow safe partnerships for red-teaming.
• Schools, caregivers & youth: Talk openly about risks, especially as misuse shifts toward peers. Simple, judgment-free conversations go a long way.
• Academia & industry: Invest in research for more robust provenance solutions and safeguards in the open-source setting, stronger detection of synthetic content, better safeguarding of children’s imagery, and safeguards for new AI systems like agents and code tools.

What’s next from Thorn

For more than a decade, Thorn has helped the ecosystem get ahead of emerging threats to children. We:

• Conduct cutting-edge research to understand the ways technology is changing the landscape for online child safety.
• Partner with builders to embed Safety by Design across the AI lifecycle.
• Provide technology (like Safer) and expert consulting to detect, review, and report child sexual abuse material and other exploitative content at scale.
• Convene cross-sector collaborations, contribute to and promote standards, and support policy that raises the baseline for everyone.

The progress we’ve seen this year proves what’s possible when safety is built in from the start – and the gaps show how much work remains.

If you’re building or deploying AI, now is the time to act. Together, we can continue to ensure even more progress—and fewer opportunities for harm.

*DISCLOSURE: The information and conclusions contained in this blog rely in part upon data that has been self-reported to Thorn by the companies participating in the Safety by Design effort. Thorn did not independently verify this data. For more information regarding data collection practices and use rights, please see the full report.