A Safety by Design conversation with Thorn, All Tech Is Human, Google, OpenAI and StabilityAI

In a world where the misuse of generative AI technologies pose a growing threat to child safety, our collective action has never been more vital.

Amazon, Anthropic, Civitai, Google, Meta, Metaphysic, Microsoft, Mistral AI, OpenAI, and Stability AI have joined forces with Thorn and All Tech Is Human to prevent the misuse of generative AI in perpetuating sexual harms against children. Committing to implementing child safety principles to mitigate the creation and spread of AI-generated child sexual abuse material (AIG-CSAM) on their platforms is not just a statement—it’s a call to action for the entire industry.

Learn more about these principles, the critical role of Safety by Design practices, and how we can work together towards a safer digital future for children in this conversation with Thorn, All Tech Is Human, Google, OpenAI and Stability AI. 

This video was originally recorded Thursday, April 25, 2024.

Transcript (note: the transcript is auto-generated)

Rebecca Portnoff, Thorn:

Hello and welcome. Thank you all for joining us here today. We’re going to kick things off with some introductions. I’ll start. My name is Dr. Rebecca Portnoff. I am Vice President of Data Science at Thorne. For those of you who aren’t familiar, Thorn is a nonprofit that builds technology to defend children from sexual abuse. I lead the data science team at Thorn, which is the team that builds our AI technology to accelerate our mission. I also lead Thorn’s engagement with the AI community more broadly, including this initiative which we’re here to discuss today. Now I’m going to pass it over to Matt to introduce himself.

Matt Soeth, ATIH:

Hey everyone, my name is Matt Soeth. I’m the head of trust and safety and global policy at All Tech Is Human. All Tech Is Human, is one of the largest responsible tech communities in the world, and we bring together people, organization, and ideas to grow and strengthen the responsible tech ecosystem.

Rebecca Portnoff, Thorn:

Awesome. Thank you so much, Matt. I’m going to pass it over to Emily now, could you please introduce yourself?

Emily Cashman Kirstein, Google:

Sure thing. Hi all. I’m Emily Cashman Khristine. I lead child safety public policy at Google. And fun fact, I am a Thorn alum, a proud one. So happy to be here. And before that, spend some time on the hill.

Rebecca Portnoff, Thorn:

Thanks so much, Emily. And now Chelsea, please go for it.

Chelsea Carlson, Open AI:

Hi, I’m Chelsea. I lead our child safety efforts for OpenAI. I’ve been in the online child safety space for around seven years with an original background in direct service work in the nonprofit space.

Rebecca Portnoff, Thorn:

Thanks so much, Chelsea. And now Ella, could you wrap things up for us?

Ella Irwin, Stability AI:

Sure. Hi everyone. I’m Ella Irwin and I lead product integrity at Stability ai. And I have been doing this type of product risk work for about 25 years now across different industries and just happy to be here to support Thorn and All Tech Is Human.

Rebecca Portnoff, Thorn:

Great. Well, we are so glad to have you all here today. Before we jump into questions for our panelists, I’d like to start with an overview of this issue and this initiative. Now, here’s the challenge that we’re tackling. Generative AI is being misused today to further sexual harms against children. It’s being used by bad actors to make AI generated child sexual abuse material. In some cases, they’re even using existing child sexual abuse material of specific children to make more abusive content of those same children. They’re using generative AI to sexualize images of children and then extort these kids for money or for more sexual content. And it’s not just images where this kind of abuse is occurring. The harms continue into text and video is just around the corner. This is all against the backdrop of an overtaxed child safety system. Hotlines and law enforcement are swamped with reports of child sexual abuse material, including millions of files depicting the sexual abuse of children.

Anything that adds to that haystack is a real problem. But in the middle of all of this, we see concrete opportunity. That’s why Thorn and All Tech Is Human brought together, open source and closed source, generative AI developers and providers, social platforms, search engines and more to define and commit to a set of safety by design principles to prevent the misuse of generative AI for furthering sexual harms against children. These principles cover the full lifecycle of AI from developed to deploy to maintain, whether it’s cleaning your training data, sets of child sexual abuse material, including signals in the generated content. So law enforcement and hotline analysts can better determine what content is AI generated or AI manipulated, whether it’s assessing your model for its capability to generate child sexual abuse material before releasing it, sharing transparently with the ecosystem what mitigations you put in place to prevent your model from being capable of producing this kind of content or removing models and services for notifying and sexualizing benign images from your platforms and search results. There are so many concrete and tangible ways that together we can halt this problem in its tracks because that’s why we’re here today to halt this problem in its tracks and to bring that positive impact for protecting children. Matt, are there any thoughts you’d like to add before we move to questions?

Matt Soeth, ATIH:

Yeah, and just a couple things to call out there, right, is the sheer volume of data that generative AI can present to this. This is a much bigger problem than just detection and removal. And what I love about this collaboration because trust and safety historically has been very collaborative, but this pulls in engineers, it pulls in platform designers, people who are evaluating the data and really shows a strong collaborative efforts in the industry to say, Hey, this is a problem that we can solve in multiple different levels and create these friction points. And really, as you said, I think very brilliantly just keep this from becoming a bigger issue than it already is. And all of that is pretty exciting to discuss as we look at what is really a complex issue with a lot of moving parts that’s going to help improve it along the way.

Rebecca Portnoff, Thorn:

Thanks so much Matt, and I really appreciate you highlighting both the need and the reality of that collaborative effort. So we’ll be moving into some q and a now starting with Ella. Ella, can you share with us about stability, AI strategies to innovate responsibly and safeguard children? We’d love to hear about why stability joined this initiative.

Ella Irwin, Stability AI:

Sure. Well look like many of my peers on this call and who have joined in on these commitments, we are investing a lot of time and money into developing safeguards. It starts from the point in time that we create training data sets, really making sure that we’re curating safe training data sets, which is actually pretty hard. We want to make sure we’re working with high quality, safe data and then making sure that we are effectively monitoring and detecting suspicious activity that’s happening on our API and on our platform. If we see bad actor behavior, we want to be able to zero in on it very quickly and take action and of course prevent any harmful content from being generated. That is of course the goal. And so that requires constant improvements and constant innovation because we need to stay one step ahead of the bad actors.

And as folks know, that can be extremely hard. And then of course, one area we’re very focused on is making sure that the open models that we release because we are an open model company, are safe and we know it’s possible to release safe models. And so we are investing quite a bit of time and energy in developing safeguards and embedding them in our models. And we know that this strategy has been working. We’re seeing chatter out there from bad actors who engage in this type of activity that they’re getting frustrated. They’re seeing that it’s harder with every new model released to be able to engage in creating CSA. And that makes us very happy. We’re very happy to continue to frustrate bad actors. Hopefully we frustrate the hell out of them.

Rebecca Portnoff, Thorn:

Thanks so much, Ella. I love that and I really appreciate you layering on specifically that open source perspective and what it looks like as a leader in open source to still absolutely be prioritizing child safety. Chelsea, I had love to move to you. If you’re able to share with us about child safety at OpenAI, what drove OpenAI to commit to these principles and what it’ll mean for OpenAI safety initiatives moving forward?

Chelsea Carlson, Open AI:

Yeah, thanks Rebecca. So I wanted to start with just a fundamental truth that guides the work that we do at OpenAI. Developing AI systems safely was the reason OpenAI was originally founded. The OpenAI mission statement is to ensure a GI benefits all of humanity in order for us to fulfill that mission, OpenAI and as well as our industry peers. For us child safety needs to be more than a priority. It needs to be a cornerstone for everything that we build to truly benefit all of humanity means protecting and empowering the most vulnerable among us. And that guiding principle shapes our research, it shapes our product development and it informs how we engage with the larger global community. Our commitment to the safety by design principles is motivated by the fact that we recognize that we have a profound responsibility and want to demonstrate our continued resolve in ensuring in that every step we take in advancing AI is measured against the highest child safety standards.

By participating in this initiative, all of us are setting that new standard for what it means to build AI systems in a responsible way. And on top of that, we’re also showing the world that it’s possible to drive innovation forward while also upholding a deep commitment to protecting children. And we recognize that no organization or company can shoulder this responsibility alone. Collaboration is fundamental to success in this space As anyone who works in the child safety capacity can tell you. And because of the importance of what’s at stake here, this approach becomes even more critical in my view. And by partnering with our peers, experts, and leading organizations, we are collectively able to pool our expertise, share our challenges, and ultimately hopefully multiply our successes. So for those reasons and more, I just want to say that I’m personally so grateful for all the work that it took to get here, and I’m looking forward to continuing this work with you all. Thanks.

Rebecca Portnoff, Thorn:

Thank you so much, Chelsea. I really appreciate the themes of responsibility and collaboration that you’re highlighting here. I think that those are core to the ethos of safety by design to begin with. So thank you for highlighting that. Emily, we’d love to hear about Google’s role as well in combating AI generated child sexual abuse material and the implementation of these safety principles. Can you share with us what this means for Google and its AI technologies moving forward?

Emily Cashman Kirstein, Google:

Of course. Thanks Rebecca. So at Google, we have a long track record of fighting child sexual abuse and exploitation online. And our approach to generative AI is no different. We do not want our products and services to be used to create or store or disseminate CSAM. And doing so is an egregious violation of our terms of service, whether it’s AI generated or not. And the principles we’re highlighting today compliment Google’s existing efforts in this space and has brought key players together in the industry to standardize how we can meet this moment across the ecosystem together. So at Google, we prohibit the promoting or generating content related to child sexual abuse or exploitation in our gen AI products. And we’ve built in safeguards to detect and prevent related results. So how are we doing this in practice, right? So first and foremost, before any product is launched are child safety experts and they have backgrounds in social work, law enforcement, child protection.

They rigorously test these products before launch. That includes testing variety of different languages, different modalities, things like visual synonyms that we’ve talked about. Something that if there’s something that someone’s trying to get blood making sure we’re testing for something like ketchup. Those are incredibly important pieces. But once a product is launched, there are real time protections too. And I should say all of those, the testing goes into how we go back to our teams, make sure those models are fine tuned and changed before a product is launched. But once a product is launched, the real time protections include at the prompt stage, we’ll check against known Cs, a hashes or check for keywords, prohibited terms, those sorts of things that might be trying to create violative content. And if there are any red flags, we’ll punt, which means we’re not going to allow the model to return a result.

Or instead we’ll have a response that says it can’t help or that this content is illegal and things like that at the output stage before anything is served to a user. We also run CS a m and other child safety classifiers over the response to check if it could be harmful and abusive, and again, wouldn’t generate a response if that was the case. And to echo what Chelsea was saying, as we all know, ongoing threat analysis and collaboration is key here. Now that we have these principles, there’s a shared language, a shared understanding across industry and civil society. And that’s key to have important and sometimes tough conversations and build upon this progress that’s already been made. And think about this as the threats morph and evolve. And these principles are really a roadmap for how we think about these protections as we respond to that evolution of this crime, especially when it comes to generative ai.

Rebecca Portnoff, Thorn:

Thank you so much, Emily. I feel like we’ve gotten a great window into how Google is engaging with these principles. And I know for me, it always makes me happy to hear that developed, deploy, maintain, get echoed across this type of work. So thank you so much, Matt. At this point I’m going to turn it over to you to ask the next question to each of us in the group.

Matt Soeth, ATIH:

Yeah, I am kind of curious about this. Obviously my previous experience at TikTok and being a trust and safety for a bit industry collaboration is often an interesting endeavor in terms of who can say yes, how they can say yes, and the internal alignment of getting people to agree to this. And so as we think about these principles and the agreements to these principles, I’m curious from this group, what are some of the biggest hurdles and obstacles in uniting tech companies around this specific initiative? And then why this really feels like unique time with the gen AI in the industry. So curious what you’re all thinking in terms of why does this seem like a momentous event and what did it take to get that internal alignment to push forward on these gen AI principles? Open question, Ella will throw it to you first.

Ella Irwin, Stability AI:

Sure. Look, I would love, I would maybe say this was hard, but it actually wasn’t hard to drive alignment internally on this. I think everybody at stability believes this is a really important area to focus on. So I think there was actually a lot of support for all of these commitments.

Emily Cashman Kirstein, Google:

Yeah, I’d agree. I would just say that I think this moment in particular is one that we all recognize deserved urgent attention and focus and that’s why we’re here. But collaboration on child sexual abuse and exploitation has always been a core pillar of how Google approaches this. And I’d venture to say how industry has approached combating CSAM for a number of years now. We know certainly at Google that we can’t and shouldn’t go it alone. So leaning into this collaboration in particular we knew was important to us and something we were going to prioritize. And that’s something we would do across different combating child sexual abuse in any form it may take now and down the line. And I think it’s also important to recognize that we weren’t starting from scratch here, right? This is a unique and pivotal moment, but there are a lot of learnings across industry and civil society that brought us here to this moment. For example, as I mentioned, we’re using technology that’s been used for years like hash matching and classifiers to combat Gen I-C-S-A-M, and it’s effective. And so while we have the building blocks and the relationships and the understanding of this need to collaborate, that made this really a safety by design moment as was very aptly named in the principles.

Chelsea Carlson, Open AI:

And if I could just tack onto that, I really 100% agree that we already sort of had a strong background of collaboration on this topic. The tech companies have a long history of working together. But if I could speak to what could be a challenge in making something like this go from an idea to where we are now, is that there’s just a lot of moving parts. The actual nuts and bolts of how we implement these safety principles are sometimes it requires quite a lot of technical experts to weigh in and point out where things are possible, where things are possible, but difficult and point out potential other solutions. And having that all shake out into something concrete, substantial and doable by everybody involved. There is a little bit of complication there, but given our history of working together and already having, as Emily said, a lot of things in place that we could jump from made it not as challenging as maybe we might’ve expected.

Rebecca Portnoff, Thorn:

Matt, I’ll just close this out here and be a little bit of a contrarian, although I certainly agree with what folks are saying about how we had many of the building blocks in place and there was a lot of learnings to come off of. So I’ve worked in the intersection of AI and child safety for over a decade now, and candidly I’ll share that what I’ve observed repeatedly is two big obstacles in uniting tech companies to date. One is this sense of the issue is too complex. It’s too difficult. It’s too horrific. I can’t look at it. And then the other builds off of that. Well, because it’s so complicated and there’s no perfect solution, I can’t try, I’m not going to try. And what these initiatives and the commitments and the people who gave their time and their energy for a year to make this happen, what this is is a testament that yes, while the issue is horrific, we have to face it. And yes, while it’s complicated, it’s not impossible to tackle. And in fact there are many feasible technological and policy solutions that we could employ to meet this moment.

Matt Soeth, ATIH:

Those are some great observations. One from having the building blocks the coalition that we are aligned on child safety as being an important issue. And sometimes, and this is a big thing, and All Tech Is Human, these are very sticky, thorny, messy issues that we need to take head on. And really, whether it’s from technical solutions, policy solutions, collaboratively, really need to work towards a solution rather than like, oh, it’s really bad. I dunno if we can deal with that right now, but we’re just like, no, we can take this, but it’s going to take this group effort in order to kind of take a big leap forward. So no thanks for that.

Rebecca Portnoff, Thorn:

Absolutely. Yeah. Well, thank you all so much for your thoughtful reflections on this initiatives. We’re going to open up some time now to answer questions from viewers, so please feel free to throw your questions into the chat and we’ll keep an eye out for those. We’re going to kick things off with a question here for open ai. Chelsea, could you share your reflections on any common misconceptions about child safety and generative AI that you’re familiar with or that you would want to address?

Chelsea Carlson, Open AI:

Yeah, actually, so one question I get quite often when I talk about the work that I do is with regards to AI generated CSAM, is that not better than non-AI generated CSAM and is it in fact beneficial to the world for people to be in possession of that instead of C? The traditional CSA? And I mean, I can give my personal take on this. I won’t give a lot of background for why I believe this, but I believe it’s just as harmful and it has the potential to be worse in some scenarios. There is the capability of fine tuning on a person’s likeness and depicting even worse abuse than ever occurred long after a victim has been safeguarded. There’s so many different iterations of how this could be very, very harmful, and the misconception that this is somehow not as harmful of an abuse type is something that I really wish more people would think about.

Rebecca Portnoff, Thorn:

Thank you, Chelsea. I really appreciate you getting to the heart of that. And certainly those who are familiar with the research know that there is a link between conception of this kind of content and hands-on abuse. And so recognizing the harms that come with this that they are real is definitely vital. Let’s move to the next question, Ella. I think this one is for you. Can you share with us some of the biggest challenges that you faced or that you’re facing in implementing child safety measures at stability ai?

Ella Irwin, Stability AI:

Sure. And I would probably say this is a common challenge based on some of the conversations I’ve had with peers in the industry. Look in other fields, and there’s folks here who have done this in many other areas, whether that’s social media or search or other areas. Normally you have historical data you can work with, you understand bad actor behavior in terms of how bad actors are exploiting your technology, what gaps are being exploited, et cetera. This is a nascent industry and for many of us, the data surrounding what bad actors are doing has been nebulous and it’s hard to get to. And certainly for a new company like stability, that is true. And so what has been very challenging is trying to solve a very difficult problem. You’ve heard from everyone how complex this is without having actual examples of the problem enough to work with to be able to solve it.

And so what we have found a lot of benefit and success from is we’ve built very good relationships with law enforcement agencies all over the world. We’ve had very good deep conversations around what they’re seeing. I mean, they are out there arresting, investigating, prosecuting the people who perpetrate this type of harm. And by sharing, I mean these are very difficult conversations psychologically. They’re very difficult on anyone who participates in these conversations, but they’re so important because you have to be able to see these examples. You have to understand exactly how bad actors are using these models and these tools, what gaps are they exploiting? What are the most popular prompts? What kind of images are being generated if you don’t actually understand the problem? We’ve brought researchers into these conversations so that they can see firsthand what’s happening because then they walk away from the conversation and they say, I know exactly how we can solve this. I know how we can close this gap without having that level of information. It’s extremely hard and it’s been difficult to even get to that point. So I think it underscores the need for collaboration and information sharing across companies and law enforcement agencies and nonprofits who are working in this space. I think it’s absolutely critical.

Rebecca Portnoff, Thorn:

That really resonates Ally, I think, in general, finding opportunities for folks on the ground who know exactly what’s happening, who are in it day to day, to be able to collaborate directly with the technologists who are building out this next generation of technology, that kind of collaboration is really critical. Thank you for sharing your thoughts. This next question is for Google. Emily, could you share with us some of your thoughts on what parents should know about AI safety for kids?

Emily Cashman Kirstein, Google:

Definitely happy to. I think it’s important for parents to know about the serious issues we raised here, the risks, the protections that are put in place specifically for CSAM and the ways that generative AI can be misused. But it’s really also important to think about the other side of this generative AI as a technology and the incredible opportunities and possibilities that come with it, and how do we ensure we think about teens in particular, how do they have the tools they need to use this technology well and prepare them for a future where it’s part of everyday life? We thought about this a lot and actually didn’t allow teens on Gemini until we felt like we had the right protections in place and educational moments in place to support them. And a key piece of that education and thinking through all this was developing a teen gen AI literacy guide, which lives on our help site, but it’s also embedded into the onboarding for teens.

And it goes through really important themes. Like AI is a machine learning model. It’s not a human. It can and will make mistakes, how important it is to think critically and evaluate responses. And this guide was written specifically for teens, but from personal experience, I can tell you it’s actually quite helpful for adults as well. So we have it in writing, but there’s also a great video that helps explain things really well. Folks can find it on YouTube and maybe I can ask the thorn folks to share the link in the chat. I think the other piece to this is, and Thorn has great resources here, having conversations is the key to everything from parenting and talking to your kids about all aspects of being online and online safety. And that’s also true when it comes to generative ai. And I know Lauren for parents is incredibly a wonderful resource on some really tough issues and conversation starters, but I think across online safety and including generative ai, there’s nothing better than being able to have conversations with your kids.

Rebecca Portnoff, Thorn:

Thank you so much, Emily, and I appreciate you highlighting the importance of those conversations and having them in a way that are really meeting kids where they are. So thank you for sharing your thoughts there. Matt, I think I’ll give this question to you because it touches on that ecosystem perspective. Can you share with us how do you think these principles are going to help the entire whole tech ecosystem keep children safe?

Matt Soeth, ATIH:

Yeah, I think we’ve touched on some of these, right? From collaboration with researchers. I really appreciate Emily’s answer because a lot of the people we connect with are tech adjacent, right? Civil society who are all talking and working on the subject, but often don’t have a direct connection to tech itself in order to what they feel would have any influence on this. So these principles not only show like, Hey, here are the things that are happening, but two, it gives us a shared vocabulary and language so we could talk about these solutions in a way that’s consistent both in industry and both out of industry as well. So as you think about safety by design and advocacy, now we have sort of a directed consistent approach across platforms. And I know too, talking to a lot of trust and safety friends, there’s a frustration when engineering, for years, the meme was engineering coming to me like, Hey, we’re launching a new feature.

It should go live on Friday. And you’re like, I didn’t know about it. Who decided all this? And then we kind of find after the fact that there are instant safety issues that come up as a result of that. Now having these principles in place, it kind of gives that playbook as it were, where you can go to engineering, you can go to data science and others and advocate be like, no, here are these guidelines. Here are the companies that are doing it. Here’s the things that we should do in order to meet the need in terms of before we launch these new features and integrate these new resources. And really just being able to have that conversation, having that tool I know is going to be a win for everybody as we think about how do we tackle this larger issue.

Rebecca Portnoff, Thorn:

Thank you so much, Matt, and I really appreciate that framing of this as a resource, as a playbook for folks on the ground to be able to just get moving on having this impact right away. So we have one last question here that if the group’s okay with it, I’m going to give to myself. The question is, can AI also actually help and keeping kids safe? And I’m very happy to say the answer is yes. My day job here is to lead a team that builds machine learning and AI to accelerate our mission, to accelerate victim identification, to stop re-victimization the spread of child sexual abuse material, and to prevent abuse from happening in the first place. We have seen, I’ve had the honor of seeing firsthand how these kinds of tools help support the prioritization and triage efforts of those folks on the ground who are having to engage with this kind of content.

So happy to say yes, there is a positive impact that can be had there as well. Well, I know we are coming up against time. Thank you so much. I’ve really enjoyed this conversation with each of you. I appreciate you taking the time to join into this important conversation. For those in the audience, if making Technology for Children is important to you, I want to invite you to consider supporting Thorn. And All Tech Is Human. Thorn is a nonprofit that relies on donor support to make initiatives like this one happen. You can learn more about the work that we do to defend children from sexual abuse@thorn.org, and please feel free to visit the QR code to become a supporter. Thank you so much for all of your time.