Between Hype and Hesitancy: How AI Can Make Us Safer Online

Speculative scenarios about superintelligence causing human extinction often drive the global conversation about AI and online safety. AI brings both risks and opportunities, now and in the future. But by focusing too much on long-term risks, we obscure our view of what is happening right now. The fact is that AI (and, increasingly, generative AI specifically) is fundamental to the systems that keep people safe online.

Trust and safety is the industry term for the teams dedicated to keeping digital services safe. From social media to search engines and dating apps to online games, any service that enables users to create and share content or interact with each other faces risks of misuse. For decades, companies have been combining people, processes, and technology to tackle tough challenges, like keeping kids safe, fighting terrorism, and preventing suicide and self-harm.

Trust and safety teams have been using automated systems for decades, going back to the earliest efforts to keep spam out of email inboxes. Today, those teams deploy increasingly sophisticated technology, including a wide array of AI systems, to detect and enforce their policies against abuse.

Describing how AI is applied in trust and safety, including its limitations and risks, helps us avoid the trap of AI determinism—the belief that AI will save us from every social problem or somehow ruin everything.

The clear and present safety risks from generative AI are widely discussed. Deepfake images, now trivially easy to produce, are used to scam, deceive, and harass. But AI does not just enable bad actors; it’s also a critical tool for trust and safety teams as they fight harassment, scams, and other forms of abuse.

Even when existential threats are put in perspective, our view of AI and safety is incomplete if we focus only on risks, and not on how AI is part and parcel of avoiding and mitigating these harms. Moreover, increasing public understanding of how AI is used in trust and safety will enhance societal decisions about who is responsible for AI governance and how it should be done.

Our organization spent much of 2024 bringing together representatives from a dozen digital companies to identify the ways they are currently using AI and automation in their trust and safety operations, as well as how they are thinking about applying generative AI. Three overarching findings paint a more nuanced picture of AI in content moderation that provides grounds for carefully considered optimism.

First, AI will not wholesale replace the trust and safety workforce, but it will aid the people who do this difficult job and alleviate some of their biggest challenges. Contrary to bold pronouncements that robots will somehow make human decision-making around content moderation obsolete, trust and safety will continue to consist of blended processes that combine people and technology. But AI advances will help shield humans from especially traumatic content, and free their attention to focus on the most challenging cases.

Second, generative AI can help companies do more and do better. In particular, as models advance, they may allow companies to better cater their services to more regions and underserved communities. They can also improve their safety products and services in a more efficient way. For example, improving the capability of AI models across languages has been a consistent challenge due to asymmetries in available training data. Notwithstanding some limitations, advances in generative AI models can potentially make equitable language performance more feasible, especially if companies increase investments in this sphere and disclose more about the languages that models are trained on.

Third, while AI’s limitations are real, there is a clear path forward. The best practices we have identified encourage collaboration across the trust and safety field and will help practitioners understand how to responsibly and effectively use AI now and in the future.

We are not understating the risks, short-term or long-term, posed by harmful uses of AI. From computer-generated child sexual abuse material to the industrialization of complex fraud and scams, bad actors are undoubtedly employing AI in novel ways around the world.

The day-to-day work of digital trust and safety involves using automation and AI to detect, review, and enforce against abusive content and conduct. The more that the public knows about the essential role of technology in this process and the changes coming with generative AI developments, the better we can all understand that the relationship between AI and online safety risks is a two-way street. Yes, AI can be an accelerant of online abuse, but it also enhances the capabilities of the teams countering that abuse.

The right response to these threats requires better use of technology, not forsaking it.