From Safety to Innovation: How AI Safety Institutes Inform AI Governance

Governments around the world have been trying to keep pace with rapid advancements in artificial intelligence. In this context, a number of governments have established AI safety institutes (AISIs). Seven jurisdictions (Japan, the UK, the US, the EU, Singapore, France, and Canada) have some version of an AI Safety Institute. Most recently, India announced public consultations to set up its own institute, indicating that this model is also being adopted by countries in the Global Majority.

A new report from Renan Araujo, Kristina Fort, and Oliver Guest compares these institutes across jurisdictions, providing insights into their characteristics, functions, and limitations in shaping global AI governance. AISIs are the “first institutional models of AI governance that different governments have adopted in a similarly shaped way,” according to the report’s authors. Given these similarities, it is worth exploring how AISIs work in practice and the challenges they may face in the years ahead.

This report focuses specifically on jurisdictions that explicitly use the term “AI Safety Institute” and identifies three countries, the United Kingdom, the United States, and Japan, as constituting the “first wave” of AISIs. The UK and the US were the first to establish an AISI in November 2023, and Japan followed suit by setting up its AISI in February 2024.

The report’s authors left the EU AI Office out of this “first wave” because it is a particularly unique institutional development that took a different and more expansive approach compared to other countries. With the enactment of the EU AI Act, the EU AI Office regulates the entire AI market in the EU, giving it a "much broader and stronger mandate than the first wave of AISIs," which are largely focused on developing safety evaluations and technical standards.

According to the report, this “first wave” of AISIs has three common characteristics:

• Safety-focus: The first wave of AISIs was informed by the Bletchley AI Safety Summit, which declared that “AI should be designed, developed, deployed, and used in a manner that is safe, in such a way as to be human-centric, trustworthy, and responsible.” These institutes are particularly concerned with mitigating abuse and safeguarding frontier AI models.
• Government-led: These AISIs are governmental institutions, providing them with the “authority, legitimacy, and resources” needed to address AI safety issues. Their governmental status helps them access leading AI models to run evaluations, and importantly, it gives them greater leverage in negotiating with companies unwilling to comply.
• Technical: AISIs are focused on attracting technical experts to ensure an evidence-based approach to AI safety.

The report also points out some key ways AISIs are unique. For one, AISIs are not a “catch-all” entity to tackle the complex and ever-evolving AI governance landscape. They are also relatively free of the bureaucracy commonly associated with governmental agencies. This may be due to the fact that these institutes have very little regulatory authority and focus more on establishing best practices and conducting safety evaluations to inform responsible AI development. Thus, AI Safety Institutes provide three key functions:

• Research: AISIs seek to advance the "science of AI safety" through empirical research focused on identifying and mitigating risks. For instance, the UK AISI has made significant progress in conducting safety evaluations of AI models and open-sourcing its testing frameworks.
• Standards: AISIs play a pivotal role in developing AI safety standards. While they do not have regulatory authority, they contribute to shaping industry practices by establishing evaluation benchmarks and safety guidelines. Standards for AI governance are far from perfect and this is where AISIs step in. For instance, the UK AISI is committed to conducting evidence-based research to “build an evaluations process for assessing the capabilities of the next generation of advanced AI systems.” That being said, the report finds that the US and the EU have the most detailed plans related to standard-setting. The National Institute of Standards and Technology (NIST), which oversees the US AISI, published an AI Risk Management Framework (RMF), which influenced countries like Japan to model their AI guidelines in accordance with the RMF.
• Cooperation: AISIs also serve as bridges between governments, industries, and academic institutions, fostering international collaboration on AI safety. In fact, the United States launched the International Network of AI Safety Institutes during the AI Safety Summit in Seoul, and its first convening is scheduled for November 21, 2024, in San Francisco.

While AI safety institutes are critical to fostering responsible AI, they also have some limitations. The narrow focus on “safety” has led to concerns that other AI harms are being sidelined, while other critics say this may hinder innovation and competitiveness. AISIs may also clash with the mandates of other institutions, creating redundancies with existing bodies that may already have the resources and capacity to do what AISIs are tasked with. The report argues that while this may be the case, redundancies or overlapping mandates are not necessarily bad. Multilateral organizations are often slow and may not keep pace with the rapidly evolving AI landscape.

“One way forward would be to pair the value add brought by AISIs with the legitimacy and representation of such international organizations,” the report says. Another set of concerns with AISIs has to do with their relationship to industry, particularly around fears that close ties with companies might lead to “regulatory capture,” undermining the impartiality and independence of these institutes. This is particularly tricky because AISIs need good relationships with AI companies to access and evaluate AI models.

The “second wave” of AISIs could look very different, especially in developing countries that might focus more on “innovation or AI benefits alongside safety.” Initiatives like China’s AI Safety Network are already underway, while India’s AI Safety Institute is currently in the works. Whether the Global Majority’s approach to AI governance emulates the status quo in the US, UK, and Europe or upends it, only time will tell.