The State of AI Safety in China, with Kwan Yee Ng & Brian Tse

Our findings in the report suggest that China is more invested in AI safety and risk mitigation than many may realize. From China’s domestic rules on generative AI and internationally directing the Global AI Governance Initiative to technical safety research and expert views on frontier risks, China is much more active on AI safety than many Western commentators might suppose, and as a result I think international coordination is more feasible than many may think.

In 2024 we are going to witness an unprecedented level of international dialogue on this topic with the Global AI Safety Summit in South Korea and France this year, the UN Summit of the Future, China-U.S. intergovernmental dialogues on AI, and more. As a leading AI power and critical player in international governance, China should be part of this discussion.

At the moment, my wonderful colleague Jason Zhou is working hard to update our report, so maybe later in the discussion we can get into some more specific developments since last October, when the report was initially released. That might be interesting to the audience.

WENDELL WALLACH: Brian, what was most surprising to you as you did this research?

BRIAN TSE: First of all, it is great to see you again, Wendell. I agree with Kwan Yee that the Chinese ecosystem of AI safety and governance is richer and more diverse than many international audiences may realize. Some of you may already know that China has introduced a series of targeted and binding regulations in recent years that include regulations on recommendation systems in 2021, the rules for deep synthesis or deepfakes in 2022, and the rules on generative AI in 2023.

There are also many more layers and approaches that China possesses for governing AI. For example, China created a national science and technology ethics review including for AI by 2023. New standards have also been issued on AI safety and security. We also found out that a number of key local governments from Beijing to the lesser-known Anhui province have issued artificial general intelligence (AGI) or large model policies with some safety and ethics provisions, and now China is debating and drafting an overarching national AI law that could be rolled out in the years ahead. There are several expert groups drafting their own suggested versions of the national AI law.

This is just the domestic governance session, which is one of the six sessions in our State of AI Safety in China report. We internally expected the report to be about 50 pages long, but it turned out to be over 150 pages, so that was a surprise to me.

WENDELL WALLACH: I have also been surprised in the United States how robust the attention to these issues is. I do not think we have anywhere near the legislation that China has put in place so far.

Do you feel when you look at the report that you were surprised that there were so many things going on? You mentioned the length, and that is what we would presume by the length of the report, but did you find a whole flurry of different activities going on that you never thought were taking place?

BRIAN TSE: I think for us, we have been involved in doing AI safety and governance work domestically in China, so it is not a big surprise to us, but we really want to advise international cooperation and understanding by reducing the information asymmetry. What I mean is that in China AI safety and governance experts follow what is happening with the EU AI Act or superalignment work at OpenAI on a daily basis. However, their international counterparts often know very little about relevant work in China.

When we published our report in English this was the first effort that we are aware of that seeks to comprehensively map out the AI safety and policy landscape in China for an international audience. We felt it was especially critical when experts and policymakers in the United Kingdom were deliberating and to some extent debating over whether the first Global AI Safety Summit should involve China during the summer of 2023.

At the time, we at Concordia AI believed that the summit represented a unique opportunity to bring major global powers including China around a table to discuss significant safety risks that would likely require international cooperation, and we believe that there are several concrete areas where there could be productive discussions and even agreement, so we made a public case for this.

For example, we thought that a top priority for the first Global AI Safety Summit would be to discuss the safety risks associated with frontier AI models and try to agree to a shared statement. Eventually, when I personally attended the AI Safety Summit at Bletchley Park, I was very glad to see that China, the United States, the United Kingdom, European Union, and more than 25 other countries signed onto the Bletchley Declaration.

WENDELL WALLACH: If I understand clearly then, this was never intended to be for a Chinese audience, a survey for them. You feel that within the safety community within China there is a fairly good understanding of what is going on. This was always specifically directed at the international audience to make sure the international audience had a much better understanding of what was taking place in China in the hopes that that would open the door for more international cooperation. Is that correct?

BRIAN TSE: That is correct. That is the core aim. While we mainly published the report in English we also released an article in Chinese, and many of our colleagues and collaborators in China also appreciated the effort. In the process we also consulted a number of experts and supporters in China just to make sure that what we communicate and publish is in line with their understanding and is comprehensive in terms of the landscape.

WENDELL WALLACH: Kwan Yee, I know that you and Brian have already mentioned a number of the elements of the report, but perhaps you can provide us with a brief overview of AI safety and security research going on in China that might not have been talked about so far. I think we are talking about AI safety and security and whether there is any work going on in alignment research.

KWAN YEE NG: Of course. Before diving into the report content, I will just briefly define here what we mean by “AI safety.” For the report we focused on safety risks from foundation models and frontier AI, which was also the focus of the Bletchley AI Safety Summit. Concretely AI systems have already demonstrated their potential to disrupt global security by enabling things like large-scale disinformation campaigns, cyberattacks, and misuse of synthetic biology. Some have argued that the horizon could also hold the prospect of more autonomous general-purpose systems that could even escape human control, so when we looked at Chinese materials—documents, speeches, surveys, et cetera—for this report we focused on material that was oriented toward these kinds of risks.

We divided, like Brian said, the report into six sections. With regard to safety alignment research, Wendell, that you mentioned, I think the first takeaway here is that technical AI safety research in China has become more advanced in just the last year. There are numerous Chinese labs across the country publishing papers on frontier safety-related topics like model autonomy, safety evaluations, potential AI misuse in chemical, biological, radiological, and nuclear (CBRN) defense and more. I do not expect us to get into the technical details in this podcast, but six months since the initial publication of our report in October—as I said, we are working on an updated version; I just want to flag the sheer increase in volume and quality of technical safety research has been one of our biggest updates since last October.

Just briefly, a few other key takeaways: The second one is that with regard to industry, Chinese companies relative to their Western counterparts have largely adopted a more passive approach to self-governance on frontier risks, but state-affiliated industry associations that coordinate between Chinese companies have been much more active.

My third point here is that the weak data we could find shows the Chinese public generally seems to think that benefits from AI development outweigh the harms, but—this is the fourth takeaway—leading Chinese experts, including top scientists and policy advisors expressed many more concerns about frontier AI risks.

With the two other sections in our report, domestic or international governance, over the last year China has stepped up its efforts to position AI governance as an area of international cooperation, especially with the release of the Global AI Governance Initiative last October by President Xi Jinping.

Finally, China has developed domestic governance tools like an algorithm registry that, while they are not currently used to mitigate frontier risks, can be shaped and employed that way in the future.

WENDELL WALLACH: Just for our audience’s understanding, the term “frontier” applications or frontier research or frontier models, that is pretty well known within technical circles, but outside of technical circles people may not know what you are referring to. You both use those terms, and I just think maybe it would help the audience understand what you are talking about focusing upon, because you are not necessarily talking about focusing on, let’s say, technological unemployment when you use a term like that.

Kwan Yee, maybe you can clarify that for us for a moment.

KWAN YEE NG: You are right, Wendell. When we talk about frontier AI risks, while we definitely think issues like technological unemployment are very important, we are more focused on issues that could be brought about for models that are at the cutting edge of AI development at the moment. An example of this would be GPT-4 or Google Gemini, generative AI systems that could have potentially a wide range of uses across different domains.

I mentioned earlier as well that there is also a concern about maybe some more narrow AI applications in safety-critical areas, for example, the development of synthetic biology, that could be used for great benefits like drug discovery but could potentially be misused, for example, for developing toxic pathogens when in the hands of wrong actors.

WENDELL WALLACH: Those are the convergence issues, the issues where AI may actually open the door for other technologies to be used in ways that are not necessarily being considered, but let me focus this a little bit more because as I read the report you are not focusing at all on military applications. That is not what your concern is here. You are really looking more at the applications of these generative models. Is that correct?

KWAN YEE NG: Yes, that is completely correct. Military applications is somewhat outside our domain of expertise, but we focus on models that could have potentially dual-use applications in civilian cases as well.

WENDELL WALLACH: When you look at generative models there are these questions around security, particularly of future systems and whether they are going to be controllable or not. If I am understanding you correctly, you see the biggest threats from that coming from these generative models, and you are looking at that most specifically in terms of what the research is that it focuses on, on large language models and other generative AI applications. Is that correct?

KWAN YEE NG: That is exactly right.

WENDELL WALLACH: Let’s talk a little bit about these frontier models. What is the government doing to regulate their development in China, Brian?

BRIAN TSE: Just as a big-picture context, China’s national AI plan in 2017 set the objective to begin developing local and sectoral-specific regulations by 2020 and begin establishing AI laws and regulations by 2025. Between 2017 and 2020 we saw binding AI regulations were confined to narrow application areas such as drones and self-driving cars, and then in 2021 the recommendation system regulations really represented the country’s first broad regulation impacting AI.

As Kwan Yee mentioned, the regulation created an important regulatory tool, a registry, so certain AI recommendation providers were now required to start filing their systems and algorithms with the Cyberspace Administration of China, and security assessments were required for systems which could promote functions that could destabilize society and mobilize public opinion. It really sought to address a mixed collection of social issues ranging from content control to privacy and algorithmic discrimination.

Later we also saw regulations on deepfakes and generative AI in the last two years. They also require developers and providers to make a filing to the registry and also conduct a security assessment, but they also introduce other safety-relevant requirements like mandating the watermarking of deep synthesis and other AI-generated content in order to address concerns around large-scale disinformation.

While some of the earlier regulatory focus has been on content control, serviceability, and data security, these regulatory tools could also be applied to regulating frontier AI risk. For example, if regulators become concerned about the risk of the AI creating and generating harmful chemical substances—and I think they are increasingly demonstrating concerns in this direction—these concerns could very easily be folded into the existing registry system and security assessment requirement.

WENDELL WALLACH: Brian, you mention these different layers of governance, and I wonder if you could perhaps clarify that for the Western audience. I think some in the West have a very simplistic model of what governance is like in China and that it is all simply top-down, but I think perhaps you can clarify for us what the different layers of governance are.

BRIAN TSE: Sure. In our report we divide up the policy landscape in terms of different layers ranging from the harder measures like binding regulations to softer measures like voluntary standards.

The first layer is what I talked about earlier in terms of binding regulations like for generative AI. The second layer we talk about in the report is standards. While they are initially introduced as voluntary, regulators frequently cite them in their regulations, so over time these standards can evolve into de facto requirements for the industry. This places the standards in a gray area between hard law and soft law.

The third layer is the science and technology ethics systems, which seems pretty unique to Chinese AI governance of the major AI jurisdictions like Europe and the United States. It tries to incentivize AI developers to align their systems with ethical principles with a primary focus on bias and privacy and with some attention to human control as well. This is not specific to AI. It is also for biotechnology and medical ethics, but at the moment the system seems to be in the early stages of construction, and relatively little public information has emerged around how the system is to be operationalized.

The fourth layer is around certifications where third parties that are sometimes government affiliated test AI systems for attributes like safety and fairness. This area is quite exciting to look out for as over the past year there has been a lot of discussion around third-party evaluation for AI safety. I think that closely relates to the establishment of national-level AI safety institutions like the AI safety institutes in the United States and United Kingdom.

Over the past six months there has been a lot of similar discussion in China around the need for robust AI safety and security evaluation systems. One of the leading players is the China Academy of Information and Communication Technology, what people usually call CAICT. They have been especially active in this regard. There are also numerous academic labs that are releasing benchmarks, evaluations, and are doing red teaming around large language models, so I think this is a fairly robust and exciting area to look for.

The final layer is local government action. Beyond the central-level ministries that are driving top-down legislation, local provinces and governments have also initiated policies promoting and regulating AI applications around large foundation models. In our latest spring edition of State of AI Safety in China report we found that there are at least six local governments that have instituted these policies since the release of ChatGPT. In particular, for Shanghai and Guangdong Province they also propose creating a national large model testing center.

China has a long tradition of trialing and experimenting policies at the local level before introducing them nationally. We can already see a glimpse of what potential national policies might be like by analyzing government action at the provincial and local levels.

WENDELL WALLACH: That is fascinating, particularly that the trials are going on at the local levels. You did mention “soft law,” which should be a familiar term to listeners of these podcasts, but for those who are not familiar with the term soft law usually refers to standards, laboratory practices and procedures, professional codes of conduct; often insurance policies come into play and other things that do not necessarily or seldom have any legally binding means of enforcement, but they actually do set a lot of the tone for how development of research goes.

You brought that up, and in America soft law can come out of countless institutions and academic, consortiums, and other ways. Is it the same in China? Do many different groups propose soft law, or does soft law come pretty much along with regulations from the primary regulatory bodies?

KWAN YEE NG: The forms of soft law that Brian mentioned—for example, the standards and certifications—sometimes are led by state-affiliated institutions like the CAICT, but the CAICT is also a coordinating body for other industries and in some cases civilian organizations. For example, the CAICT just last week published a round of results from its AI safety benchmark, which evaluated a few large models on various questions for safety. These models included both Chinese models but also one Western model, Llama 2. In this process I believe 17 different organizations were consulted to create this benchmark. Often in these cases, apart from the justice example, in other certifications as well industry is consulted in the process to develop the standards and certifications.

WENDELL WALLACH: So industry is informing or functioning as a driver of what is coming out of this more centralized institution from what I gather. Can you share with us some of the other drivers that get governments to act or direct attention to the governments as to the need for action on the development of AI?

KWAN YEE NG: One that has already been mentioned is social stability and content moderation, which is a key driver of the regulations we have already talked about, but a second here is AI as an engine of economic growth. As China’s economy is undergoing structural changes and does face challenges in the form of slowing growth and demographic changes, the government sees AI as a key component to boost productivity and maintain economic competitiveness. This focus on AI as an engine or growth is especially evident in recent policy statements and government work reports. For example, in this year’s Two Sessions, China’s premier called AI “an important engine for the new productive forces” and the government work report this year also talked a lot about a new AI Plus policy, which seems to place greater emphasis on AI applications in industry.

I want to talk about here a third factor that I believe is often overlooked, genuine concern about AI safety and this is closely tied to very practical concerns here about national security and the safety of the Chinese people. The Chinese government recognizes that AI systems have the potential to be misused or behave in unintended ways, and this has been evident even at the highest levels of Chinese leadership. For example, last April the politburo, the top 25 or so highest-ranking officials in China, met and the official meeting readout stated that, “China must attach importance to the development of artificial general intelligence, but it must also at the same time prioritize the prevention of risks associated with AGI.”

To wrap up, while China is undoubtedly committed to advancing AI technology the leadership is also driven to protect national security and the well-being of citizens by ensuring AI safety, and I think it is important to acknowledge that.

WENDELL WALLACH: Great. That was one thing I noticed when I was in China, that the Chinese government on all levels were much more sensitive to what was coming from the bottom up, to public concerns. We noticed even during COVID-19 when there started to become really vocal protests against the lockdowns, that the lockdowns were removed very quickly, and I think that is something that we in the West do not understand about China, the extent to which the governments are trying to maintain their antennae on what the public feels and what the public is concerned about and tend to respond relatively quickly if they think it is in order.

BRIAN TSE: That is right, Wendell.

Can I just add a data point on this point? When the initial draft for generative AI regulations came out in April of 2023 it proposed very restrictive measures on the outputs of these chatbot models. One clause in the draft required developers to ensure that outputs are “accurate and truthful.” Basically they cannot contain any false information at all. Chinese industry felt that this would be technically unfeasible given the persistent issues of hallucination as we know in large language models.

After several months of public industry consultation, those very restrictive measures were relaxed and rolled back in the final version that was released in August of 2023. I think this illustrates that Chinese national policy does respond to concerns and feedback from the public and industry.

WENDELL WALLACH: That is very interesting. Government constraint on corporations from what I understand is much more robust in China than comparably in the European Union or North America. This seems to suggest that there is concern as there has been in North America and Europe that robust regulations will stunt innovation and growth. Is this just one example of that or is this the primary example of that?

BRIAN TSE: I think China is trying to have the best of both worlds when they look at AI governance in the European Union and the United States. When they look at the European Union, the approach seems too restrictive, and the European Union does not have a thriving domestic innovation ecosystem as compared to the United States and China. When they look at the United States, it does not yet have binding regulations and the voices and incentive from the industry tend to be quite dominant. On balance, I think the mainstream AI governance view in China is that robust governance and policy of AI is needed in order to promote responsible industry innovation but also to ensure public safety at the same time.

In addition China is also keenly aware of the needs of developing countries and believes that responsible AI innovation can help the world meet some of the Sustainable Development Goals.

KWAN YEE NG: Can I just add a point to this as well, Wendell? I think there is a lot of room to debate on the degrees of this, but I do think we need government regulation to restrain companies to some extent. As we saw from the OpenAI board incident last year, we cannot always rely on private companies to responsibly self-govern. In recent years we have seen intense competition between AI developers to quickly build and deploy their AI systems. This competition I think has raised valid concerns about a potential race to the bottom scenarios where people or actors compete to develop AI systems as soon as possible while cutting corners on safety. It can be challenging in such situations for AI developers to unilaterally by themselves commit to stringent safety standards because this could put them at a competitive disadvantage. This is where it could be helpful for governments to step in and coordinate commitments.

Also, regulation does not have to be at odds with innovation and progress. In fact, regulation can empower safe development. For example, take Anthropic’s Responsible Scaling Policy, where they recognize that the more powerful models can bring greater beneficial applications, so their policy is not to prohibit development but to enable the development and use of models with the appropriate precautions in place.

WENDELL WALLACH: As I understand it, the governmental policy is a carrot-and-stick approach, that the carrot is a tremendous amount of funding going into AI research, and the stick of course is that you have a bit more robust regulation of particularly the corporations than we are seeing elsewhere in the world. Is that your perception of what is going on?

BRIAN TSE: In terms of the industry, if we look at the large language models or generative AI landscape there are now more than 200 models as of early 2024, and there are at least three categories of developers with varying incentives and relationships with the government.

First there are the large technology companies like Baidu or Alibaba. They developed and invested in AI to improve their current products and services such as search engines for Baidu, and this is quite similar to Google and Microsoft. Some of them have also decided to explore the overseas market, for example, using these large models in e-commerce services in Southeast Asia by Alibaba.

Second, there are also venture-backed startups like Triplo AI that developed ChatGRM or Moonshot AI that developed Kimi. By the way, both of these products are part-time startups by professors related with Tsinghua University, and they have a tech vision of pushing the boundaries of developing artificial general intelligence and would often aspire to be the OpenAI of China.

Finally, there are also government-backed lab like the Beijing Academy of AI or Shanghai AI Lab. They do not have the short-term commercial incentive for building products or getting venture capital funding, and they often work very closely with academia in China. I think this is also part of the reason why Shanghai AI Lab has emerged as a key player for frontier AI safety research, and this type of player does not really exist in many of the other major AI jurisdictions.

WENDELL WALLACH: What is your subjective sense of the robustness of the research going on in AI safety and AI security? I know many of us in America think that for all the talk about AI safety and security very little funding is going into research compared to the expansion of the frontier models and that not only is very little money going into the research, but the research is dominated by the corporate elite, which are going to get the benefits from that research, so they may not actually want robust regulations that get in the way of their doing what is in their economic interest.

I am just trying to get a feel from the two of you. Subjectively do you feel that the research going on in these fields is adequately robust, or would you like to see much more funding going into the safety and security side of AI development?

BRIAN TSE: I certainly think that the AI safety ecosystem could be more robust, both within China and globally. For example, we should have much more funding for AI safety relative to investment in capabilities. I think globally AI safety accounts for less than 5 percent of the total R&D going into AI, and over the last six months numerous consensus statements from top scientists have suggested that this level should be increased to 30 percent of the total AI R&D.

I also think that this cannot be dominated by a few tech companies that are driven by commercial incentives. I also think we need to have a robust international ecosystem of third-party auditors, international organizations, and nonprofits that help ensure that the risks of AI are mitigated and also that the benefits of AI can be equally distributed around the world. This is an ambitious vision, but I think we should try.

WENDELL WALLACH: Let’s come back to that vision a little bit further on, but how about you, Kwan Yee? What is your sense of the robustness of the development? Do you agree with Brian that you would like to see perhaps as much as 30 percent of investment going toward AI safety and security, or do you have a somewhat different sense of what is going on?

KWAN YEE NG: I definitely do want to see increased investment in AI safety and AI governance resources. I do not have a strong opinion, to be honest, about the exact percentage. I think this is also a place where we could get into debates and splitting hairs on the exact percentages that companies and public funders should dedicate, but I do think that more investment is certainly needed for a more robust ecosystem.

With regard to the robustness of the AI safety research going on in China, like I mentioned earlier I do think technical safety research has blossomed in China over the past year. There are now over a dozen research groups in the country working on various aspects of safety including alignment, robustness assurance, and evaluations.

One data point, every two weeks we publish a newsletter on the state of AI safety in China, and very recently we were looking at the latest papers coming out from Chinese research groups on technical AI safety, and we had over 30 publications. It was a ridiculous amount, and we could not possibly cover that in our newsletter, but that is the sheer volume that is coming out. I think the quality of that research is also rapidly improving.

For example, there was a paper mostly championed by researchers at Microsoft Research Asia on mitigating the risks of AI misuse in scientific fields, and this is already an early, emerging, and burgeoning area of discussion around the world, and to have such research coming out of China at this early stage is I think incredibly promising.

WENDELL WALLACH: That is great. You are seeing a significant ramping up of attention to these concerns in AI. Is that what you have also seen since the first report? Has anything else come up that you want to underscore between the first report that you are going to get into this update beyond just the expansion of attention and projects working in this area?

KWAN YEE NG: I think another area of update is just the robustness of discussions among Chinese experts. In our first report in October we outlined the historical evolution on Chinese expert discourse on AI safety starting like around 2016, but the developments had fairly little discussion going on, with even some Chinese media deriding discussions about AI threats as AI threat theory, but over the past year this has become a much more vociferous field of discussion.

For example, over the past month, Xue Lan, who is the head of Tsinghua University’s Institute for AI International Governance, suggested at a roundtable to focus on bottom lines on AI risks and international discussions given “obvious interest” in things like preventing loss of control over AI among countries.

Similarly academician Gao Wen, who is the director of the National Peng Cheng Lab and previously presented on AI to President Xi Jinping, has written and spoken multiple times on AGI risks and potential risk-prevention strategies. He has also talked about these topics more recently as well. I think the Chinese AI expert discourse has continued and flourished over the past few months.

I cannot talk about all the senior Chinese experts who have expressed concerns about frontier AI in this podcast obviously but there is a website called Chinese Perspectives on AI Safety that contains profiles of these Chinese experts and their statements on AI risks if people want to look further as well.

WENDELL WALLACH: Those people you have talked about have been colleagues of ours, and I have known particularly Xue Lan for many, many years, and they have been friends of the AI & Equality Initiative, so it is kind of thrilling to see this go forward, also Yi Zhang, who works very closely with that group and is a member of the UN AI consultative body, is one of the advisors of this project. That is just great because I think that gives you some hope that we are moving at least in China in the right trajectory.

Maybe I can go to the negative side with you a little bit. This does not have to be China-specific for you in your response, but maybe you will start this one, Brian. What keeps you up at night?

BRIAN TSE: Yes. That is a great question, Wendell. I think what keeps me up at night is the accelerating progress of AI and the inability of global institutions to keep up with this pace. If we look at the pace of AI progress before the deep learning era, AI computational power was doubling approximately every two years in line with Moore’s Law, and since 2012 this growth has dramatically accelerated to a doubling every three to four months, exceeding Moore’s Law by a lot.

I also think the recent pace of progress has caught most of the world by surprise. Before GPT-3 was released, expert forecasters expected that it would take another 80 years to develop human-level generative AI systems, and after GPT-3 in 2020 the average forecast was shortened to 50 years, and then after ChatGPT in 2023 it was shortened further, to less than ten years. I think our human psychology and indecisions are inclined toward linear thinking, and exponential growth from emerging technologies like AI does not really align with this linear perspective.

Of course, this does not apply just to AI. We see that many governments and societies around the world struggled to predict and manage Covid-19 as the infection rates increased exponentially. I think the challenge is, how do we create a robust global ecosystem in AI safety and governance and also international cooperation that keeps pace with this exponential progress from AI?

WENDELL WALLACH: How about you, Kwan Yee? What keeps you up at night?

KWAN YEE NG: I think making sure the global majority is included in AI safety and governance conversations is hugely important and neglected at the moment. I want to put an asterisk here; I am not an expert in this area, but I think that regulation cannot be pushed forward by policy elites and technology companies alone and we must consider the interests of ordinary citizens and developing countries as well. We should ensure equal rights, opportunities, and rules for all countries in AI development and governance. We know that the development of frontier AI systems will be transformational for all of humanity, so giving everyone a say in this process is morally critical.

I think there is a practical side to this as well. The cost of training AI is going down because of lower compute costs and progress in algorithms, and it is also pretty tough to prevent model weights from being stolen or leaked at the moment, so while at the moment we might we see powerful capabilities concentrated in the hands of wealthy countries, this might not be the case in the longer term, and we might also see powerful capabilities proliferate to actors in the global majority and even nonstate actors.

I think another point here about the practical importance is that major AI nations might also link their agreement on governance frameworks to ensuring access for their allies. For example, the Chinese government actually promotes wide access to AI technologies for developing countries through international fora like the United Nations and through policy documents, so I think we should really consider how we can share the benefits of AI widely and ensure equal participation in governance. Given the role of China as a major member of the developing world but also a leading AI power, it can have a particularly strong role to play on this issue.

One early idea is to incorporate underrepresented languages in large models, like the Chinese Peng Cheng lab is currently doing, creating a data set on languages across the Belt and Road Initiative. Another idea is to foster AI safety and R&D talent especially from underrepresented groups.

WENDELL WALLACH: I understand that is mainly focused on China or the countries that it is in alliance with on the Belt and Road Initiative. Is that correct?

KWAN YEE NG: I believe that is the case for the Peng Cheng Labs project, but there are other examples of projects in this direction. For example, there is a project called Southeast Asian Languages in One Network Data (SEALD) that is doing the same for Southeast Asian languages, and there is a project by the Māori Data Sovereignty Network that is trying to do something similar as well.

WENDELL WALLACH: Both of you have been recurring quite often to international governance as one of the elements that we need moving forward, and as you know the AI & Equality Initiative and Anja’s and my work has been particularly focused on that element, myself for many years now. There is always this difficulty in it in terms of how China is perceived in regard to international regulations and whether it would come to the table or perhaps more importantly whether some of the Western countries, particularly the United States, necessarily want it at the table or want it at the table for what reasons.

Perhaps you could tell us a few things about what you think your government is actually doing to promote international cooperation and what additional steps you think need to take place for some degree of international cooperation to come about, not just in standards but in ongoing meetings and putting in some robust controls or limits on some of the more dangerous applications. Do you want to start with that, Brian?

BRIAN TSE: First of all, I want to say that the Chinese AI ecosystem probably has more desire and willingness to work with the international community on AI safety than many expected. I think that is becoming more evident since we published the first report in October. We can just go by the feel of these developments. China signed onto the Bletchley Declaration at the first Global AI Safety Summit as I mentioned, then it also supported launching the International Scientific Report on Advanced AI Safety, chaired by Yoshua Bengio. This is equivalent to the Intergovernmental Panel on Climate Change.

When the United States initiated the first UN General Assembly Resolution on AI, China along with 120 other countries cosponsored the idea. Then when President Xi met with President Biden in San Francisco, they also agreed to launch intergovernmental talks on AI including AI safety. There are also a number of AI safety consensus statements coming out from top Chinese and Western academics, including three Turing Award winners.

I think these developments over the last six months show that international cooperation, including between China and the United States, on AI safety is possible. This is a key message that I would like to communicate with the audience.

Perhaps Kwan Yee could share a bit more about some of the suggestions and recommendations we have.

KWAN YEE NG: I am happy to do that, but I want to add a few more notes on what China has been doing with regard to international AI governance first.

Wendell, you mentioned participation with regard to ethical principles and standard setting, and it has certainly been the case that China has been active in these areas since 2016, but I think last year really marked a phase shift in China’s efforts to step up international AI governance.

For example, last April China elevated international cooperation on AI governance within one of China’s flagship foreign policy programs, the Global Security Initiative, and then last October President Xi Jinping announced the Global AI Governance Initiative, which sets out China’s core positions and values on AI development and for international cooperation on AI.

Regardless, much of global AI governance remains fractured across geopolitical fault lines, and in response China has been advocating for what it calls “true” multilateralism, which primarily emphasizes the central role of the United Nations and greater representation of developing countries.

There could be a concern here that definitely parallel AI governance efforts need not be conflicting with each other. Exclusion of China can lead to parallel AI governance efforts and this might hinder the necessary coordination needed for joint efforts on AI safety. We have seen this over the past two years as Brazil, Russia, India, China, and South Africa (BRICS) has made multiple announcements relating to AI, including creating an AI study group. The Global AI Governance Initiative was also notably unveiled at the Belt and Road Forum.

At the same time, I would say that China’s positions on maintaining human control of AI systems and preventing their misuse by terrorists could be areas for international cooperation. For example, ensuring human control has been a consistent element in China’s domestic AI policy documents and also in its expressions on the international stage.

For example, in 2021 their Ministry of Foreign Affairs submitted a position paper to the UN Convention on Certain Conventional Weapons where China called for relevant weapons systems to remain under human control. Since then China has expanded this position to include nonmilitary systems as well with this Local AI Governance Initiative, calling for all AI research entities to ensure human control over AI.

The initiative also explicitly calls for international collaboration on terrorist’s misuse of AI, which most concerns, say, expressed by UN Secretary-General Guterres and UN representatives from countries including Japan, Mozambique, and Ghana when AI risks were discussed for the first time at the UN Security Council last summer 2023.

WENDELL WALLACH: This is great. I think it is very important that a Western audience, particularly the audience we have for these podcasts, which are largely those interested in international affairs, is aware that there are many opportunities for cooperation here, and there have been some steps taken forward.

Of course, detractors are going to point out what has not happened or what has failed. For example, recent meetings on regulating or in effect outlawing lethal autonomous weapons to carry nuclear weapons failed. I do not think we need to get into the technicalities over the disagreements, but you would have thought that would have been one of the most obvious agreements that countries could have come to.

Of course, that is a different area than in your bailiwick. I know that you are not focused on the military applications so much, but we are probably going to have to get over some hump as far as the regulation of lethal autonomous weapons before we can even focus more on some of the joint research.

We are coming to the end of our time together, but I think you have been wonderfully informative. I wonder whether we can just finish up with you sharing some thoughts or other points you would like to bring up with our audience here.

Kwan Yee, why don’t you start out with that?

KWAN YEE NG: Thank you for that, Wendell. I just want to close with some concrete recommendations on our side for international cooperation on AI safety and governance. We could maybe also discuss some of these ideas. I would be curious to hear your thoughts on this as well, Wendell.

One idea we have here is focusing on dual risks. You mentioned that there have been difficulties getting to terms with agreements on the uses of AI in the military, but we also do recognize that cutting-edge AI systems are approaching human-level performance in a range of safety-critical domains, and it is incredibly important for the international community to agree on a set of, say, red lines, bottom lines, on AI development.

There has been some difficulty getting to agreement on the international stage, but in a recent international dialogue in Beijing top scientific experts and policy experts as well agreed on five red lines and released a consensus statement, including that AI should never be used to help with designing biological weapons, launching cyberattacks, or autonomously self-replicating.

Upon agreeing on these red lines, we could consider having an international network of partners for AI safety evaluations and testing, and this could help with sharing information about emergent risks and eventually develop into shared standards. One example is that just recently the United States and the United Kingdom AI Safety Institutes signed a memorandum of understanding to develop a shared approach to evaluations and commit to jointly testing a publicly accessible model. This kind of partnership could be expanded internationally as well. Flowing on from that, the results of AI safety evaluations testing could also feed into decisions on whether and how to scale up large models responsibly.

I mentioned earlier Anthropic’s Responsible Scaling Policy. Bio-safety labs with higher-stakes research and development could have more stringent safety protocols and practices with an analogy to, say, biosecurity level 3 for labs, and this could help with balancing both the needs for promoting innovation and managing risks, and we could hope to see international standards along this vein as well.

WENDELL WALLACH: I think this is a great idea. I am actually sad to see it start up from the bottom up, meaning the United States and the United Kingdom. I think these are the kinds of initiatives that I am hopeful will come out of the United Nations. I am a little skeptical whether they will come out of the United Nations, but I am hopeful. This list of recommendations for international cooperation around exactly initiatives like the one you have outlined is where we go, and we get there quite quickly.

Brian, what about you? What would you like to share with our audience before we close?

BRIAN TSE: I think I have felt a little bit more hopeful about the trajectory of AI safety and governance over the past year. Right now world leaders and major institutions around the world are seriously thinking about the opportunities and risks posed by advances in AI.

There are certainly people who are skeptical of greater cooperation between major AI powers on AI safety, but in response I will say two things: One is that the United States and the Soviet Union managed to tackle global challenges. When they realized the potential catastrophic risks from global nuclear war, they came together to agree on a set of disarmament and nonproliferation treaties. Even at the height of the Cold War they worked together to eradicate smallpox, which was an incredible achievement.

Second, I think the United States and China in the 21st century can do much better than the Cold War. There is enduring interdependence between the world’s two largest economies and there is more cooperation to publish AI papers between the United States and China than any other pairs in the world, more than even between the United States and the United Kingdom, and both countries’ governments and people really desire and benefit from global stability and not chaos. If we look at the trajectory of international dialogue on AI safety over the last six to twelve months, I think there is reason for hope.

WENDELL WALLACH: Wonderful. There is nothing like ending a podcast on hope, and I hope that this is the message that we have been able to convey throughout this podcast, that what is going on both in China and with China on the international stage when it comes to AI safety and security is much more robust than perhaps many of our listeners might have been aware of before your most informative discussion today.

I want to thank you ever so much and wish you all the best in your future work.

KWAN YEE NG: Thank you so much, Wendell. It was a pleasure talking with you today.

BRIAN TSE: Thank you so much, Wendell.

WENDELL WALLACH: Thank you ever so much, Brian and Kwan Yee, for sharing your time, insights, and expertise with us. This has indeed been another rich and thought-provoking discussion.

Thank you to our listeners for tuning in and a special thanks to the team at the Carnegie Council for hosting and producing this podcast. For the latest content on ethics and international affairs, be sure to follow us on social media at @carnegiecouncil. My name is Wendell Wallach, and I hope we earned the privilege of your time. Thanks again.