ChinAI #335: Rereading Stanford's 2025 AI Index

Greetings from a world where…

the Hawkeyes have lost 12 straight games to ranked opponents

…As always, the searchable archive of all past issues is here. Please please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay support access for all AND compensation for awesome ChinAI contributors).

Notes on China’s AI ecosystem from Stanford’s AI Index (2025)

Let’s switch it up this week. In place of a translation, I want to take some notes on Stanford’s AI Index Report, which was published earlier this year in April. Released annually since 2017, its evolved into an authoritative resource on AI trends. I’ve used it in my own research. One of my favorite stats to cite is that the U.S. leads the world in the number of “hybrid” academic-corporate AI publications (at least one co-author from industry and one co-author from academia).¹ From this year’s report, I learned a lot from the section on notable models, and where they were developed: “In 2024, the United States led with 40 notable AI models, followed by China with 15 and France with three” (p. 46).

After slowly digesting this 457-page report, my main critique is that — at least, regarding the China-relevant portions — the effort suffers from too much data and too little context. Let’s take a closer read of three sections:

First, according to the index, China leads the world in AI-driven clinical trials (number of clinical trials that mention AI, p. 317), but this is a very unhelpful indicator.

• To start, the report doesn’t specify where they get their data on AI-related clinical trials by region, but it’s very likely they used the ClinicalTrials.gov (when I searched for mentions of “artificial intelligence” and sorted by region, the numbers were consistent with those reported by the index).

• The problem is that there are significant gaps in geographical coverage for ClinicalTrials.gov data. For Chinese clinical trials in 2023, per one study, only 1,817 out of 7,658 studies were registered on ClinicalTrials.gov.

• Furthermore, clinical trials measures do not reflect effective use of AI in healthcare. One blog about transparency in medicine points out: “A larger number of trials does not equal more gain in useful scientific knowledge. For example, the UK excelled at generating robust and useful evidence on potential Covid treatments precisely because it focused its efforts on a small number of large, well-designed trials.”

• Equipped with better background knowledge about China’s slow integration of medical AI (see, for example, ChinAI #162), we should be skeptical of giving too much weight to this “China leads the world in AI-driven clinical trials” indicator. In that ChinAI translation, Leiphone finds that China’s tech giants have struggled to get approvals for their medical AI products. As of 2021, only 15 AI products had successfully passed the certification process from China’s National Medical Product Administration. Now, with this context, we can leverage the AI index’s data to present a very different picture: by comparison, the U.S. FDA approved 129 AI medical devices in 2021 (p. 309).

Second, the index repeats the flawed narrative of an AI optimism gap between Asian and Western countries: “A large majority of people believe AI-powered products and services offer more benefits than drawbacks in countries like China (83%), Indonesia (80%), and Thailand (77%), while only a minority share this view in Canada (40%), the United States (39%), and the Netherlands (36%).”

• I’ve covered this in previous issues (ChinAI #331), so I won’t belabor the point here. I do want to reiterate my skepticism about using online samples of Chinese respondents as representative of the overall population. In that post, I argued: “To me, here’s the bottom line. Regarding findings from online surveys…they reflect the views of highly educated adults in China, very likely concentrated in the coastal provinces — not the general Chinese public.”

• As evidence of this AI optimism gap, the index cites Ipsos online surveys, but note what the Ipsos survey emphasizes in its methodology section: “Samples in Brazil, Chile, China… are more urban, more educated, and/or more affluent than the general population. The survey results for these countries should be viewed as reflecting the views of the more “connected” segment of their population.” These are not apples-to-apples comparisons.

Third, the GitHub indicators cited in the index severely understate China’s investment in open-source AI software. The Stanford report’s graph of GitHub AI projects by geographic area (p. 78) shows a decline in open-source AI projects from China-based developers (figure below). Crucially, the report maps AI projects to geographic areas using IP address geolocation.

Source: Stanford AI Index 2025

• Again, context matters! In his amazing paper on open-source software policy, Jeff Gortmaker shows that IP address-based measures cannot capture the activities of many Chinese developers, who face government restrictions on accessing the GitHub platform. Specifically, when analyzing Chinese contributors to open-source web development frameworks, he found: “By 2023, a 70,000- contributor gap between China’s self-reported and IP-based lines likely reflects increased use of virtual private networks (VPNs), which, when used to bypass the GFW (Great Firewall), register as foreign IP addresses.”

• Chip Huyen provides a more accurate assessment of global contributions to open-source AI software. She collected 900 of the most popular open-source AI tools. Among the top 20 accounts on GitHub (in terms of stars), 6 originated in China. That’s a stark contrast to the index’s metric of just 2% of GitHub AI projects coming from China.

Look, I don’t want to be overly harsh here. There is a lot of great material in this 450-pg. report. These large-scale efforts are a tough balancing act, and one could argue that the objective is to start conversations like this one.

Still, the danger is that most people reading and aggregating these metrics will accept them without a critical eye. I wonder if a 200-pg. report with half as much data and two times as much expertise/context/point-of-view would be more effective.

ChinAI Links (Four to Forward)

Must-read: The State of Chinese AI Apps 2025

Working with Unique Research (非凡产研), a Chinese data provider, Tech Buzz China has released a report focused on AI products. Rita Luan and Wei Wu analyzed 100 of the largest AI apps from around the world, finding: “23 came from Chinese developers—and 19 of those generate most of their revenue overseas. Only four focus mainly on domestic users.” Rui Ma, founder of Tech Buzz China, wrote up some other top-line findings here.

Should-read: Why Trump’s cuts to scientific research are a big win for China

For The Washington Post, Katrina Northrop and Rudy Lu highlight the impact of the Trump administration’s cuts to science funding and revocation of international student visas. Below are a few choice quotes from the article:

When Jonathan Kagan, an immunologist at Harvard Medical School, attended a conference in Suzhou in May, he said Chinese scientists kept telling him the same thing: “We hope Trump is president for life, because it is the best thing to happen to Chinese science.”

Suspensions of research grants have caused top researchers to reconsider the U.S.’s commitment to science leadership:

Terence Tao, a celebrated UCLA mathematician sometimes called the “Mozart of Math,” recently had $26 million of U.S. National Science Foundation grants suspended. Though the grants were later reinstated, Tao, who was born in Australia, said universities in China had since been in touch, trying to lure him there. He never previously considered leaving the U.S., where he has lived for more than 30 years. But the attacks on higher education are forcing him to rethink his assumptions about the U.S.’s ability to sustain science leadership. “I’m not certain about anything anymore,” he said.

Up-and-coming researcher Alex Liu accepted a post at the Shenzhen Bay Laboratory:

Take Alex Liu, 38, a Chinese-born mosquito researcher who earned his PhD at Auburn University in Alabama. “I really didn’t find that many opportunities in America,” he said in his office, where he keeps an Auburn football on his desk.

Should-read: A Project is Not a Bundle of Tasks

Steve Newman, Google Docs founder, pushes back against assumptions for short timelines of Artificial General Intelligence:

Impactful software engineering projects are usually large, complicated beasts – if not at first (Writely), then as they grow and mature (Google Docs). Certainly there are complex projects taking place inside the big AI labs. Analysis based on the automation of specific tasks misses the forest for the trees. Software engineering will not be fully automated until AIs can handle projects of the 100-person-year scale (or more). How long until that happens?

It’s impossible to predict with any certainty. But we can take a stab. The graph I presented at the top of this post shows current AI models achieving a 50% success rate at tasks that take about two person-hours. The paper where that graph originated finds that the manageable task size doubles every 7 months. It would take about 16 doublings to get from two person-hours to 100 person-years³; at 7 months per doubling, that puts us around the beginning of 2035. Advancing from a 50% success rate to full competence might take a few more years, putting automated software engineering in the late 2030s. Adjusting for real-world conditions might push the schedule out further.

Should-read: The Cyber Offense-Defense Balance for Trailing-Edge Organizations

Alongside Twm Stone, GovAI fellow Ben Murphy co-authored an excellent working paper on how AI will affect the cyber offense-defense balance. Their key insight is that AI’s boost to the defenders will help well-resourced companies but might not translate to “trailing-edge organizations”, which “rely heavily on legacy software, poorly staff security roles, and struggle to implement best practices like rapid deployment of security patches.”

Thank you for reading and engaging.

*These are Jeff Ding’s (sometimes) weekly translations of Chinese-language musings on AI and related topics. Jeff is an Assistant Professor of Political Science at George Washington University.

Check out the archive of all past issues here & please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay for a subscription will support access for all).

Also! Listen to narrations of the ChinAI Newsletter in podcast format here.

Any suggestions or feedback? Let me know at chinainewsletter@gmail.com or on Twitter at @jjding99

1. Zhang, Daniel, Saurabh Mishra, Erik Brynjolfsson, et al. The AI Index 2021 Annual Report. Stanford Human-Centered Artificial Intelligence Institute, 2021, p. 23.