ChinAI #255: Panic buying, speculative booms, and whack-a-mole — what lengths will Chinese companies go to get an NVIDIA A100 chip?

Greetings from a world where…

policy is policy and businesses is business [政策是政策，生意是生意]

…***We’ve hit a bit of a lull in paid subscriptions lately, so please consider subscribing here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay support access for all AND compensation for awesome ChinAI contributors). As always, the searchable archive of all past issues is here.

Feature Translation: After the Aug 2022/Oct 2023 controls, how are Chinese companies getting high-end AI chips

Context: It’s October 20, 2023, and the hottest betting action in Hong Kong (just 60km from the world’s top gambling hub in Macau) is on who will get the 300 newly arrived 8-card A100 servers. Total value: 800 million RMB. This is three days after the U.S. extended the scope of export controls on chips to include close substitutes for the NVIDIA A100 (which had been banned a year earlier). Procurement teams from e-commerce giant Pinduoduo and software leader Kingsoft arrived overnight to compete over the servers. “Where did the goods come from? No one knows, it’s said to be a very clever company, and it was obtained through NVIDIA’s unauthorized channels; what’s the specific price? There is no precise figure.”

In a longform piece (link to original Chinese), Meiri Renwu [每日人物] reporters trace the lengths Chinese companies will go to get high-end AI chips. Two of the key interviewees are Zipeng Wang and Wenbo Sun (pseudonyms), who work at server manufacturers, which means they’ve seen all of this up close — panic buying, speculative booms, and whack-a-mole. Under the Renwu platform, Meiri Renwu was the source of a previous ChinAI feature article, which asked six different large language models to summarize China’s 2023 (ChinAI #250).

Key Passages: Seemingly unlimited demand + limited supply = speculation.

• Who’s going to get those 300 units of 8x NVIDIA A100s? Pinduoduo’s e-commerce business doesn’t do much with large models, and there hasn’t been much news about its own research on large models. “However, in the AI era, it is better to bet first than to miss out.” As for Kingsoft, industry circles think that it can’t afford such a large order. “Ambiguous transactions, as well as rumors, speculations, and bets, have all become a footnote to the madness of large models and A100s.”

NVIDIA CEO Jensen Huang attends annual company meetings in China

• Xu Qing, the Meiri Renwu reporter, captures this: “Demand seems to be unlimited, but NVIDIA's foundry, TSMC, has limited production capacity. In addition to competition among major manufacturers, the U.S. chip blockade against China has also made it more difficult for domestic tech giants to purchase high-end chips like the A100. All parties are panic buying the A100’s ‘successors’, including the A800, H100, and H800s, and the prices have skyrocketed. At the craziest time, in just one week, the price of a server composed of 8 A800s can increase from 2.3 million to 3.3 million. For a while, Wenbo Sun experienced misgivings: Why did it seem like he was speculating on Bitcoin?”

The rise of large language models, combined with the U.S.’s Oct. 2022 controls, set off a money-burning competition over chips

• From the article: “The A100, which was once not in short supply, suddenly became a hot commodity. And just in October 2022, the U.S. Department of Commerce imposed additional export restrictions on high-performance chips, prohibiting U.S. companies from selling high-end chips to Chinese entities, and the A100 was among them. Policy is policy, and business is business. NVIDIA launched the A800, a special version of the A100 for China. Wang made an analogy. Compared with the A100, the A800 ‘is like making the oil pipe of a car a little thinner, but the engines and tires are still the original.’”

• Notably, Bytedance emerged as the top buyer of chips. “A person familiar with the matter revealed that Bytedance purchased approximately 13,600 A100 chips. Based on the minimum price of a card of 70,000 RMB, these chips alone are worth 952 million RMB.” The article continues, “Based on its internal data, a major tech giant estimates that Bytedance currently has more than 200,000 computing power chips, of which about 100,000 are high-end computing power chips such as the A100/A800/H800s. A small part of these chips are rented out, and the high-computing chips that are not rented out are used for the training and inference of Bytedance’s large models.”

• The large AI labs were the first to recognize the need to rapidly procure NVIDIA chips. “Wang and Sun both received news that Bytedance and Alibaba were the first to place orders with NVIDIA, ‘both placing orders for nearly 10,000 cards.’ The reason why Baidu was the first to launch Ernie Bot (Wenxin yiyan) was because it purchased computing chips in advance.”

During the Gold Rush, it was the shovel sellers that made the most money. In the same way, Chinese large tech companies have started to rent out computing power.

• The article relates, “In Alibaba, the monthly rental price of an 8-card server with computing power is around 150,000-200,000, and the minimum price is 4 units. This means that the cost of a machine can be recovered as long as it is rented out for 8 months.”

• A price war has commenced between Bytedance and Alibaba: “Everyone has heard about how many chips Bytedance has stockpiled. A price war has also begun. Bytedance's Huoshan Cloud sales team will ask customers, ‘How much does Alibaba want from you? We'll give you a 40% discount.’” We see this competition reflected through the choices of key clients. Of the Chinese large model startups, Kai-fu Lee’s 01.AI rents compute from Alibaba, while Zhipu AI uses Bytedance’s cloud.

What happens next?

• Can Chinese companies build chips competitive with NVIDIA’s A100s? Huawei’s Ascend process have tried to fill the gap, but switching from one type of chip to another is not a plug-and-play process: “However, some insiders commented: ‘Huawei's ecosystem is worse than NVIDIA's. Switching to another ecosystem is a huge and complex project. Everyone has been using NVIDIA's cards and ecosystem for too long, so it is difficult to switch.’”

• Others are questioning this overall trajectory. The article quotes one associate professor complaining about how the skyrocketing price of computing power has hindered his own project: a “small model” approach to equipping drones with AI programs to check whether wires remain intact in uninhabited areas in southern China (the programs identify where there are cracks in the wire insulators and then report the location for repair). From the article: “He doesn’t understand the huge number of large models today. ‘There are so many large language models, and then what?’ In his opinion, many large model products are more of a gimmick with no practical uses, and spending so much money, resources, and computing power is almost a waste.”

FULL TRANSLATION: Internet giants, it’s hard to buy A100s even if you have money

ChinAI Links (Four to Forward)

Must-read: Computing Power and the Governance of AI

A group of researchers from OpenAI, GovAI, etc. have published an impressive report on how governing compute can contribute toward ensuring the safety and beneficial uses of AI. I especially liked the section on the use of compute regulations to enforce certain rules (e.g., enforcing “compute caps” via physical limits on chip-to-chip, hardware-based remote enforcement, and preventing risky training runs via multiparty control). The appendix on the compute-uranium analogy was also illuminating.

Should-apply: African-American China Leadership Fellows Program

The American Mandarin Society’s AACLF program is accepting applications for rising African-American professionals interested in China-focused careers. Mentors include Dannielle Andrews, a Senior Foreign Service Officer with the U.S. Department of State who served as the chief of the Economic Section at the American Institute in Taiwan, and Keisha Brown, a historian of modern China at Tennessee State University. This was posted in the American Mandarin Society’s latest newsletter issue, which also featured an interesting interview about innovation and productivity growth.

Two personal plugs:

1. I’m recruiting a 2-year postdoc (job ad link) to work on the global governance of emerging technology. Substantive interest in China’s international engagement with safety and security issues linked to hazardous technologies is especially welcome. Based at George Washington University’s political science department, the postdoc will also have the opportunity to affiliate with George Washington University’s Institute of Security and Conflict Studies (ISCS) and present work at research-in-progress seminars at the ISCS, which has a vibrant community of predoctoral fellows, visiting scholars, and PhDs in residence. Deadline is March 25.

2. Zoom talk at Center for East Asian and Pacific Studies (Illinois): This upcoming Friday I’ll be talking about a working paper titled: “Who is Us?: The Globalization of Innovation and Challenges to Assessing Technological Dependence.” The paper addresses a gap I’ve noticed in the analysis of China’s indigenous innovation drive (an increasingly hot topic): how does China measure success in the first place? And how does globalization complicate that assessment?

Thank you for reading and engaging.

These are Jeff Ding's (sometimes) weekly translations of Chinese-language musings on AI and related topics. Jeff is an Assistant Professor of Political Science at George Washington University.

Check out the archive of all past issues here & please subscribe here to support ChinAI under a Guardian/Wikipedia-style tipping model (everyone gets the same content but those who can pay for a subscription will support access for all).

Also! Listen to narrations of the ChinAI Newsletter in podcast format here.

Any suggestions or feedback? Let me know at chinainewsletter@gmail.com or on Twitter at @jjding99