The dirty data item caught my eye. If BAT labs are redoing pre-training six months in, it suggests the bottleneck in China's model race is shifting toward corpus quality and evaluation sets, a quieter place to compete and a lot harder for outsiders to track. Put next to the Ulanqab token capital piece, you get a picture of a system that can build compute fast and is still sorting out what goes into it. Voting for #4 as the feature translation.
Great show.
#2, #8, #9
The dirty data item caught my eye. If BAT labs are redoing pre-training six months in, it suggests the bottleneck in China's model race is shifting toward corpus quality and evaluation sets, a quieter place to compete and a lot harder for outsiders to track. Put next to the Ulanqab token capital piece, you get a picture of a system that can build compute fast and is still sorting out what goes into it. Voting for #4 as the feature translation.