Basically - "any model trained with ~28M H100 hours, which is around $50M USD or - any cluster with 10^20 FLOPs, which is around 50,000 H100s, which only two companies currently have " - hat-tip to nearcyan on Twitter for this calculation.
Specific language below.
" (i) any model that was trained using a quantity of computing power greater than 1026 integer or floating-point operations, or using primarily biological sequence data and using a quantity of computing power greater than 1023 integer or floating-point operations; and
(ii) any computing cluster that has a set of machines physically co-located in a single datacenter, transitively connected by data center networking of over 100 Gbit/s, and having a theoretical maximum computing capacity of 1020 integer or floating-point operations per second for training AI."
So if I make a 10 Trillion param mixture of experts model out of fine-tuned variations of the same 300b model I am safe right?
Or how about I train a 4 Trillion param model on a new architecture that can utilize distributed compute? If contribution to GPU pools for training is encrypted and decentralized then good luck.
Fuck OpenAI. We will take this fight to the streets and win.