I saw deepinfra. Their price is $0.7-0.95/million tokens for llama2 70b.
How is that possible? Even the quantized 70b models is 35 gbs.
How do you minimize costs of the GPUs, bandwidths, and on the software side?
On their article:
“I think other than the WhatsApp team, they are maybe first or second in the world to having the capability to build efficient infrastructure to serve hundreds of millions of people.”
But technology is not magic, can someone shine some light on running cost-effective AI clusters? I was looking at vast.ai etc but renting GPUs directly in that way would be much more expensive.
Batched inference.