minus-squareThe_Hardcard@alien.topBtoLocalLLaMA@poweruser.forum•NVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLMlinkfedilinkEnglisharrow-up1·1 year agoThat’s the speed of 4.8 TB/s memory bandwidth. 5.3 TB/s coming in a little over three weeks. linkfedilink
That’s the speed of 4.8 TB/s memory bandwidth. 5.3 TB/s coming in a little over three weeks.