rihard7854@alien.topB to LocalLLaMA@poweruser.forumEnglish · 10 months agoNVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLMgithub.comexternal-linkmessage-square23fedilinkarrow-up11arrow-down10
arrow-up11arrow-down1external-linkNVidia H200 achieves nearly 12,000 tokens/sec on Llama2-13B with TensorRT-LLMgithub.comrihard7854@alien.topB to LocalLLaMA@poweruser.forumEnglish · 10 months agomessage-square23fedilink
minus-squareThe_Hardcard@alien.topBlinkfedilinkEnglisharrow-up1·10 months agoThat’s the speed of 4.8 TB/s memory bandwidth. 5.3 TB/s coming in a little over three weeks.
That’s the speed of 4.8 TB/s memory bandwidth. 5.3 TB/s coming in a little over three weeks.