a_slay_nub@alien.topBtoLocalLLaMA@poweruser.forum•is there any other tools like vLLM or TensorRT that can be used to speed up LLM inference?English
1·
10 months agoLmdeploy is another one
Lmdeploy is another one
Care to elaborate on what the actual reality of this EO is?
Bit disappointed by the coding performance but it is a general use case model. It’s insane how good gpt 3.5 is for how fast it is.