I’m only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I’ll admit I barely know what I’m doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5…
I’m only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I’ll admit I barely know what I’m doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5…
What’s the latest t/s on a 4bit model with TGI? is there a difference compared with HF transformer loader?
The attention layers get replaced with flash attention 2, there’s kv caching as well so you get way better batch1 & batchN results with continuous batching for every request