I’m only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I’ll admit I barely know what I’m doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5…
I’m only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I’ll admit I barely know what I’m doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5…
The attention layers get replaced with flash attention 2, there’s kv caching as well so you get way better batch1 & batchN results with continuous batching for every request