Thanks, this is interesting. This all said, it still looks like B is a much more important factor than quantisation down to Q3, meaning a 20B Q3 is going to write better than a 13B fp16. And such it seemed to me personally but I haven’t done any rigorous testing.
Yeah but it’s 50% off plus the cost of training GPT-4 which is not gonna be 13B.
I just use Linux. Stop the X server or don’t install it altogether. (If you have it, just Ctrl+Alt+F1, log in, sudo systemctl stop lightdm.) Enjoy the entirety of your VRAM minus 1 MB available for you, so from there you run oobabooga or an API server or whatever and connect to it from your laptop.
As an added benefit, I can leave the GPU and all the noisy fans and their heat somewhere else in my home. And you don’t need to connect a display to the Linux box either, just setup openssh-server, keys and work remotely.
Try to use it for coding, it’ll be as good as offshoring.