Title essentially. I’m currently running RTX 3060 with 12GB of VRAM, 32GB RAM and an i5-9600k. Been running 7B and 13B models effortlessly via KoboldCPP(i tend to offload all 35 layers to GPU for 7Bs, and 40 for 13Bs) + SillyTavern for role playing purposes, but slowdown becomes noticeable at higher context with 13Bs(Not too bad so i deal with it). Is this setup capable of running bigger models like 20B or potentially even 34B?
Don’t quite know about 34B and beyond as i never tested it on myself, but you can more or less easily run a 20B model with these specs. I also have a 3060 with 32gigs of RAM and i get around 3Tokens/ second while generating using u-amethyst20B(I believe this is the best, or at least the most popular 20B model at the moment) Q4KM after offloading 50 layers to GPU.