Your comparison proves his point! 13b will fit snuggly in your 6900 this is a head on comparison of the cards!

Welcome to the rabbit hole 😁. On a serious note, going for the newer generations pays dividends, in my opinion.

Oh god 🤦 But seriously we need a wiki with a leader board with votes😁

I’ve noticed this extensively when running locally on my 8gb rx580. And the issue is pretty bad… I’ve run exactly the models you stated.

But when I run on (big) cloud GPU on vast.ai (eg on rtx 3090 or A6000) the problem vanishes…

vast.ai is pretty cheap ($10 deposit)you can experiment on there and see.

I’ve used gpt4 to help write articles for my blog. So I just picked some of the good articles that it wrote (eg Lutris game manager) and prompt the testing one to write (800 words) and then compare. This has worked really well for me. Vicuna 33b was the best alternative I’ve found in my small tests in creative writing… Although I cant locally host it on my PC :/

an 8b model? surely releasing larger ones is good for their own game :/