So I’m looking into Threadripper pro systems, which can offer a pretty good memory bandwidth as they are 8 channel, and can have a huge amount of RAM. (I can put a 3090 or two in there too.)

I’m wondering how much the core count is going to affect performance. For example, the 5955WX has 16 cores while the 5995WX has 64 cores. They can both use the same memory though. There’s little point spending extra if the limiting factor will be somewhere else.

  • jeffwadsworth@alien.topB
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    I use a Ryzen 12 core and can use llama.cpp with the 70b 8bit fine. Do not bother with hyper-threads, though.