Hey everyone,

Looking to get into some ML.

Can a GTX 4070ti with 12gb VRAM alongside 32gb ram run 13b comfortably?

I seem to read conflicting opinions on this.

Thank you!

  • Arcturus17@alien.topBEnglish
    1·
    1 year ago

    I’ve got a 3060 Ti 8GB and 16 GB RAM and I can run 13B GGUFs with 30 layers offloaded to GPU and get 8-12 t/s no problem. I cannot run a 20B GGUF at all though.

    If you want to run GPU inference only though, you’ll need 16+ (more likely 20+) GB of VRAM.