I’m currently using 1650 4GB, AMD 5600, 32GB RAM.

I got some spare cash to throw to learn more about local llm.

Should I get: A. 64 GB RAM (2 X 32GB) B. 3060 12GB C. Intel A770 16GB.

I’m using openhermes 2.5 Mistral 7b q5k_m gguf, ok-ish Performace for Silly tavern with koboldcpp. But when context goes above 3k, it crawled.

Please let advise which option you think I should take first. Thanks bunch.

  • bebopkim1372@alien.topBEnglish
    1·
    1 year ago

    My main computer is M1 Max Mac Studio. It has 64GB of memory and I can use up to 48GB of video memory. However, it is difficult to use because the modules, libraries, and software support are not very good to use. If you’re a software developer, you will have tough time to make everything working well.

    I bought 4060 Ti 16GB 2 months ago, and I felt that it was very easy making everything runs well on CUDA Development Kit. With Metal from Apple Silicon, I had quite tough time. With Metal environment, I almost always had quite minor problems, and sometimes there was no solution at all. But with NVIDIA’s GPU, things like this never happened. Only small VRAM is the problem.

    I have no experience with A770, but I guess it is similar with Metal.