I recently got a 32GB M1 Mac Studio. I was excited to see how big of a model it could run. It turns out that’s 70B. It is a Q3_K_S model so the 2nd smallest for 70B in GGUF format, but still it’s a 70B model.
As many people know, the Mac shouldn’t be able to dedicate that much RAM to the GPU. Apple limits it to 67%, which is about 21GB. This model is 28GB. So it shouldn’t fit. But there’s a solution to that thanks to these smart people here.
https://github.com/ggerganov/llama.cpp/discussions/2182
They wrote a program to patch that limit in the kernel. You can set it to anything you want. So I cranked mine up to 92%. I also do these couple of things to save RAM.
-
I don’t use the GUI. Just simply logging in and doing nothing uses a fair amount of RAM. I run my Mac headless. I ssh in.
-
I stopped the mds_stores process from running. I saw that it was using up between 500MB and 1GB of RAM. Its the processes that indexes the drives for faster search. Considering my drive is 97% empty, I don’t know what it was doing to use up 1GB of RAM. I normally turn off indexing on all my machines always.
With all that set, the highest I’ve seen in use memory is 31.02GB while running a 70B Q3_K_S model. So there’s headroom. There maybe a lot more. Since my goal is to not swap. I noticed that when I log into the GUI while it’s running a model, the compressed RAM goes up to around 750MB but it still doesn’t swap. So I wonder how far memory compression would let me stretch it. I do notice that it’s not as snappy. With no GUI login, the model runs right away after the model is cached after the first run. With a GUI login, it pauses for a few seconds.
As for performance, it’s 14 t/s prompt and 4 t/s generation using the GPU. It’s 2 and 2 using the CPU. Power consumption is remarkably low. Using the GPU, powermetrics reports 39 watts for the entire machine but my wall monitor says it’s taking 79 watts from the wall. Using the CPU powermetrics reports 36 watts and the wall monitor says 63 watts. I don’t know why it’s so much more efficient at the wall between GPU and CPU. It’s only a 3 watt difference in the machine but 16 watts at the wall.
All in all, I’m super impressed. The M1 32GB Studio may be the runt of the Mac Studio lineup but considering that I paid about what a used 3090 costs on ebay for a new one, I think it’s the best value for performance I have to run LLMs. Since I plan on running this all out 24/7/365, the power savings alone compared to anything else with a GPU will be several hundreds of dollars a year.
You may try and run one of Q4 models without problems: because llama.cpp uses mmap to map files into memory, you can go above available RAM and because many models are sparse it will not use all mapped pages and even if it needs it, it will swap it out with other pages on demand… I was able to run falcon-180b-chat.Q6_K which uses about 141GB on a 128GB Windows PC with less than 1% SSD reads during inference… I could even run falcon-180b-chat.Q8 which uses about 182GB but in this case SSD was working heavily during inference and it was unbearably slow (0.01 tokens/second)…
Yes. I’ve done that before on my other machines. Llama.cpp in fact defaults to that. The hope for me was that since the models are sparse that the OS would cache the relevant parts of the models in RAM. So the first run through would be slow but subsequent runs would be fast since those pages are cached in RAM. How well that works or not really depends on how much RAM the OS is willing to use to cache mmap and how smartly it does it. My hope was that if it did it smarty with sparse data that it would be pretty fast. So far, my hopes haven’t been realized.