Seems like M3 Max is best suited for Large Language Model training. With 128 gb unified memory essential let us train models on billions of parameters! Pretty interesting.
You probably need to wait for the Mac Studio refresh announcements for something more clearly relevant to LLM devs. Hopefully those will have 256GB or more unified memory configs, but likely something for 2024.
That said, it’s handy to be able to run inference on a q8 70b model on your local dev box, so the 96GB & 128GBs are interesting for that.
Get an M1 Ultra Mac Studio 128GB for $1300 less. It has 800Gb/s memory bandwidth vs the 300Gb/s on the M3 Max. Both will have the same 97GB of VRAM to play with, but you’ll utilize far more of the processor on the M1 Ultra.
300GB/s memory bandwidth is the cheaper M3 Max with 14-core CPU and 30-core GPU.
The only CPU that can have 128GB RAM is M3 Max with 16-core CPU and 40-core GPU, and that one has 400GB/s memory bandwidth).
What are your thoughts for someone who needs a dev laptop anyway. For dev a $3200 version is enough. 128Gb one would cost me $5k.
Is almost $2k extra worth it?