Was wondering if theres anyway to use a bunch of old equipment like this to build an at home crunch center for running your own LLM at home, and whether it would be worth it.
Was wondering if theres anyway to use a bunch of old equipment like this to build an at home crunch center for running your own LLM at home, and whether it would be worth it.
Another consideration is that I was told by someone with multiple cards, that if you split your layers across multiple cards, they don’t all process the layers simultaneously.
So, if you are on 3x cards, you don’t get a parallel performance benefit of all cards working at the same time. It processes layers on card 1, then card 2, then card 3.
The slowest card will obviously have the worst speed. Not sure what this will do for your load times of a model or your electricity bill, as well as the fact you need a system big enough to fit them all in.