Currently I have a 4090x2 and I am looking to upgrade my machine. Due to well known reasons, the price of 4090 and 3090 is insanely high right now, and I see another option: magic modding a 3080 with 20g of vram.
My aim is to use qlora to fine tune a 34B model, and I see that the requirement for fine tuning a 34B model using a single card from qlora is 24g vram, and the price of 4090x2 is about equal to 3080 20g x8. so what would be a better choice for a multi-card?
4090x2 or 3080 20g x8?
What kind of mobo would you put the 8x 3080s into without killing the bandwidth and creating tons of headache for yourself in getting it to work? There was a similar post here recently. Just stick to the 4090s.
emmmm, I’m not sure, all I can do is just plug them into the motherboard.
Do you have a link to the post please? I’d like to check it out.
Does your motherboard even have 8 PCIe slots? You’ll be needing server boards. Even workstation boards typically support only 7 (and can’t fit them all due to size)
Using a PCIe splitter cable, split the PCIe 4.0x16 x1 into PCIe 4.0x8 x2. And all gpu use a PCIe extension cable.
I would recommend against this, not to say you are incapable o lr wouldn’t figure it out, but it’s a head ache. Just the power consumption on something like this alone would drive you to a sizable power bill, not to mention lack of any kind of Nvlink, your 4090 setup is a much better option in my opinion.
You already have rtx 4090 x 2? Then why upgrade, i don’t get it. Tell me more about that magic modding, I can’t find anything related online.