On the Eleuther AI discord, someone once asked that question. And someone else replied that yeah, obviously having 1 matrix instead of 2 should be better in theory, but then, in practice, empirically, that makes things worse. Why? Noone knows.

First, I tried vast.ai because I had heard good things about it and it had instances with good hardware and not super expensive instances. It was a dumpster fire. Instances having random technical problems, the support never responding.

Then I tried lambda labs. It was super awesome, good price, basically the best thing ever, except they were often out of capacity and didn’t have suitable instances available.

Then I heard somewhere about oblivus.ai. Tried it, was also a dumpster fire, basically the same as vast.ai - technical problems with instances (even non-community ones), support that never responded.

Then I tried google cloud compute and it’s expensive and complicated but at least it works. That’s what I use nowadays.