Not super knowledgeable about all the different specs of the different Orange PI and Rasberry PI models. I’m looking for something relatively cheap that can connect to WiFi and USB. I want to be able to run at least 13b models at a a decent tok / s.
Also open to other solutions. I have a Mac M1 (8gb RAM) and upgrading the computer itself would be cost prohibitive for me.
What do you define as “decent” tokens per second? Do you have a budget yet? Do you want to run the 13B at full precision or a quantized precision?
20 tok/s seems like the minimum I would be sane with lol