I’d like to run a local model on a server in a data center. Assumptions:
- Lots of DDR4 RAM (assume 256GB with most available to the AI work) and storage, but no standalone GPU.
- Looking for a web frontend and Auto GPT-ish ability (i.e., it can search for answers on the web) if such a thing is currently available with locally hosted models.
- I’d like to train it on policy reports (assume 10-12 pages of text per report), but I can live without this if training is wholly off the table without a dedicated GPU.
I’m very comfortable with Linux, running servers, virtual environments, etc. But not spun up on the latest in locally hosted LLMs. Assume I’m an idiot about all of this and point me in the right direction? Thanks!
1: What is your budget?
2: Do you have access to the data center and are you able to put in a GPU?
3: Does the server have SXM sockets or just PCIE?Well, the budget is $0 at the experimental stage – we wanted to see what we could achieve by throwing a lot of RAM at the problem.
If we have to add a GPU, Yeah, we have access to the server. Current box would accommodate a Gen3 PCIe, FHHL slot. Frankly, if we have to invest in a GPU, we’ll also upgrade the whole server to a later-gen/more-powerful CPU, NVME storage, etc.
you could get a gpu like a p100 16gb for simple ai work or a v100/A4 for slightly more heavy duty work
p100s only cost around 170$, so it’s cheap to upgrade the gpu