Exciting and worrying… I have gone to great efforts to use safetensors… I would have to see every model packaged in executable format… but then again I have seen comments about llama.cpp behavior changing for the same model and settings (not sure if it is true but that could be bad)

Well then… Thanks! I’ll use llama.cpp and be happy. Glad to hear llamacpp_hf is crazy and not me. Which tool do you prefer outside of Oobabooga?

It’s only been a day but have you changed? I find this model misspells a lot with the gguf i downloaded.

So helpful… but Yi and llamacpp_hf just falls apart for me… complete gibberish on Oobabooga. Exl hf … fine. Llama.cpp fine… Min-P is there and I can apparently use it but temperature last is missing :/

I’m on Pop, lol. I could get it to compile, but I must have missed a step for nvidia acceleration

It is… but koboldcpp doesn’t have a executable for me to run :/

I could never get up and running on Linux with Nvidia. I used Kobold on Windows, but boy is it painful on Linux.

Please try 70b down to ~30b with a llama 2 model. Thanks!