Text generation web UI. The install script has worked perfectly every time I’ve run it, and the miniconda environment it creates is useful both within the web interface and for running LLM in python scripts. The interface also makes installing and using new models a breeze.
Text Generation UI as the backend and sillytavern as the front end.
KoboldCPP where proper transformers/cuda isn’t supported.
Yep pretty good combo! I also use ooba+Silly and for internet query and pdf ingestion I use LolLLMs Great stuff!
Previously when I was more VRAM limited - koboldcpp. Now, I mainly use modified cli exllamav2 chat.py and oobabooga 50/50. Chat.py is about 8 token/s / 45% faster then oobabooga with the same model and exllamav2 loader for some reason, and I like having fast generation more than having nice UI. You forgot to mention SillyTavern, I think it gets a lot of use among coomers.
Bettergpt with llama.cpp server and its openai adapter, sleek, supports editing past messages without truncating the history, swapping roles at any time etc.
Text gen web ui. Let’s me use all model formats depending on what I want to test at that moment.
Text Gen Web UI. Works great on Mac. I use ggufs, since Llamacpp supports metal.
rocm version of KoboldCPP on my AMD+Linux
KoboldCPP. Double click Kobold Icon, Load, select preset, Launch. 10 or so second later you’re good to go. Easy, quick, efficient.
Text Gen Web UI + Silly Tavern for me. Works like a charm.
I just switched to KoboldCpp from Text Geb UI 2 days ago.
The OpenAI extension wouldn’t install for me and it was causing issues with SillyTavern which I use as a frontend.
I’m actually really happy now that I’ve switched.
KoboldCpp is so simple is great. I’ve written a simple batch file to launch both KoboldCpp and SillyTavern. All I have to do if I want to try a new model is edit the part of the batch pointing to the name of the model and it just works.
On top of that I can load more layers onto my GPU with KoboldCpp than Text Gen UI so I’m getting faster speeds.
Have you tried installing the missing package files it shows when u tried installing the openai extension?
I had the same issue and installing those missing packages via the cmd_windows in the same folder.
If you have coding skills,->https://github.com/mzbac/LLM_Web . Can deploy to local server or cloud
my own: https://github.com/knoopx/llm-workbench reasons: fast, private, lightweight, hackeable
You’re kidding me. I recently surfaced my own UI with the same name. damn it. -> https://github.com/sedwards2009/llm-workbench
I used to use Text Generation Web UI, but I changed to KoboldCpp because it’s more lightweight. Besides, I realized I didn’t use all the features of the textgen UI. KoboldCpp as the backend and SillyTavern as the frontend when I want to chat. KoboldCpp alone when I want to play with models by creating stories or something.
LM Studio - very clean UI and easy to use with gguf.
Found KoboldCpp on a guide somewhere and only used that. I barely even know much about anything else. I Just use GGUF and never worry about the so-called “VRAM”