turamura@alien.topOPBtoLocalLLaMA@poweruser.forum•Exllama outside of text generation webui?English
1·
1 year agoHi, thanks for your comment!
I saw e.g., the “inference.py” in the repo which I think I could utilize. It actually looks kind of simple. However, I am struggling with what to provide as the “model directory”. Should I just download a Huggingface model (for example, I would like to work with TheBloke/Llama-2-70B-GPTQ), and then specify this as model directory? Or what kind of structure does ExLlama expect as model directory?
Got it to work! Thank you!!