I wish Mozilla would just stick to Firefox, and invest the rest of the money into some dividend paying fund, so they aren’t so reliant on Google for funding for their software engineers.

46G goliath-120b.Q2_K

So the smallest one I found (I didn’t quantize this one myself, found it on HF somewhere)

And it was very slow. about 13t/s prompt_eval and then 2.5t/s generating text, so only really useful for me when I need to run it on my laptop (I get like 15t/s with 120b model on my 2x3090 rig at 3bpw exl2)
As for the models it’s self, I like it a lot and use it frequently.

TBH, this ram thing is more helpful for me because it lets me run Q5 70b models instead of just Q4 now.

I wish we had a UI like this for GGUF (for Apple)

Hey mate, no need to do this now. There’s a terminal command you can run instead. I did this on my M1 and it works fine.

https://old.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_increase_vram_allocation_with_sudo_sysctl/

> sudo sysctl iogpu.wired_limit_mb=57344

I did that for my 64GB, you’d want to change the 57344 to whatever you want for your 96GB

64GB M1 Max here. Before running the command, if I tried to load up goliath-120b: (47536.00 / 49152.00) - fails

And after sudo sysctl iogpu.wired_limit_mb=57344 : (47536.00 / 57344.00)

So I guess the default is: 49152

That totally worked. I can run goliath 120b on my m1 max laptop now. Thanks a lot.

The reason? Political topics are often dominated by those who have most resources, including more ideological zealots. I have no intention to explain things over and over again. Most people deserve to be handled by a bot if they want to to engage in political discussions, because they’re just a biobot themselves (NPC meme).

This won’t work, because you can convince bots like ChatGPT and get them to agree with your political views. This doesn’t work with real humans though.

Thanks for this, I’m about 3GB short of running Goliath-120b on my 64gb mbp.

lmao

What hardware are you running these on now?

I can run the 3.0bpw exl2 of Goliath on my 2x3090. But for Venus, I could only load it when I dropped the context down to 2048.

Are the spelling issues with the 120b’s because we’re running them at 3bpw vs 4+ for the 70b and smaller?

Yeah okay, I’ll give them a try again. I only ever tried one, and it was completely insane, always ended up with something sexual and after a while started randomly spamming words like ‘sex toy’.

Looks like it was taken down / experimental: https://old.reddit.com/r/LocalLLaMA/comments/16qrdpa/plotbot_13b_finetuned_llama_2_model_for_writing/

haha damn, I should have taken the NSFW warning seriously before clicking the huggingface link in front of people lol.

Is this model any good for SFW stuff?

WizardLM-70B for general gpt-like assistant.

WizardCoder or Codebooga for coding.

I use them daily, but I test models all the time too.

Right. Kind of feels like Intel are leaving money on the table by not writing software for this lol

My 3090’s don’t draw 350W for inference, more like ~200 tops.

I did manage to draw 350W from one by running whisper to subtitle something though.

I’m finding I prefer llama.cpp now as well (the last few days), though for work I usually use Oob + gptq.

If you have it handy, could you post the compile command you used to get llamacpp built for the A770?

Have you tried it though? I’ve been trying for a few months / updates and it doesn’t work.

If HF is blocked there, how are we getting models like Yi and QWEN?

Agreed, CoT makes a big difference. This is WizardCoder Q6 with CoT:

https://imgur.com/a/oFfSQa2

And without: https://imgur.com/a/TchxGhC

Orca-mini refused to write me a batch script to backup my halo mcc replays, because it was unethical lol