Did anyone manage to get them working? I tried GGUF/GPTQ and running then unquantized with trust-remote-code and they just produced garbage. (I did try removing BOS tokens and still same thing)
I’ve completely fixed gibberish output on Yi-based and other models by setting the RoPE Frequency Scale to a number less than one, which seems to be the default. I have no idea why that works, but it does.
What I find even more strange is the models often keep working after setting the frequency scale back to 1.
Did anyone manage to get them working? I tried GGUF/GPTQ and running then unquantized with trust-remote-code and they just produced garbage. (I did try removing BOS tokens and still same thing)
Yeah, exactly the same thing. Produced absolutely rubbish whatever i tried. I tried 8B 15B and 23B
I’ve completely fixed gibberish output on Yi-based and other models by setting the RoPE Frequency Scale to a number less than one, which seems to be the default. I have no idea why that works, but it does.
What I find even more strange is the models often keep working after setting the frequency scale back to 1.
What value specifically worked?