I just released the NeuralHermes-2.5-Mistral-7B model, which is a DPO fine-tuned version of OpenHermes-2.5-Mistral-7B. Teknium, the creator of the SFT model, confirmed on Twitter that this version improves benchmark scores in AGIEval, GPT4All, and TruthfulQA.
Take is a simple proof of concept: I used Intel’s orca_dpo_pairs (from neural-chat-7b-v3-1) in a ChatML format, and only trained it for one hour on an A100 (using Goole Colab). But it shows the potential of DPO to boost the performance of SFT models, basically for free. I released all the code so that everyone can easily experiment with it and find better parameters (it shouldn’t be difficult). You can also access the W&B project.
Note that the preference dataset is also entirely synthetic, with preferred answers coming from GPT-4/3.5 and rejected responses coming from Llama 2 13b chat. It’s a very cheap and efficient way to convert an instruction dataset (OpenOrca in this case) into a preference dataset. I wasn’t very successful in my previous experiments with DPO using other datasets, so I think there’s something very interesting with this one. We can easily reproduce this dataset and improve it with other sources.
I just wanted to share these thoughts and experiments with the community. I’m writing an article about DPO and this model is actually a lucky by-product of it. I’ll share it when it’s ready.
If you want to try the model, TheBloke already provided GGUF and AWQ versions of it.
It holds up pretty decent! What Mirostat Tau value would you recommend with it?
The improvement is so small it can be a margin of error
New favorite model!
what does it feel like to generate tokens?
Wow
nice job!
really cool! what do you think about using gpt3.5 as the worst output in the hopes to resurface some extra edge?
Yes, I’d say it’d probably work better than the current approach. If you look at the reward plots on wandb, it feels like the problem is too easy for the model, hence slight improvement.
I find it odd that your chosen rewards went negative… Doesn’t this imply that the chosen samples became less likely than they were under the base model? You still get model improvements, since the rejected rewards got even less likely, but it’s still odd feeling. Any thoughts there?
Would be cool to see this in a 34b and 70b.
works really well to get it on the 16k version https://huggingface.co/NurtureAI/OpenHermes-2.5-Mistral-7B-16k
would it have to be a different dataset?what is the difference between normal and 16 K?
It’s a good question, I can give it a try. Ideally, you’d want a 16k version of the preference dataset to make sure that DPO doesn’t ruin it. But considering the low number of training samples, it probably works fine.
