Wondering what everyone thinks in case this is true. It seems they’re already beating all open source models including Llama-2 70B. Is this all due to data quality? Will Mistral be able to beat it next year?
Edit: Link to the paper -> https://arxiv.org/abs/2310.17680
I think it’s plausible. Gpt3.5 isn’t ultra smart. It’s very hood most of the time, but it has clear limitations.
Seeing what mistral achieved with 7b, I’m sure we can get something similar to gpt3.5 in 20b given state of the art training and data. I’m sure OpenAI is using some tricks as well that aren’t released to the public.