I’ve been pondering something recently. Did you notice that achieving over 70% on the well-known HumanEval pass@1 hasn’t been making major headlines? Models like WizardCoderV2, Phind, Deepseek, and XwinCoder have all surpassed the 67% reported in GPT-4’s report. Some of them are even closely tailing the 82% of GPT-4 API’s. So, are these models really performing that well?
Here’s something intriguing: I found this image in the latest release of XwinCoder’s repo: Xwin-LM/Xwin-Coder at main · Xwin-LM/Xwin-LM (github.com)
It shows that GPT-4 achieves a 60% pass@1 on APPS-introductory, which is higher than CodeLLaMA-34B’s pass@100 (56.3) and XwinCoder-34B’s pass@5 (43.0). Interesting, isn’t it?
This suggests that judging a model based on a single benchmark might not provide the full picture. This leads me to a couple of questions:
- What exactly is the gap here? How can we definitively say one model outperforms another?
- How are other recent models performing on benchmarks like APPS and DS1000?
I’m interested in hearing your thoughts on this. Has anyone experimented with these new models? What was your experience like?
You must log in or register to comment.
This suggests that judging a model based on a single benchmark might not provide the full picture.
Duh… This has been a recurring problem with all these “benchmark leaderboards”. It turns out that “training on the testing set is all you need”…
I haven’t got round to trying the xwin coder models, but the precursor 70b chat model was extremely impressive when compared against both chat GPT 3.5 and 4.
If you look at something like evolinstruct data its so similar to humane al itd be a surprise if models trained on that data (or other synthetic data) dont perform well
As a rule of thumb i only generally trust base models (even then its iffy) on benchmarks and for finetuned models only by using it