I’ve been pondering something recently. Did you notice that achieving over 70% on the well-known HumanEval pass@1 hasn’t been making major headlines? Models like WizardCoderV2, Phind, Deepseek, and XwinCoder have all surpassed the 67% reported in GPT-4’s report. Some of them are even closely tailing the 82% of GPT-4 API’s. So, are these models really performing that well?
Here’s something intriguing: I found this image in the latest release of XwinCoder’s repo: Xwin-LM/Xwin-Coder at main · Xwin-LM/Xwin-LM (github.com)
It shows that GPT-4 achieves a 60% pass@1 on APPS-introductory, which is higher than CodeLLaMA-34B’s pass@100 (56.3) and XwinCoder-34B’s pass@5 (43.0). Interesting, isn’t it?
This suggests that judging a model based on a single benchmark might not provide the full picture. This leads me to a couple of questions:
- What exactly is the gap here? How can we definitively say one model outperforms another?
- How are other recent models performing on benchmarks like APPS and DS1000?
I’m interested in hearing your thoughts on this. Has anyone experimented with these new models? What was your experience like?
Duh… This has been a recurring problem with all these “benchmark leaderboards”. It turns out that “training on the testing set is all you need”…