Could you please share a citation for the mentioned research papers?
Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.
Could you please share a citation for the mentioned research papers?
I’m interested in seeing this as well.
He probably means that, although scaling might still deliver better loss reduction, this won’t necessarily cash out to better performance “on the ground”.
Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3’s in-context learning. GPT4 displays no new abilities.* Yes, it’s smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there’s nowhere trivial left to go. But returns do seem to be diminishing.
(*You might think of multimodality, but they had to hack that into GPT4. It didn’t naturally emerge with scale, like, say, math ability.)
Could you please share a citation for the mentioned research papers?
Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.
https://arxiv.org/pdf/2206.07682.pdf
Very interested to see how these larger models that have plateaued are being trained!
I’m interested in seeing this as well.
He probably means that, although scaling might still deliver better loss reduction, this won’t necessarily cash out to better performance “on the ground”.
Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3’s in-context learning. GPT4 displays no new abilities.* Yes, it’s smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there’s nowhere trivial left to go. But returns do seem to be diminishing.
(*You might think of multimodality, but they had to hack that into GPT4. It didn’t naturally emerge with scale, like, say, math ability.)