Research papers have also observed diminishing returns issues as models grow.
Hell maybe even GPT-4 was hit by this and that’s why GPT-4 is not a single giant language model but running a mixture of experts design of eight 220B models trained for subtasks.
But I think even this architecture will run into issues and that it’s more like a crutch. I mean, you’ll eventually grow each of these subtask models too large and might need to split them as well, but this might mean you run into too small/niche fields per respective model and that sounds like the end of that road to me.
I mean, that is literally how any form of AGI will work. No one in the field has ever thought one model will be capable of reaching AGI. All these models are highly specialized for the task in which they are trained. Any move towards an AGI will be getting many of these highly specialized AI’s to work in conjunction with one another. Much like how our own brains work.
Could you please share a citation for the mentioned research papers?
Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.
Could you please share a citation for the mentioned research papers?
I’m interested in seeing this as well.
He probably means that, although scaling might still deliver better loss reduction, this won’t necessarily cash out to better performance “on the ground”.
Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3’s in-context learning. GPT4 displays no new abilities.* Yes, it’s smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there’s nowhere trivial left to go. But returns do seem to be diminishing.
(*You might think of multimodality, but they had to hack that into GPT4. It didn’t naturally emerge with scale, like, say, math ability.)
Research papers have also observed diminishing returns issues as models grow.
Hell maybe even GPT-4 was hit by this and that’s why GPT-4 is not a single giant language model but running a mixture of experts design of eight 220B models trained for subtasks.
But I think even this architecture will run into issues and that it’s more like a crutch. I mean, you’ll eventually grow each of these subtask models too large and might need to split them as well, but this might mean you run into too small/niche fields per respective model and that sounds like the end of that road to me.
I mean, that is literally how any form of AGI will work. No one in the field has ever thought one model will be capable of reaching AGI. All these models are highly specialized for the task in which they are trained. Any move towards an AGI will be getting many of these highly specialized AI’s to work in conjunction with one another. Much like how our own brains work.
You should double check what mixture of experts actually means 🙂
Could you please share a citation for the mentioned research papers?
Last I looked into this, the hypothesis was that increasing parameter account results in a predictable increase in capability as long as training is correctly adapted.
https://arxiv.org/pdf/2206.07682.pdf
Very interested to see how these larger models that have plateaued are being trained!
I’m interested in seeing this as well.
He probably means that, although scaling might still deliver better loss reduction, this won’t necessarily cash out to better performance “on the ground”.
Subjectively, GPT4 does feel like a smaller step than GPT3 and GPT2 were. Those had crazy novel abilities that the previous one lacked, like GPT3’s in-context learning. GPT4 displays no new abilities.* Yes, it’s smarter, but everything it does was possible, to some limited degree, with GPT3. Maybe this just reflects test saturation. GPT4 performs so well that there’s nowhere trivial left to go. But returns do seem to be diminishing.
(*You might think of multimodality, but they had to hack that into GPT4. It didn’t naturally emerge with scale, like, say, math ability.)