I think we’ll get better models by having LLMs start to filter out less quality data from the training set and also have more machine generated data, particularly in the areas like code where a AI can run billions of experiments and use successes to better train the LLM. All of this is gonna cost a lot more compute.
ie for coding
LLM proposes experiment, it is run, it keeps trying until its successful and good results are fed back into the LLM training and it is penalized for bad results. Learning how to code has actually seemed to help the LLM reason better in other ways, so improving that I would expect it to help it significantly. At some point, if coding is good enough, it might be able to write its own better LLM system.
But I wonder if the degree of freedom that you have in coding is just too much for RL to work. For Chess and Go or teaching robots how to move you still have a rather finite number of degrees of freedom whereas there should be much more Combinations of code.
Maybe a kinda risc language could be used initally and expanded over time although chatgpt is already doing some amazing things with more complex languages.
There was a paper which I couldn’t find at the moment, which says in the early stages of the gpt, when they added code into its knowledge base, it got better at reasoning. I think that math might help in some other ways, but code can be used to solve math problems and do more than math in anycase.
It’s strange that coding is so much better than math given that they are theoretically equivalent, and the the difference is only in the distribution of things we find interesting. I guess a lot more code is repetitive/mundane compared to harder math?
Code can do things like launch severs, build out system architectures, read in a file, write pixels to the screen, call system calls, call a calculator app or talk to a quantum computer, move a robot, etc… significantly more than math.
I think we’ll get better models by having LLMs start to filter out less quality data from the training set and also have more machine generated data, particularly in the areas like code where a AI can run billions of experiments and use successes to better train the LLM. All of this is gonna cost a lot more compute.
ie for coding LLM proposes experiment, it is run, it keeps trying until its successful and good results are fed back into the LLM training and it is penalized for bad results. Learning how to code has actually seemed to help the LLM reason better in other ways, so improving that I would expect it to help it significantly. At some point, if coding is good enough, it might be able to write its own better LLM system.
But I wonder if the degree of freedom that you have in coding is just too much for RL to work. For Chess and Go or teaching robots how to move you still have a rather finite number of degrees of freedom whereas there should be much more Combinations of code.
Maybe a kinda risc language could be used initally and expanded over time although chatgpt is already doing some amazing things with more complex languages.
Coding is not Important to make better LLM, it’s all about math
There was a paper which I couldn’t find at the moment, which says in the early stages of the gpt, when they added code into its knowledge base, it got better at reasoning. I think that math might help in some other ways, but code can be used to solve math problems and do more than math in anycase.
I think OP is responding to (without commenting on correctness…)
Yes, it’s all about math problems, code is just tool to express them
It’s strange that coding is so much better than math given that they are theoretically equivalent, and the the difference is only in the distribution of things we find interesting. I guess a lot more code is repetitive/mundane compared to harder math?
Code can do things like launch severs, build out system architectures, read in a file, write pixels to the screen, call system calls, call a calculator app or talk to a quantum computer, move a robot, etc… significantly more than math.