My sense of it is that most training is still just using the APIs to talk to the GPU, and the art is more in the assembly of the training set than it is optimizing the APIs. There are serious researchers working on improving AI, but they’re figuring out how to make the data get in and out of the GPU faster in a way that doesn’t hurt later quality. But that’s not a code optimization problem, it’s much more of an “understanding the math” kind of problem and whether you’re then using Python or Go to tell the GPU to execute this slightly different math isn’t much of a concern.

I think it’s a lot like data science. Getting the good clean data to work with is actually the hard part. For training, getting great training sets is the hard part.

If you just wish to write code that uses AI, or train models for that purpose, the current Python toolkit is more than sufficient, especially given how quickly everything is moving right now; we might have totally different architectures in three years, and Python will be quicker for that R&D iteration than Go is.

Finally, on your personal thing - I’ve been coding for 39 years. I’ve worked in BASIC, Assembly, C, C++, Perl, Python, and Go (and 37 other languages here and there). Go to Python isn’t going to be a difficult jump. Especially now that you can…use an AI to help you if you’re at all confused how to turn a Go concept into a Python one.

The model training is also GPU-intensive. If you read about model training costs, they talk about things like “millions of GPU-hours.”

As I understand the process (note, I am a software developer, but do not professionally work on AI), you’re feeding lots of example text into the GPU during the training process. The hard work is curating and creating those examples, but that’s largely either human-intensive or you’re using a LLM to help you with it, which is…GPU-intensive.

Optimizing the non-GPU code just isn’t much of a win. I know all the cool kids hate Python because it’s typeless and not compiled, but it’s just not much of a bottleneck in this space.

The runtime of your code basically doesn’t matter. You hand it off to a GPU for all the hard calculations, and even a fast environment, your code is going to spend 99% of its execution time waiting to get stuff back from the GPU.

I’m sure Python’s I/O polling sleeps are just as efficient as Go’s.

While speed decreases with context length, I wonder if with a small context you’re fitting entirely in GPU, but as the context length increases, you exceed the available VRAM and it has to offload part of it to CPU?

To check if this is the issue, take a chat that’s performing poorly and load a smaller model on it.

If that is your issue, you can try to fine-tune your context length to fit inside your available VRAM, or use a smaller model if you need the longer context.