Why hasn’t Rust gained the same recognition as Python in AI, ML, and DS, despite its significant ecosystem potential and notably better speed compared to Python?
Also, why haven’t people actively developed Rust libraries for data analysis and processing, similar to what has been done in Python?
You must log in or register to comment.
A lot ML/AI is research where iteration speed is more important than runtime performance or long term maintainability (software engineering).
A lot more programmers know Python than Rust, so it’s easier to prototype a new project in a new domain without having to learn a new language at the same time.
A lot of Python calls C libs to do the heavy lifting, and that’s where the “real” ML/AI code is written, rather than the web apps built around it.
All that said, at PostgresML, we’ve found Rust great for not just ML and AI, but also database and web application development.
https://postgresml.org/blog/postgresml-is-moving-to-rust-for-our-2.0-release
python has everything we need and is well supported across the field. why in the world would we change?
Because python allows you to prototype and iterate quickly, whereas in Rust you have to fight the compiler every step of the way to convince it to do what you want. People have been trying to build DL frameworks in languages such as Swift and C++ (dlib, Flashlight) but none have taken off.
Python can be a pita due to stuff like lack of multi-threading, but for most things it is quick and easy to experiment in, and the amount of code you have to write is not too far off from the corresponding mathematical notation, so for now I think it will keep its position as the most popular language for AI/ML.
Before we could use python, most researchers were using Matlab, which was really holding down progress due to its closed-source nature.
Training jobs for LLMs are quite possibly the biggest programs ever run in terms of compute and yet they use python. “Better speed” is often not true in a way that is actually relevant since cpu speed is generally just not your bottleneck.
Because data scientists are not software engineers
Python enables rapid prototyping. Bindings such as numpy make CPU speed less of an issue.
Not only that, the numerical computation and high performance computing landscape of Rust is very primitive.
Rust hasnt even become mainstream in browsers or embedded systems yet. Its purpose. Why would it be used in ML when languages like julia arent popular?
As some above comments mentioned, iteration speed is the most important in ML, so I don’t see python being replaced by something else soon. However, there are some Rust projects wrapped with python for better efficiency and safety (Huggingface’s Tokenizers and Safetensors for instance).