For rust there is candle. There are example binaries in the crate that you can just build to run e.g. a llama model. TheBloke also links to Candle in the documentation of models that it can run.

Another rust lib is burn, which has promising support for different backends, but can’t run too many models yet.