I like to follow articles benchmarking OSs on phoronix a lot. Whenever Arch looks bad I see comments riddled with saying that is because the default scheduler sucks. I feel fairly compitent with Linux but for some reason schedulers seemed like this black box that lives in the realm of places where I normally break my OS from not paying close attention.
Is it a program run by something like systemd? Is it a config or patch of the kernel? Which ones are good and how important are they?
Anyways, any advice on schedulers would be appreciated.
Faster here has many different meanings. The tests in that benchmark are more batch processing tasks. But that is not the only measure of speed. Zen I believe is more tuned for desktop use and keeping the system in a more snappy state which comes at the cost of some raw throughout performance. There is forever a tradeoff between latency and throughput, optimising for one is generally done at the expense of the other and both are optimising for performance, just different kinds.
I only mentioned zen as one of the official alternative kernels that is worth a look at though, making no claims as to if it the best for anyones particular workloads.
This is why it is best to do your own benchmarks for the stuff you care about and pick the best solution for your situation.