I finally accepted that Docker Swarm is not enough for my use cases, so my plan is to put together a k8s cluster with my workstation and my 2 home servers.
You should look into K3S
I’m aware of it and it’s quite likely what I am going to be using. Have you any experience with it?
My main challenge at the moment is that I didn’t find a good architecture to scale stateful services horizontally on Docker Swarm. I’ve been happy with everything else, except this. I can scale my object storage by just adding new nodes to minio storage pools, I can provision new services fine and ansible + portainer has been enough for me on my CD pipeline.
The one thing that is still a bit fragile on my setup is that any volumes for services like a database or elasticsearch end up tied to the server, and there is no good solution to make HA for them. I tried glusterFS but quickly learned that it can corrupt a database if your service gets rescheduled to a different server and it restarts before the replication is complete.
I think that what I am looking for is something like a Ceph cluster and just use that a proper driver for it, but unfortunately I didn’t find any docker plugin that is well-maintained for that. This is the only reason that I’m looking into kubernetes at all.
I have a bit of knowledge on how it works and when its use could be a good choice, but I have only played around with it. Either way I would strongly suggest you look into Rancher and Harvester by Rancher. I’m most likely to personally use them too thanks to their help with setup and configuring security for your K3S cluster. Also look into thier Longhorn software when it comes to the storage part. I hope I could be of help!
deleted by creator
