@arch consider #microk8s & #ansible?

Cc: @geerlingguy

Jemand aus meiner Bubble daheim mit sowas wie #minikube #k3s #microk8s unterwegs? Ich überlege derzeit meine docker (compose) Umgebung auf eine solche Variante umzustellen.
Einziger Grund: die aktuelle Variante läuft zu stabil und ich such eine Herausforderung

Gibt es da Erfahrungen & Vorschläge was sich da eignet? OS ist ein Ubuntu LTS Server.

#RukiiNet #SelfHosting update:
Got a long electricity #blackout today, almost an hour.
What happens is that all the nodes reboot, but faster than the router-switch in between them, so #MicroK8S #Kubernetes daemons die, and don't respawn without a reboot.
Second, it takes a long time for all the pods to be refreshed from the Unknown state.
Third, Jiva starts resyncing, which tends to fill up one of the smaller hard drives in one of the nodes. Should probably buy a larger one, 500 GB isn't quite enough. When it syncs, it uses this layered file system thing, where it accrues deltas and finally does a garbage collection to squash the space. But it never gets there if the hard disk is already filled up in between. There I need to manually delete the Jiva replica PVC, and let it resync from scratch.
Then Redis pods get stuck as against all reason they corrupt their append-only files if they do a hard reboot. And when #Bitnami #Redis reboots and has a corrupt AOF file, instead of just recovering it, it dies with an error message for how to recover the file manually... It's a huge hassle, it's easier to just scale down Redises, delete PVCs and rescale them back up.
Then #ElasticSearch is dead because of unreachable shards or whatnot. That means I need to bash into the master pod and delete all the indices with curl -XDELETE. After that, I need to bash into Mastodon web pod and do tootctl search deploy all over again...
Gods I hate blackouts.

#RukiiNet #SelfHosting update:
Since deploying #MicroK8S from scratch to a cluster of 3 stable nodes, and disabling WiFi from them (as WiFi interfaces seemed to somehow degrade networking), the cluster has been boringly stable.
So I guess it's 100% uptime from now on.

#RukiiNet #SelfHosting update:
Got a new server, so could replace the one that goes down every day. Took the opportunity to reinstall the cluster from scratch at least up to a point.
Of course #MicroK8S updated and of course it doesn't even start up on Ubuntu anymore (apparently --logfile flag on kubelet was deprecated or something), I don't know what to say anymore. I reverted to an older version, I'm so tired of filing bugs as there are so many.
Well, cluster is fine and great, but then I think all the Bitnami containers updated and gods they somehow managed to break all the file permissions in all the containers somehow. I am currently using a bunch of ad-hoc workarounds, but I really need to deep-dive in those permissions. I mean, it would be fine if we run the container as one user, and mount the file systems with the same, all good. But then somehow it writes files with permissions it cannot itself modify anymore, and the doesn't boot up, saying "cannot mkdir /bitnami/postgresql: permission denied" or something like that. The permission is not denied because of any cluster side things, it is denied because the damn container entrypoint somehow manages to create those directories in a way it can't access them anymore on subsequent boots.
This is all so ridiculous. But I guess nothing some deep thinking won't fix.

#RukiiNet #SelfHosting update:
Just after writing this #Curie went down again, and it didn't help that the #NFS pods were all on a different node. It all went down regardless.

Even got some data corruption again, it's always a huge manual hassle to bring everything back up. I read somewhere that #MicroK8S tends to be bad with hard reboots if some specific singleton cluster pods like coredns or calico or nfc controller or hostpath provisioner are on the node which goes down. I wonder if it's possible to just add replicas for those...

I found a new (old and known) bug with #OpenEBS, and a mitigation. In some cases, #Jiva has replicas in a readonly state for a moment as it syncs the replicas, and if the moon phase is correct, there's an apparent race condition where the iSCSI mounts become read-only, even though the underlying volume has already become read-write.

To fix this is to go to the node which mounted these, do "mount | grep ro,", and ABSOLUTELY UNDER NO CIRCUMSTANCE UNMOUNT (learned the hard way). Instead, I think it's possible to just remount these rw.

There's also an irritating thing where different pods run their apps with different UIDs, and the Dynamic NFS Provisioner StorageClass needs to be configured to mount the stuff with the same UID. I originally ran this by just setting chmod 0777, but the apps insist on creating files with a different permission set, so when their files get remounted, their permissions stay but the UID changes, and after a remount they don't have write access to the files anymore.

This compounds with the fact that each container runs on its own UID, so each needs its own special StorageClass for that UID... Gods.

I got the new #IntelNUC for the fourth node in the cluster to replace the unstable Curie node, but memories for it are coming Thursday.

#RukiiNet #SelfHosting update:
Ok, the issue with Dynamic NFS Provisioner that it tends to leave some weird zombie containers running after hard reboots, perhaps a #Containerd issue, tends to leave these very important pods hanging in ContainerCreating, because "Volume already mounted at more than one place":
https://github.com/openebs/dynamic-nfs-provisioner/issues/153

So, I can fix that manually by assigning/moving these pods to specific hosts it seems, because ReadWriteOnce these things lean upon is node specific. But it's all so very fragile.

I would love it if #MicroK8S #Kubernetes and #Containerd could get over their issues and just kill the zombie containers which shouldn't exist as far as Kubernetes knows. Or if only they would somehow show me which container is mounting the offending share, I could make a script to kill those...

Also, Redis volumes are now moved back to being hostpaths. These don't need to be persistent, actually if they are it just produces more trouble for hard reboots, because the AOF file gets corrupted and needs manual fixing or just deleting and recreating the PVC which is easier actually. I tried Jiva volumes for those which was insane, it took so much resources that my Mastodon became very slow, and it doesn't make sense to replicate Redis disk backs anyhow.

Still waiting for a new #IntelNUC node to maybe swap out #Curie which keeps being unstable due to hardware issues. It tends to go down daily now. And when it goes down, pods migrate around and take a lot of memory, and this interestingly kills the kubelite daemons on the two other nodes, and I have to reboot them as well manually when that happens...

#RukiiNet #SelfHosting update:
After fighting with an unstable host due to memories I believe, and the whole cluster always going down when one node went down, I deep dove into what actually happens.

Turned out as I had installed #OpenEBS #Jiva for replicating volumes on my #MicroK8S using their official Helm charts, it didn't work at all. It just made all the replicas correctly, went through all the motions and then stored all the data on a single pod ephemeral store! I had to take the cluster down to investigate, that took a weekend more or less.

I found out that if OpenEBS Jiva is installed as a MicroK8S plug-in, pointing it specifically to their Git main, and not to a tagged release which doesn't work, then it works. I tried to find out the difference between the Helm chart this installs and the one I had installed, with no luck. I think I installed OpenEBS Jiva Helm chart before and that didn't work, while MicroK8S plug-in installs OpenEBS chart with Jiva enabled as a setting.

Anyhow, ordered a new #IntelNUC again, to reduce my maintenance actions due to one flaky node as well. But as I recreated basically the whole cluster with functioning OpenEBS now, and restored all the (daily) backups once again, it seems everything works and probably single node going down shouldn't take the whole Mastodon down anymore regardless.

During all this I have also filed a lot of issues to the relevant projects on GitHub and documented my findings there, so that people getting the same errors can find solutions.