Selfhosted

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

4 acronyms in this thread; the most compressed thread commented on today has 20 acronyms.

[Thread #47 for this sub, first seen 15th Aug 2023, 07:25] [FAQ] [Full list] [Contact] [Source code]

I'm in the process of migrating my servers to NixOS. It takes a lot of time and the learning curve is steep, but I have one config shared for all the servers and PCs. I have setup the servers to automatically pull the latest configuration everyday and even restart if there's a kernel update.

This means I just need to update my laptop and push the changes to the repository, and all the servers will also update.

I haven't had this setup long enough to know if things will break unexpectedly with updates tho. NixOS has a great feature where you can rollback to a previous configuration (generation) with a single command. You can always keep using containers to isolate updates, if you want (Nix allows you to declare those in the config as well).

As an example, you can take a look at my config.

EDIT: Systemd timers have an option to randomize the time a service runs, I use it all the time. The option for Nix's config pulling is using systemd timers, so you can use that.

A few simple rules make it quite simple for me:

• Firstly, I do not run anything critical myself. I cannot guarantee that I will have time to resolve issues as they come up. Therefore, I tolerate a moderate risk of a borked update.
• All servers run the same be OS. Therefore, I don't have to resolve different issues for different machines. There is then the risk that one update will take them all out, but see my first point.
• That OS is stable, in my case Debian so updates are rare and generally safe to apply without much thought.
• Run as little as possible on bare metal and avoid third party repos or downloading individual binaries unless absolutely necessary. Complex services should run in containers and update by updating the container image.
• Run unattended-upgrades on all of them. I deploy the configuration via Ansible. Since they all run the same OS, I only need to figure out the right configuration once and then it's just a matter of using Ansible to deploy it everywhere. I do blacklist kernel updates on my main server, because it has ZFS through DKMS on it so it's too risky to blindly apply.
• Have postfix set up so that unattended-upgrades can email me when a reboot is required. I reboot only when I know I'll have some time to fix anything that breaks. For the blacklisted packages I will get an email that they've been held back so I know that I need to update manually.

This has been working great for me for the past several months.

For containers, I rely on Podman auto update and systemd. Actually my own script that imitates its behavior because I had issues with Podman pulling images which were not new, but which nevertheless triggered restarts of the containers. However, I lock the major version number manually and check and update major versions manually. Major version updates stung me too much in the past when I'd update them after a long break.

I set up flexo for Arch Linux update caching and squid proxy for Alpine, Debian. This stops me from having to download the same files over and over.

apk upgrade -U in a cronjob daily and hope it does not break.

I did setup Unattended Upgrades and it works OK, but there was a little bit of work involved in getting it setup just right. I don’t relish the idea of doing that another 40 or so times for the rest of my fleet.

automate it! I run unattended-upgrades on dozens of servers without any problems: [1] [2]. Configuration is actually really simple.

I use other methods for things that are not distribution packages [3], but for APT upgrades unattended-upgrades is the only correct™ solution.

I consolidated everything on to one a bit beefier VPS so that if I update that one VPS 90% of stuf updates itself. The rest are 3 RPis which run Home Assistant in different places, those I go throuch manually and update when I see that there is a new update.