Lemux Server Updates & Issues

If you're on this server hopefully you can tell the difference. Things should be quicker and no delays between clicks. Just as a look behind the curtain, this is what I'm running now if you were interested in hosting yourself:

Proxmox VE 8.2.2

i5-11400

64GB RAM

2x Intel Enterprise D3-S4610 3.84TB SSDs for VM/CT storage

1x Crucial T500 NVMe with DRAM cache for OS (Ext4)

ZFS pool with mem cache (ARC) enabled for thin-provisioned VM/CT storage

ZFS ARC uses a ton of RAM but it gets released as needed. The performance boost is worth it though. The OS drive was TRIM'd with blkdiscard before only partitioning 2/3 of the drive. The rest of the empty space is used by the controller for wear leveling. The performance is increased with the tradeoff being a sacrifice of space.

Next is upgrading the proxy and backup servers, but that won't affect lemux. As before if you experience any issues, please post them in this community.

~minnix

You may or may not have noticed there's been server issues the past couple of days. We've had some severe weather here in Texas with massive power outages. The UPS only has so much battery power. For now we have power.

I have enterprise SSDs on the way which should solve the speed and availability issues (lots of reasons not to use consumer SSDs in a HA cluster which I'm currently doing but won't get into the details). Since I'll have to open the server anyway I will be updating the hypervisor and backup servers to the next major releases since point release updates will end at the end of July. This will take some time with my schedule but if all goes well it shouldn't be more than a day or two after testing (famous last words). There hasn't been a new release for Lemmy since January so that will stay the same, currently 19.3.

What will for sure happen: I'm noticing a decrease in speed on the server and occasional timeouts when posting. This is 100% an I/O problem. I believe it's due to a cheap PCIe adapter used for connecting more SSDs than I actually need to the server. I will be removing this adapter and using the native SATA ports in the near future, probably replacing the current consumer grade SSDs with enterprise grade as well. This will result in maybe a day's downtime as I migrate containers to the newly created cluster from backup.

What will possibly happen: Right now I'm restricted to a 1Gb network due to my current NICs. This results in slow performance when backing up the server at night as the bandwidth is saturated. I could restrict the backup speed in order to preserve I/O for the rest of the server, which would result in slower backups, or I could bite the bullet and spring for 10Gb NICs. Since this is just a hobby project that I fund myself, I still haven't decided what I will do. I'll be checking current prices on NICs and looking at what my mobos support.

What may happen in the future: As the lemmy ecosystem continues to grow and mature, several new UIs have appeared that offer more functionality, theming options and additional admin/moderation tools. Some of these are progressing rapidly and I'm keeping an eye on them to see if they continue to receive support past the beta stages. If they seem like viable options, I may do some testing and implement them at different times just to see if they are actual improvements to how lemux currently operates. I'll make sure to make users aware beforehand via posts in this community if this happens.

~minnix

Due to authentication bug, 19.3 was released yesterday.

release notes

change log

Hopefully this will sort out the federation bug which seems to be the main point of this release.

Release Notes

Change log

https://github.com/LemmyNet/lemmy/issues/4288

Waiting on the next update to address this. In the meantime, regular restarts seems to fix this temporarily. Server will be restarted 4 times a day until fix is available. If you notice random downtimes this is the reason. Takes about two minutes to come back up so should be ok as far as minimal downtimes.

Release notes

Bugfix update:

0.19.0 has a critical bug where sending outgoing activities can stop working. The bug is fixed in this version. It also fixes the “hide read posts” user setting, fixes a problem with invalid comment paths, and another fix for private message reports.

So basically posts and comments weren't posting (including my own). For once this was not the fault of my own ineptitude.

release notes

Any issues list them here. You may have to re-login again since the upgrade.

Latest update to 18.5. Any issues comment here.

Release notes

If you're seeing some posts missing, there was a problem with a corrupt database. When restoring from backup, one of the backup disks was corrupted in the array. An offsite backup was restored, but those are only backed up once a week. The bad disk has been replaced and everything checked, and offsite backup frequency has been increased. Hopefully everything is stable for the time being. Sorry for the screw up.

Latest update to 18.4. Any issues comment here.

Release notes

If you'd like me to apply LCS https://github.com/Fmstrat/lcs to your account let me know. I'm currently testing it on my own account. Basically what it does is enable users on smaller instances like this one to see a wider amount of content from across the fediverse when browsing "All" instead of just what they are subscribed to, similar to Reddit's All. Current default instances that it pulls from are beehaw.org, lemmy.world, lemmy.ml, sh.itjust.works, and lemmy.one. You can add or subtract instances as well.

Any issues let me know

If you had an issue uploading media in the past it has been fixed. See commit

Server has been updated to 18.1. Any issues go in this thread.

Since lemmy isn't coming back up properly after backup I set a cronjob to reboot after backup. Unfortunately shutdown was configured properly but startup had the wrong command leading to qm. Fixed and should work tomorrow.

When the server goes down for backups at night and reboots, the lemmy service is not returning so I have to reboot it again in the morning. If you experience outages between 3:30 to 10:00 CST this is why. Looking into it now.

One person had issues subscribing to a community but was eventually able to subscribe through the Connect app. Other than that, no issues.