Unfortunately the server went offline today for approximately 12 hours. This is due to a deficiency in the backup routine, where if the vps hoster I use turns off the server/has an interruption, during the window of about 30 minutes where the backup runs then the script does not put everything back in place to ensure the server can start up again.
The good news is, this new issue that was not thought about should be resolvable with a reboot script, that uses an if or statement to check if the backup did do what it was supposed to and if not take some fairly simple logic to put things back as they should be.
In the next couple of weeks I will implement and test such a solution.
However it does raise another issue in that lemmy had some growth and then shrinking, that has stabilised and left us with a constant but slow increase in users, lost some of the admins that used to help check this server and they have abandoned their lemmy deployments for whatever reason and there are less people around actually checking of things are working.
Sadly I didnt notice the server was offline for 12 hours as I was working and in meetings all day and only got around to checking lemmy later in the afternoon. This is disappointing as the fix only took a few minutes - less than 10 to login, check what happened and correct it.
Had I been made aware of an issue earlier, many hours of downtime would have been saved.
To be clear, this server is run extremely cheaply. This means we dont have as many automated checks as other more expensive lemmy servers and we must rely on our members to tell us if things go wrong. This design choice was chosen in case lemmy becomes less popular and so the cost of the server is no issue. Essentially I can pay and run it out of my pocket just as a public service.
What would be great is if anyone can join the admin team to do some actual man hours work like, respond to new users joining the server, checking its functional and messaging me if not, blocking a user here or there if we get a complaint and so on. All of these tasks have become harder as the admins we used to rely on together have lost interest.
Finally if nobody is able to help, and the expectation is this simply continues as is for free indefinitely then that is totally fine also. The server is in no danger of being turned off or anything like that, requests and issues that come up will just take longer to be fixed as a result. To be clear I dont mind, if I cant use the lemmy server I just login to it and fix it and then carry on my merry way. This mostly affects people who cant do that.
Just wanted to make an appeal for some admins to improve the areas we are weakest but also clarify nobody is under and obligation to do anything and the server remains free regardless. The weakest area is not enough humans helping by messaging and opening communication lines if there are issues. If you can type a message and send it to someone, use this server and want to make it better then you are qualified to help.
Any help is of course greatly appreciated. My commitment is that you will never be expected to do anything to ‘keep it going’ but I also sleep, work, and so on so this is why I am asking. Totally up to whatever anyone feels comfortable with. Zero obligations.
Thanks
-P
