this post was submitted on 28 May 2024
5 points (85.7% liked)

Myserv One Official and only lemmy community.

116 readers
1 users here now

Any issues will be posted here or you can ask for help.

If the post is off topic or just a general post please label it such:

[Chat] YourPostTitle

General chat allowed if not spam.

founded 1 year ago
MODERATORS
 

Hi there,

Unfortunately the site was down for about 2 days over the 26-27 May this month.

This was due to the datacentre we use being affected by a storm/tornado: (https://apnews.com/article/tornadoes-weather-texas-oklahoma-93001e1d81f120d0d55c8dfa24938250) which took out power to the facility for just under 2 days. At no time was there any risk of loss to the server due to off site backups, however we did have to wait for the centre to come back online.

While this server is run extremely cheaply, at just $20 USD per month (approx) which includes paying for the compute, domain, email, and off site backup as well as object storage, it does mean that there is not redundancy when a site wide outage occurs. The low cost means that there is no risk of this server ever being unable to meet its costs and be paid for, but it does mean we sometimes can have an outage if a major event happens such as what happened and a delay before being brought back online.

The decision for this was because this server is designed to help reduce load on the rest of the fediverse and does not store much local content so a couple days a year the server is out was deemed acceptable during the design phase.

If you have any questions or comments feel free to post and I will try my best to answer them :)

  • P
top 6 comments
sorted by: hot top controversial new old
[–] [email protected] 2 points 5 months ago (2 children)

Thanks for keeping us posted!

Looking at Lemmy posts right now, it seems like many servers were affected - I see a lot of beehaw posts and not a lot else at the moment. I'm curious how big of an impact this was.

[–] [email protected] 3 points 5 months ago

Thank you, I am busy looking at the federation issue right now and hope to resolve it soon. This instance is just a single server so if its turned off then the impact is total, but in terms of the datacentre the entire site was down and power was slowly restored to parts of it and is still ongoing, we are luckily back online but some parts are still down and damaged. The site is 1515 round table drive Dallas TX.

I also purged the post you reported.

[–] [email protected] 2 points 5 months ago (1 children)

Federation is returning to normal and should be working again fully by tomorrow. There is some catch up to do.

[–] [email protected] 2 points 5 months ago (2 children)

Good news! Thank you for managing the server!

[–] [email protected] 3 points 5 months ago

Just to let you have an update, we are caught up with all instances except lemmy.world which is still 3 days behind. You can monitor or track the federation progress here: https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=lemmy.myserv.one&from=now-12h&to=now

[–] [email protected] 2 points 5 months ago

All good, sorry for the downtime :)