As always, the transparency is appreciated. Some growing pains are certainly to be expected
this post was submitted on 24 Aug 2023
1087 points (97.4% liked)
Lemmy.World Announcements
29030 readers
9 users here now
This Community is intended for posts about the Lemmy.world server by the admins.
Follow us for server news π
Outages π₯
https://status.lemmy.world
For support with issues at Lemmy.world, go to the Lemmy.world Support community.
Support e-mail
Any support requests are best sent to [email protected] e-mail.
Report contact
- DM https://lemmy.world/u/lwreport
- Email [email protected] (PGP Supported)
Donations π
If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.
If you can, please use / switch to Ko-Fi, it has the lowest fees for us
Join the team
founded 1 year ago
MODERATORS
Whenever I get frustrated by the outages I remind myself: still better than reddit.
Thanks to lemmy I don't doom scroll anymore because either there is an outage or I read all the new content I'm subscribed to in 10-20 minutes
That's mostly how I redditted for years. It was mostly for those moments in between things, on the toilet, laying in bed at night. Not something I did for long periods of time.
Look at this guy with his reasonable social media use. YOU THINK YOU'RE BETTER THAN ME??
Scrolling a few miles or so a day isnβt too bad, is it?
Huh, I could keep up with my daily feeds before, now not so much.
I fallback to one of the many instances that federate with lw. Donβt get frustrated at all bro
Yup
Maybe its just the times I'm accessing but its seem better this week to me compared to the last few ones.
I don't blame you for this, but the uptime records are incomplete at best. I've experienced the site being down (and confirmed with Down for Everyone or Just Me), yet status.lemmy.world showed all systems operational. As I'm writing this, status.lemmy.world is missing most data up to yesterday and dash.lemmy.world shows 16 days uptime.
I have lots of respect to you for even having these. I also remember status.lemmy.world work mostly fine some time ago. But as of right now, both uptime monitors fail to serve their purpose.
You need to hover over the status bar to see if there is any down time for that day. We can enable it to log incidents every time there is a burp, but we are still tuning alerts as we only have it create a incident when we ACK it in PagerDuty. You can always check the dashboard for up to the minute stats, as well as https://lemmy-status.org/endpoints/_lemmy-world We'll add this info to make things clearer <3
EDIT: Added more info to our status page, thanks for the feedback Machefi!
EDIT2: Also the missing data is due to us removing and adding more specific monitors for the different infra services.
Excuse me stop being so cool, you're raising the bar too high for everyone else thank you
Good luck, no pressure.
On your Cloudflare account, if there was a change in the CNAME/A record being proxied vs. DNS only, that could cause an issue, as Cloudflare would then strip headers off the request that your Apache/Nginx would be looking for.
If you enabled HTTP DDoS protection in your Security -> WAF tab (I think thatβs where it is) that could do this too. Might be worth disabling.
Also check for any headers your HTTP load balancer might be expecting, that Cloudflare could be stripping.
Might be worth tailing the webserver logs to see what happens to requests coming in from Cloudflare.
Thanks for keeping us up to date, I don't mind downtime personally as long as I'm aware of what's going on
Everyone talking about the downtime including lemme and me just enjoying lemmy like never before. I've experienced no downtime so far
The beauty of a federated app is that I do not have to rely on lw or lemme. Use another instance and go on with my day π
load more comments
(2 replies)
So what you're saying is we were the ddosers all along πΉ
Only thing that needs to be said here is: thank you for the best Lemmy instance (in my opinion) out there!
Could look into Dacentec if you need more cheap servers. I use them for my stuff. YMMV since you're getting a hell of a lot more traffic than I am but they haven't blocked Cloudflare on mine yet so that's a plus. :)
Hetzner is a pretty good provider. It would be nice to stick with them if possible.
load more comments
(3 replies)
I brows all, never noticed as outage lol.
You guys rule. Thanks for all the hard work.
Is there anything we can do to help? Donations? Tech volunteers? Visit hosting company with a baseball bat?
I imagine spreading out across instances, and having accounts in instances beyond the biggest one helps, as it reduces traffic and strain on lemmy.world and its servers.
lemmy.world still deserves support, like with donations, though helping reduce the strain is something anyone can do, albeit small, though free of charge.
The transparency is appreciated.
Thanks for the transparency. I think you lost some trust with the Discord debacle, but this makes me feel a lot better about things.
So the recent reports about ddos attacks were false alarms?
This is about what's happening the last few days I think? Lemmy World has been a lot more stable compared to even a week ago but still has some daily downtime. But not for hours every day anymore
load more comments
(1 replies)
Thanks for keeping things running as Lemmy grows!
For what it's worth, I'm on all the time and have barely noticed the outages.
load more comments
(1 replies)
Thanks for the update!
Thanks for the updates and transparency.
The dashboard requires a login to view. At least from here it does. Is there a way to view without making an account?
load more comments
(2 replies)
Thanks for sharing the dash link, and for your work!
view more: next βΊ