this post was submitted on 24 Dec 2023
7 points (100.0% liked)

Announcements

242 readers
1 users here now

Announcements about system updates or other things related to this instance.

founded 1 year ago
MODERATORS
 

As you may have read in other threads, this version of Lemmy (0.19.1) seems to have bugs in outgoing federation on some instances.

As a temporary fix, we have added a scheduled restart of Lemmy every hour. It only takes a few seconds to restart, and the big advantage is that your comments and posts are only delayed up to 1 hour before they federate to other instances. You probably wont notice the restart even.

This will be in effect until a bug fix arrives from Lemmy developers, probably after new years sometime.

Thanks for reading and merry x-mas to everyone. :)

top 19 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 1 year ago (1 children)

Thanks for the update Manager! Happy holidays!

[–] [email protected] 4 points 1 year ago* (last edited 1 year ago)

Same to you Lewdiculous :)

[–] [email protected] 1 points 1 year ago (1 children)

I'm still having problems. None of my comments or posts go outbound. But thank you for working on the problem!

[–] [email protected] 3 points 1 year ago* (last edited 1 year ago) (2 children)

Yeah its really annoying and makes it hard to have any discussions at all.

I will try now to restart only the lemmy docker container and see how it seems to work out. If the restart seems to work reliably many times in a row, I can schedule it to happen automatically.

Not giving up on making this work somehow. :)

Edit: Tried a restart of only the lemmy container and it does give like 20 seconds of downtime so not ideal. But maybe worth doing a couple of times per day at least.

[–] [email protected] 1 points 1 year ago
[–] [email protected] 1 points 1 year ago (1 children)

I can't vote, comment, subscribe, or post anywhere at lemmynsfw. Is because it's defederated here? Or is the cause because of the federation problems you've been seeing?

[–] [email protected] 3 points 1 year ago

Hi,

It's because of the very annoying outgoing federation issues we have now on Lemmy 0.19.1.

All incoming data works very well but outgoing doesn't get sent until a restart of the Lemmy software, and then only once after restart. We will do a quick restart a couple of times per day and then your comments, posts etc will federate but of course it's really annoying with that long time in between your activity and others seeing it.

Hopefully there is a fix from Lemmy developers after new years. It has a bad idea to upgrade Lemmy before Xmas, and lesson has been very learned, believe me. :)

[–] [email protected] 0 points 1 year ago* (last edited 1 year ago) (1 children)

Just this day caught lemmy.today down twice, and first time it was super very dead for much longer than a few seconds or minutes.

Wonder if others had the same experience.

But first, enjoy Christmas holidays! All that stuff can be sorted out later, if there even is a way to fix it on the server admin side.

Thanks for keeping us informed!

[–] [email protected] 1 points 1 year ago (2 children)

Yes, it seems the restart is not as reliable as I was hoping for.

Its a docker compose restart of the entire stack but that has proven to work only sometimes, which is not good enough. This afternoon it failed once (as you noticed), then worked 4 times, then failed once more.

We will figure out a more reliable way to restart, but for now, the restart has been removed again.

Really sorry about all this. Just trying to find a way to make federation work again since its very frustrating when it doesnt.

Anyway, merry Christmas and thanks for being understanding during all this.

[–] [email protected] 3 points 1 year ago (1 children)

We must have all been very naughty this year for Santa to bring us this heap of bugs. :/

[–] [email protected] 2 points 1 year ago (1 children)
[–] [email protected] 1 points 1 year ago (2 children)

Sadly, even with the scheduled restarts it seems like only about half of my posts and comments make it out to other instances.

[–] [email protected] 3 points 1 year ago (1 children)

We had to remove the scheduled restarts since they didn't work reliably and brought the server down. So they are not in effect right now. We do manual restarts a few times per day only.

Still hoping for a fix from Lemmy devs soon...

[–] [email protected] 3 points 1 year ago (1 children)

Well, at least it appears that they are aware of the problem, and one guy said he may have found a fix for it. There is a pull request open that might fix it, so hopefully, this'll be over soon.

[–] [email protected] 2 points 1 year ago

That sounds awesome. I really hope it fixes the problem, this is so frustrating. Thanks for the links!

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago) (1 children)

As far as I can tell, not one of my comments has made it out in the last 4 days. No replies, no votes, not visible on any instance outside of lemmy.today.

Such is life on the bleeding edge...

Edit: seems like at least some got pushed out, as I'm finally getting responses again.

[–] [email protected] 2 points 1 year ago

Everything is pushed out but only when we restart the server... Hopefully a fix for that soon!

[–] [email protected] 3 points 1 year ago (1 children)

For what it's worth, I set up a cronjob on my instance to restart just the lemmy container every 6 hours. It looks like 0 */6 * * * docker container restart lemmy-lemmy-1. Federation has been pretty good so far, but it has only been in practice for less than a day.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

Yeah I could try to restart only the Lemmy container instead of the entire stack. Worth trying, thank you. :)

I tried it and it does give 20 seconds of downtime but could be worth doing a couple of times per day anyway. Its so frustrating with no outgoing messages...