PSA from Admin Team: The update completed roughly two hours ago. Since that time, the Admin team (and other site admins) have been working on the noted performance issues. We believe we have found a solution, but we still need time to test this out. You may still see brief outages and differences in performance as we are testing different configurations. We are trying to prevent rolling back.

While I know this can be frustrating - especially today - please keep in mind we have a team of volunteer techies (from around the globe!) collaborating on this issue. It is an inspiring situation. Also keep in mind that lemmy.world is quite a bit larger (and more active than any other instance). As such, we are a bit of a ‘test instance’ in regards to high volume requests. This is just part of the growing pain. We appreciate your understanding.

@[email protected] will provide a debrief once we have completed testing.

Server feels a lot better now than it did an hour ago. Comments are going through quickly and upvotes are working for me.

Performance is extremely uneven. Sometimes loads instantly, sometimes I get a timeout. Upvotes don't show up until reload. Still a lot better than the last attempt.

The server hamsters seem to be on fire, but I won't hold it against the service/community/individuals (except spez; fuck you spez). Lemmy is in the unfortunate position of being forced into a development model called "fuck it, we'll do it live!"

edit: commenting and editing seems to be fine.

Heyo! Small update from someone who is watching the upgrade live; it's ongoing still. Seems like they are still facing some performance issues. So grab your popcorn and wait! 🍿 (Btw, i am not a sysadmin, just a moderator)

TFW you run a small server with an open source social platform for shits any giggles and one month it explodes a thousandfold because of one greedy pig boy.

looking good so far, the instance is back, testing to post!

Posting and seems fine from Jerboa, can't log in using wefwef.

Just a heads-up. I ended up having to create a new login at lemm.ee because even after the improvements in speed and so on with the upgrade, for all intents and purposes it looks like my account here may have been wiped. Can't login from anywhere, getting "Record not found" whenever I try to sign in on Connect, "incorrect login" from Jerboa, and a spinning button and then nothing on a PC browser, basically treating my account as if it doesn't exist. Some folks appear to be able to login though. Is this an ongoing server issue, or did my account actually get wiped? Same username (Grangle1) as on lemm.ee.

Good luck.

Finally able to register. Glad to be here.

And… we’re back!

Thank you for the heads-up. 🤙

Good luck! Posted 19:59 CET.

My lemmy server is also running the same versions are you are now. I was getting timeouts with lemmy.world federation so I recompiled lemmy_server with timeout changes:

diff --git a/crates/utils/src/lib.rs b/crates/utils/src/lib.rs
index e5d07db2c..e2c592d82 100644
***
a/crates/utils/src/lib.rs
+++ b/crates/utils/src/lib.rs
@@ -18,7 +18,7 @@ use std::time::Duration;

 pub type ConnectionId = usize;

-pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub const REQWEST_TIMEOUT: Duration = Duration::from_secs(13);

 #[macro_export]
 macro_rules! location_info {
diff --git a/src/lib.rs b/src/lib.rs
index cc77ca48f..45c621a7c 100644
***
a/src/lib.rs
+++ b/src/lib.rs
@@ -37,7 +37,7 @@ use tracing_subscriber::{filter::Targets, layer::SubscriberExt, Layer, Registry}
 use url::Url;

 /// Max timeout for http requests
-pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(10);
+pub(crate) const REQWEST_TIMEOUT: Duration = Duration::from_secs(16);

 /// Placing the main function in lib.rs allows other crates to import it and embed Lemmy
 pub async fn start_lemmy_server() -> Result<(), LemmyError> {

Wow, this is much faster now that caches are built.

Is the update done? I hope not because can't i upvote😅

You know, maybe I'm being overly dramatic, but I really feel like we're at the beginning of something special. I realize that Lemmy has existed for some years, but the whole Reddit thing is really pushing it into the limelight, and it's new to me (I joined just before the blackout).

Seeing all the developers and admins dealing with the inevitable bugs, working together to keep the federated instances operating together during a huge surge in users, like a flotilla of disparate ships lashed together facing a major storm - it makes me excited, nervous, and hopeful. I fully intend to support developers and instances financially, but also want to give a big "yay team" shout-out.

looking good so far, the instance is back, testing to post!

Seems I can't upvote

Edit. I can upvote, but with a lot of delay!

Tried to logout, website on mobile won't release my session, and when I tried to login on jerboa it said password incorrect, not sure if that's a ui issue or a lemmy.world issue though

Nice work with everything so far, really liking it. What all is being changed/fixed in the new revision? Is it just more efficient code to support higher site traffic?

The last few minutes the site seems to be very fast! Let's hope it stays this way!

Just looked. First 1/2 loads were slow but after that it's lighting fast! I think by not everyone establishing a Websocket connection and just loading once performance should increase a tad bit.