38
submitted 4 months ago* (last edited 4 months ago) by [email protected] to c/[email protected]

Hey all! I've done a lot of database maintenance work lately and other things to make lemdro.id better. Wanted to give a quick update on what's up and ask for feedback.

For awhile, we were quite a ways behind lemmy.world federation (along with many other instances) due to a technical limitation in lemmy itself that is being worked on. I ended up writing a custom federation buffer that allowed us to process activities more consistently and am happy to say that we are fully caught up with LW and will not have that problem again!

Additionally, on the database side of things, I've setup barman in the cluster to allow for point of time backups. Basically, we can now restore the database to any arbitrary point in time. This is on top of periodic automatic backups which also gets pulled to storage both on my personal NAS as well as a Backblaze bucket (both encrypted of course).

Today, I deployed a new frontend at https://next.lemdro.id. This one is very early stages and experimental but is being developed by https://lemm.ee and seems promising!

If you live outside of the US and experience consistently long load times I want to hear from you! I am deploying the first read replica node to Europe soon, so if you live in that region you'll soon notice near-instaneous loading of content. Very exciting!

Finally, looking for feedback. Is there anything you want to see changed? Please let me know!

[-] [email protected] 23 points 4 months ago

I feel like Dune was a good outlier to this. It's the only movie I've seen in theater in the last few years and I really enjoyed not having everything explained to me

107
submitted 6 months ago by [email protected] to c/[email protected]

Google today announced a handful of wearable and navigation updates, starting with public transit directions in Google Maps for Wear OS.

70
submitted 6 months ago by [email protected] to c/[email protected]
20
submitted 6 months ago* (last edited 6 months ago) by [email protected] to c/[email protected]

I am rolling out the Photon UI as a replacement to the default lemmy UI right now. Initially, only about 50% of requests will be routed to Photon, determined by a hash of your IP address and user agent (sorry for any inconsistencies...). As I determine that this configuration is stable I will be slowly increasing the percentage until Photon is the new default frontend for lemdro.id.

If you have any difficulties, please reach out. Additionally, the "old" lemmy frontend will remain available at https://l.lemdro.id

Edit: I am aware of some problems with l.lemdro.id. It wasn't designed to run on a subdomain so I'll need to add a proxy layer to it to redirect requests. A task for tomorrow!

FINAL EDIT: https://l.lemdro.id is now fully operational, if you choose to use the old lemmy UI it is available there

9
submitted 6 months ago by [email protected] to c/[email protected]

Over the course of the last couple weeks, I managed to root cause and solve the problem causing stale sorting on lemdro.id. My apologies!

94
submitted 7 months ago by [email protected] to c/[email protected]

We typically like Pixel phones a lot, but we have some reservations about Google's quality control

61
submitted 7 months ago by [email protected] to c/[email protected]

Google Maps is changing with pretty significant redesigns across key surfaces, including when searching for directions...

[-] [email protected] 16 points 7 months ago

I don't understand why this exists. Polestar has enough problems lol

14
submitted 7 months ago* (last edited 7 months ago) by [email protected] to c/[email protected]

Edit: Upgrade went off without much of a hitch! Some little tasks left to do but we are now running 0.19.3. Seems like a lot of the federation issues have been solved by this too.

You will have to re-login and 2FA has been reset.

This update is scheduled to take place this weekend. No specific day or time because I am infamously bad at following those. I will try to minimize impact by keeping downtime to lower-traffic periods.

Ideally, there will be no downtime, but if there is it is likely to only last an hour maximum. During this time I will add an "under maintenance" page so you can understand what we are up to.

Feel free to join our Matrix space for more information and ongoing updates! My apologies for how long this took - I was in the middle of a big move and a new job.

Additionally, there may be small periods of increased latency or pictures not loading as I perform maintenance on both the backend database and pictrs server in preparation for this upgrade.

[-] [email protected] 40 points 8 months ago* (last edited 8 months ago)

Basically, the lemmy backend service for some reason marked every instance we federated with as inactive, which caused it to stop outbound federation with basically everyone. I have a few working theories on why, but not fully sure yet.

TL;DR lemmy bug, required manual database intervention to fix

This was a stressful start to a vacation!

For a more detailed working theory...

I've been doing a lot of infrastructure upgrades lately. Lemdro.id runs on a ton of containerized services that scale horizontally for each part of the stack globally and according to load. It's pretty cool. But my theory is that since the backend schedules inactive checking for 24 hours from when it starts that it simply was being restarted (upgraded) before it had a chance to check activity until it was too late.

theory:

  • scheduled task checks instances every 24 hours

  • I updated (restarted it) more than every 24 hours

  • it never had a chance to run the check

  • ???

  • failure

This isn't really a great design for inactivity checking and I'll be submitting a pull request to fix it.

2
Recent Downtime (lemdro.id)
submitted 9 months ago* (last edited 9 months ago) by [email protected] to c/[email protected]

There was a brief (~5 minute) period of downtime on lemdro.id recently. This occurred during a routine database upgrade when some strange issue caused a member of the cluster to become inconsistent and refuse to form quorum.

This locked the entire cluster into an invalid state which took some troubleshooting to fix. My apologies.

I will be rolling out read replicas for folks on the East coast of the US as well as those in Europe sometime in the next week, you should notice a pretty dramatic reduction in latency if you are from those areas. Additionally, other recent changes have increased reliability and decreased latency which may or may not be noticeable.

I will post another update before I start rolling out the read replicas since it is kind of a big change (and I will schedule a time for it)

1
submitted 9 months ago by [email protected] to c/[email protected]

I recently rolled out an infrastructure upgrade this last Monday (Dec 4) with the intent of reducing peak response times and removing occasional scaling errors.

Unfortunately, my metrics system showed slightly elevated error rates, so I've decided to rollback these changes for now. I will make another announcement before I roll them back out in the future. Thanks for your support!

[-] [email protected] 16 points 9 months ago

eBay doesn't have workers, it's a marketplace that facilitates transactions between sellers and buyers

13
submitted 10 months ago by [email protected] to c/[email protected]
[-] [email protected] 18 points 11 months ago

I've gotten an email like this before for lemdro.id. I think it's a generic phishing email since the community links look like email addresses (and actually often are)

[-] [email protected] 26 points 11 months ago* (last edited 11 months ago)

If you support changing the default frontend to Photon, upvote this comment! If you don't support it, downvote this comment!

27
submitted 11 months ago* (last edited 11 months ago) by [email protected] to c/[email protected]

Hey there everyone. I think the Photon project has matured enough to the point where I feel ready to replace the default lemmy frontend with it. Since this instance serves roughly 1000 people now, I figured this was worth holding a vote on!

Please check out Photon as currently hosted at https://p.lemdro.id.

If you support changing the default frontend to Photon, upvote my comment on this post. If you don't support it, downvote that same comment.

Thanks!

[-] [email protected] 17 points 1 year ago

wow, this one is relatable. So far I keep succeeding so I haven't been demotivated yet...

[-] [email protected] 19 points 1 year ago

Freesync yes, HDR soon™

[-] [email protected] 18 points 1 year ago

It is my opinion that there is only "tension" because you say there is. I do not believe that most people feel that way; it isn't exactly an active war zone. Nevertheless, I respect your decision as moderator of this community, and I'll leave you be in the future. Best of luck!

[-] [email protected] 16 points 1 year ago

I'm the backend guy at lemdro.id, I started the instance. I don't moderate the [email protected] community (or any of them... I purely do infrastructure), but I am quite aware of the drama around recent events involving the two android communities.

The intent behind banning all lemdro.id users from applying to be mods feels quite... retaliatory? I'm not sure there is anyone on lemdro.id who would apply, but depriving them the opportunity simply because they are associated with an instance that hosts a community which had another community merge into it?

It is your community, you can of course do what you want. I just wish we could drop the drama, act like adults, and move on. Tired of the vitriol.

[-] [email protected] 15 points 1 year ago

Hey everyone, this exploit was present in the custom emoji feature of Lemmy. Because lemdro.id does not use custom emojis, we are not vulnerable at this time.

view more: next ›

cole

joined 1 year ago
MODERATOR OF