this post was submitted on 10 Jul 2023
295 points (99.3% liked)

FREEMEDIAHECKYEAH

1 readers
1 users here now

๐Ÿฟ ๐Ÿ“บ ๐ŸŽต ๐ŸŽฎ ๐Ÿ“— ๐Ÿ“ฑ


๐Ÿดโ€โ˜ ๏ธ Wiki / ๐Ÿ’ฌ Chat


Rules

1. Please be kind and helpful to one another.

2. No racism, sexism, ableism, homophobia, transphobia, spam.

3. Linking to piracy sites is fine, but please keep links directly to pirated content in DMs.

founded 1 year ago
MODERATORS
 

This post explains the incident well but long story short some hackers were able to compromise user and admin accounts through stolen authentication cookies on some instances.

Before things were clear on exactly how this happened, we pulled the plug on our instance to mitigate the risk. We probabaly should have hastily wrote an announcment post before doing that but the situation seemed critical so we didn't want to waste any time.

Few hours later, people were able to figure out the issue and promptly fix it. Turns out this vulnerabilty could only be exploited if an instance had custom emojis which thankfully ours didn't, so users using this instance should be safe from the hack. lemmy.fmhy.ml now runs on v18.2rc which has fixed this vuln to be extra secure.

Sorry for the downtime and we will try to communicate the problem better in the future.

P.S. After somone mentioned exploding-heads on a recent post and why we are still federated with it, we took some time to view it carefully and decided it's an instance that systematically breaks our rules and to defederate with it. We will shortly post our defederation policy soon to give a better idea on how we will decide on which instance to defederate from moving forward.

top 28 comments
sorted by: hot top controversial new old
[โ€“] [email protected] 60 points 1 year ago (4 children)

I also want to mention this happened at 2 or 3 AM EST, which only had 2 admins on, me and the hoster. I should've made a post, but given the severity, lack of info, and lack of staff members, we both decided to shut it down immediately until it was patched. I personally apologize for the downtime but I hope it's understandable why we did it this way. In case something like this happens again, I'll post it in the FMHY Divolt server.

[โ€“] [email protected] 38 points 1 year ago

Sometimes you unfortunately have to choose between safety and communication. In this case you chose safety, which even if it didn't end up being warranted was the right call IMO. Thanks for putting in all the effort.

[โ€“] [email protected] 28 points 1 year ago

Not only understandable, but I'm glad that's how you guys handled it. Leaving it up seems irresponsible to me, although I'm not blaming any admins who did.

[โ€“] [email protected] 5 points 1 year ago

I'm glad you guys had the time to do it at all. Also shows this layered universe stuff is not that bad, so much harder to do harm.

[โ€“] [email protected] 3 points 1 year ago

Website going down was confusing but under the circumstances best course of action.

On the bright side apps still worked :)

Awesome to have the site back up and running.

[โ€“] [email protected] 40 points 1 year ago (2 children)

Sorry for the downtime and we will try to communicate the problem better in the future.

You did the right thing, going down immediately.

The solution for the announcement problem is to do it "out of band", as the communications engineers call it. That is: use a different medium. May be a webpage or even twitter or whatever. Ideally let everybody know your method in advance (make an announcement policy :-)) so the users know where to look for the info.

[โ€“] [email protected] 10 points 1 year ago

Yes this is exactly how to do it. Maybe have a mastodon account to post updates if the instance is down ?

[โ€“] [email protected] 5 points 1 year ago* (last edited 1 year ago)

A custom error ?

CODE 666 EVIL IS TRYING TO GAIN CONTROL !

Just kidding . A script kiddy is trying to hack us.

[โ€“] [email protected] 39 points 1 year ago (1 children)

Appreciate your proactive measures and quickly getting the server up again and patched. Also thanks to all of the admins for their hard work going into the server!

I support the decision regarding exploding heads.

Related.. Has fmhy also blocked Meta/threads? While I don't think we should be like Beehaw over here, there are certain places that deserve defederation and Meta/corporate interests are at the top of that list for me.

[โ€“] [email protected] 36 points 1 year ago* (last edited 1 year ago)

Right now it's not even mastodon compatible let alone lemmy. There are some arguments on how federating with them will allow people to migrate to a more privacy respecting instance and still view threads content, and some users say this will allow them to still communicate with their friends who don't want to switch away from threads.

So while we do lean towards defederating from it, it's some months away before we need to actually decide and till then we are simply listening to and discussing both sides of the argument.

[โ€“] [email protected] 18 points 1 year ago

thank you for being proactive and also admitting you can improve for future responses... awesome!

[โ€“] [email protected] 12 points 1 year ago

I'm glad you guys acted swiftly. Security is important.

[โ€“] [email protected] 11 points 1 year ago

Thanks for doing your guys part.

[โ€“] [email protected] 9 points 1 year ago

I remember phpBB having a popular emoticon addon, or maybe arcade addon, that ended up compromising a few of the message boards I frequented in the early 00s. Times never change.

[โ€“] [email protected] 9 points 1 year ago

Perhaps a mastodon account or something with a quick "yeah it's down we're working on it" so we don't think the admin team have been captured by rogue elements and have been forced to work against their will to smuggle radioactive material ๐Ÿ˜‰

Seriously nice work getting it sorted!

[โ€“] [email protected] 7 points 1 year ago

Appreciate the hard work!

[โ€“] [email protected] 7 points 1 year ago

Thank you for making quick moves to protect us!

[โ€“] [email protected] 7 points 1 year ago

it's good you acted swiftly! Thank you for the update

[โ€“] [email protected] 5 points 1 year ago

Thanks for the update.

+1 for a backup Mastodon account ๐Ÿ™‚

[โ€“] [email protected] 4 points 1 year ago

Good thing you are back guys.

[โ€“] [email protected] 3 points 1 year ago

Hey, compared to vlemmy this was nothing.

[โ€“] [email protected] 3 points 1 year ago

Appreciate all the hard work! And I'm sure as things go on, things will get easier to manage

[โ€“] [email protected] 3 points 1 year ago* (last edited 1 year ago)

Nice to know what happened. Quick action with no communication is probably better when security is on the line. I was internally debating whether it was a hosting issue or ya'll had been targeted by some government agency. Speaking of, have you thought about putting up a warrant canary?

[โ€“] [email protected] 3 points 1 year ago

I can upload images in img section of the post but I can't upload them in comments, "private" messages and bodies of the posts, is this related to this exploit and it was turned off on purpose or is it an unrelated issue?

[โ€“] [email protected] 2 points 1 year ago

so what security concerns should we be worried about? I didnt have my computer on or logged in, am I at risk of a virus or personal information being leaked or something?

load more comments
view more: next โ€บ