Sysadmin

5574 readers

1 users here now

A community dedicated to the profession of IT Systems Administration

founded 5 years ago

MODERATORS

[email protected]

113

TIL in 1997, a Reply All storm took down all of Microsoft's internal Exchange system (techy.news)

submitted 1 year ago by [email protected] to c/[email protected]

15 comments fedilink hide all child comments

It started off with an employee sending an email to a distribution list called "Bedlam DL3" asking to be taken off the list. With 13,000 recipients and everyone replying all with, "Me too!" and other messages, it was estimated that over 15 million messages were sent through the system in an hour. This crashed the MTA service due to a recipient limit. Each time the MTA service recovered, it would attempt to resend the message again which lead to a crash loop.

As a result of the incident, the Exchange team introduced message recipient limits and distribution list restrictions to Exchange, which is something we all use today!

More on the story here: https://techcommunity.microsoft.com/t5/exchange-team-blog/me-too/ba-p/610643

cross-posted from: https://techy.news/post/2224

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 6 points 1 year ago* (last edited 1 year ago)

Back when I was in university, I worked IT support there on the side. One day, a teacher wanted to send a mail to one project group, but accidentally send it to the whole university. Every student, every employee. We didn't reach 13k people but it was a few thousand.

The thing is, in Outlook (which was used for school mail) the default reply button which is looking simply like arrow, was the reply-to-all one. Reply to sender was hidden a few clicks away. Needless to say, this caused similar issues. With the first people just politely trying to tell said teacher he might have maken a mistake, then people went in replying asking people to stop using reply-to-all, and it didn't take long for hell to break loose after that.

To make matters worse, a few smartasses ran some scripts putting the whole receiver list on all kinds of spam advertisement lists, causing a flood of spam send to everyone simultanously with all the reply-to-all-replies. And then people replied to those too. Guess they figured they wouldn't get caught with everyone receiving mails from everywhere. They did tho, and got seriously reprimanded.

The server automatically changed from instant delivery to synchronising every 5 min, but that still meant hundreds of mails every 5 min. Eventually we had to turn off the mail server to make it stop as trying to tell that many people to stop replying is impossible and it clearly wasn't going to die out on it's own.

It was a long day at work, and one I will likely never forget. But I feel like any bigger sized company that has excisted for some longer time has had their own version of this issue by now. I never understood why'd they make reply-to-all the default, instead of reply-to-sender with the to-all version as a smaller button next to it... At least they now added the warning in Outlook "your distribution group has X amount of people, are you sure" or something along those lines when sending to distribution groups of a few dozen or more...