I can't be bothered to walk down to the basement, so practically my server is also only accessible by SSH

I'm 150+km away from my server, with literally everything on it lol

Don't say stuff like that. You're gonna give me a heart attack.

I like how posting got fairly fast. Then we started putting absurd amounts of ram into servers so now they're back to slow.

Like we have a high clock speed dual 32 core AMD server with 1TB of ram that takes at least 5 minutes to do it's RAM check. So every time you need to reboot you're just sitting there twiddling your thumbs waiting anxiously.

Found the debian user!

Never touch a running system
Until you have a inviting hole in your system

Nevertheless, I'm panicking every time I update my sever infrastructure...

There's always backups... Right?

.... Right?

If you have the bandwidth... it is absolutely worth it to invest in a maintenance mode for your system, just check some flat file on disk for a flag before loading up a router or anything and then, if it's engaged, just send back a static html file with ye olde "under construction" picture.

There's a package called molly-guard which will check to see if you are connected via ssh when you try to shut it down. If you are, it will ask you for the hostname of the system to make sure you're shutting down the right one.

Very usefull program to just throw onto servers.

Oops.

Since you're using sudo, I suggest setting different passwords on production, remote, and personal systems. That way, you'll get a password error before a tired/distracted command executes in the wrong terminal.

Happens to everyone

Just having a multitude of terminals open with a mix of test environment and (just for comparison) an open connection to the production servers...

We were at a fair/exhibition once and on the first day people working on an actual customer project asked us, if they could compare with our code.
Obviously they flashed the wrong PLC and we were stuck dead at the first hours of the exhibition.
I still think that this place was cursed, as we also had to do multiple re-soldering of some connections of our robot and the sherry on top was the system flash dying - where I had fucked up, because I just finished everything late at night and didn't made a complete backup of everything.
But it seems, if luck runs out, you lose on all fronts.

At least I was able to restore everything in 20mins. Which must be some kind of record.
But I was shaking so much from the stress, that I couldn't efficiently type anymore and was lucky to have a colleague to just calmly enter what I told him to and with that we're able to get the show case up and running again.

Well, at least the beer afterwards tasted like the liquid of the gods

I was making after hours config changes on a pair of mostly-but-not-entirely redundant Cisco L3 switches which basically controlled the entire network at that location. While updating the running configs I mixed up which ssh session was which switch and accidentally gave both switches the same IP address, and before I noticed the error I copied the running config to the startup config.

Due to other limitations and the fact that these changes were to fix DNS issues (and therefore I couldn't rely on DNS to save me) I ended up keeping sshing in by IP until I got the right switch and trying to make the change before my session died due to dropped packets from the mucked up network situation I had created. That easily added a couple of hours of cleanup to the maintainence I was doing

Best thing I did was change my shell prompt so I can easily tell when it isn't my machine

Make an alias for Ehen you type shutdown it dies restart and if you want to shutdown make an alias that goes like

Yesireallywanttoshutdown

Networking, we had a remote office in Europe (I'm in the US) and wanted to reset a phone. Phone was on port 10 of the Cisco switch, port 1 went to the firewall (not my design, already in place).

Helping my coworker, I tell her to shut port 10.

Shut port 1, enter.

Ok... office is offline and on another continent...

i have the horrible habit of using shutdown now because of my personal computers. a lot more fun.

I install molly-guard on important machines for this reason. So fast to do a reboot on the wrong ssh session

If a tree falls in the woods...