INC-224, never forget.

I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.

15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.

15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.

I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.

We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.

We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D

Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new "current" version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.

Girl did dabs on break with her gf came back zonked out since she'd never smoked weed before.

Ended up slapping manager and getting taken away by ems

Cook got arrested at work one time when cops came to pick her up at her job. She was 4 feet tall so we joked they picked her up and carried her away. She had to use a step stool to make the soup and someone would hide the stool from her so she'd be pissed the next morning.

Same place had a cook drinking lean and offering it to people.

Retirement home btw

Previous HR was well beyond retirement age essentially working to have something to do and one day emailed all of management a spreadsheet asking us to verify our information. That sheet contained each of our full names, addresses, phone numbers, birth dates, social security number, etc.

To my knowledge nothing of significance happened. I have my credit frozen.

Guy found a gun in the customer’s stuff

Guy starting waving it around and playing with it, pulled the fuckin trigger, almost shot one of his coworkers

Cops came, guy said he was moving a cabinet and it went off which obviously no one believed, somehow he wasn’t arrested, idk

Guy was fired over the phone before he left the customer’s house

The overnight IT guy was caught watching porn while working (this was over a decade ago, he was in the office every night and not a remote worker). How was he caught? He was saving the pornographic photos on a shared network drive...

When confronted, he didn't try to deny anything, his explanation was simply, "That's just my thing."

Worked at a place where our CIO was completely unqualified to be a leader, much less a leader in IT. She was a micromanager who took the position of "telling stakeholders" instead of "working with stakeholders" so any project she was on was really her pushing through whatever agenda she had at the time. Meanwhile her deputy CIO was stealing computer equipment from the server room but I digress....

April fools one year and I decide to prank it up. I moved the hinges (not the door handles) of the freezer/fridge in the breakroom so that the handle and hinges were on the same side. It's a fifteen minute job to move everything so I did it the night before the 1st.

The next morning our hungover CIO stumbles into the breakroom and cannot get the fridge to open. After a few seconds of futile tugging on the handle, she gave up and took her lunch to her office.

Others in the office figured it out pretty quickly and had a good chuckle.

Later on that day CIO sends out a nastygram about pranks being unprofessional, property damage, someone was going to be in huge trouble, yadda yadda....

But she's not the director. The director tells her to basically fuck off, it was a funny prank, and perhaps she needed to lighten up.

She never found out it was me.