You know the song that goes, "Rainy days and Mondays always get me down..." by The Carpenters? Well, it makes a perfect theme for this true IT blooper, in which a rainy Friday turned Jurriaan van Doornik's Monday into a classic Dilbert type of day.
It was Monday morning, 7:30 a.m. Network administrator Jurriaan van Doornik arrived at the office early to change the automatic backup media and run a manual backup of some mission-critical systems.
Walking down the hallway on his floor, Doornik noticed that it felt uncomfortably stuffy and warm. The closer he got to the server room the worse it felt. This is rarely a good sign.
He cautiously opened the door, steeling himself for the worst, when he was hit by a wall of hot air worthy of a steamy day in the Amazon. "It had to have been somewhere between 40 and 50 degrees Celsius in there," Doornik remembered.
Now we all know how important climate control is in a server room. So, after the servers spent a whole weekend in a sweat box, Doornik was very concerned about a hardware meltdown.
"I immediately did a quick check of all the systems," he said. "The main network switch was crying about overheating, but luckily it hadn't shut itself down yet, and there didn't seem to be any damage to our 18 servers; they were still running -- although extremely warm to the touch."
Amazed, Doornik set out to get the room cooled down quickly. Unfortunately, it was a closed-in office. The windows didn't open and the door faced an interior hallway; the heat had nowhere to go. He went on a mad hunt for table fans and set them up to push the hot air out of the server room while he worked on getting a replacement A/C unit from a rental company.
Somehow he managed to get things under control. But he still had no idea what had happened. Did the janitors forget to turn the A/C back on after a secret weekend sauna session? Or had one of the interns conducted an unsanctioned experiment to find out if it is possible to roast marshmallows over a file server?
When the repairman arrived, he quickly discovered the problem: the air conditioner's power-surge protection switch died during a Friday afternoon thunderstorm. Because it wasn't a self-correcting switch, the A/C never came back on.
The obvious lesson from the Blooper Vault: Assume the worst! If your server room A/C doesn't have a self-correcting switch on its surge protector and a backup power source in case of emergency, go out and get them. Or at least be well-stocked on marshmallows.
