This week has been pretty hectic at $work. On Tuesday at 1545 (3:45pm) our director came into the bullpin and announced that we had lost power to the server room and the UPS would die in approximately 10 minutes.
The database administrators immediately tried to log into every database server to shutdown the databases. Needless to say they did not get everything and every server crashed hard. Luckily the new flexpod is being hosted offsite since we have some cooling issues to fix. Most of us were at work until midnight with some staying until 0400.
The server administrators were not affected since we were moved out of the building that contained the server room about three years ago. We found the root cause was some construction work on another floor of the building.
One of the workers had caused an electrical short. Instead of tripping the breaker on the floor where work was being done, the main building breaker tripped. The generator has plenty of time to fire up and take the load. However, it was discovered that the bypass had been tripped the last time our contractors inspected the generator. And we went dark….