On Wednesday, Feb. 10, there was an outage at our hosting service provider, which caused a downtime of several hours. I have not detected any data loss, thankfully, and apparently one reason for the lengthy downtime was the filesystem checking of each server to verify to that all of the data was intact. My deepest apologies to everyone affected by the downtime and for the inconvenience it has caused. Here is the statement from the hosting provider.
On February 10, 2:14pm CST, WiredTree experienced a brief, one-second outage at our data center located in CoreSite at 427 S. LaSalle in downtown Chicago. This issue effected one segment of our datacenter space and caused any servers effected to reboot.
The event occurred during a scheduled, annual UPS [Uninterrupted Power Supply] maintenance process that required the UPS to be transferred from normal operation to “maintenance bypass.” Preliminary indications show the main switchboard supplying power to the critical load detected an “under- frequency condition” alarm, which resulted in the ATS (automatic transfer switch) transferring back to utility power and a one-second interruption of critical power. The cause of the “under frequency condition” has been identified as a failed generator.
The generator in question has been removed for immediate, emergency maintenance. As a result, emergency power is being provided by an alternate generator at this time. All scheduled UPS maintenance has been halted until the emergency generator in question can be repaired and fully-tested, estimated by close of business tomorrow.
We are deeply sorry for any inconvenience this has caused. Server uptime is as important to us as it is to you and you can rest assured that we will take every precaution to prevent this situation from happening again.