Saturday, June 28, 2014

Network Outage

There's currently a network outage that's affecting not only our servers but the entire data center (as well as their website/customer service portal). They are aware of the problem and are keeping us updated through Twitter (see why having multiple channels of communication is a good thing?).

We will keep you informed as we receive information. Hopefully this will be a short-lived outage (*crosses fingers*). It's extremely rare that the network goes down, but it does happen.

Update 4:28pm: Still no further word, but we're monitoring the situation and have requested an update from the NOC.

Update 5:02pm: This from the data center (which admittedly doesn't tell us much at all):
"All hands are on deck working to fix this problem. We will share all details when the engineers emerge. Thanks for your patience."

Update 5:35pm: Annnnd we're back UP! Thank you again for your patience. Had been anything we could have done to speed up the fixing we would have.

 Yesssssss, thank you, Internetz!

Update 6:24pm: I spoke too soon. The NOC engineers are working to fix the problem. During this time services will be on and off (up and down).

From the data center:
"Reserve troops are heading into our managed services team HQ so we can answer your questions as soon as normal communication is restored."

All communication into the data center is down (their website, ticket system, phone lines, etc), and I'm sure they're doing the best they can, and will update us as to what's going on as soon as possible. Preferably after they've fixed the issue.

Update 8:31pm: The servers as well as all services are back up. This time the data center's own website is back up also, which is a very good sign (it's been down for the past 5 hours). Hopefully all services will STAY up this time.

This was an extremely unusual event, as I've said before; something very catastrophic must have occurred. As soon as we know what happened, we'll certainly pass it along, if for no other reason than to satisfy all of our curiosity. And again, we do apologize for this disruption.

Tuesday, June 24, 2014

Emergency Maintenance on Apollo Hosting Server

A major security flaw has been discovered in critical vendor-supplied software that helps power our  servers' infrastructure. Emergency maintenance was done to patch the server (the kernel was upgraded), which required a server reboot and resulted in 3-10 minutes of downtime.

We apologize for this disruption in service, even I wasn't notified ahead of time, but in the interest of security it was necessary. Services are coming back up now as I write this. Thank you for your patience.