Network Outage Resolved

There was a network outage early this morning, starting at roughly 6am PDT, lasting approximately an hour, that has been resolved. This was a wide-reaching network issue, affecting the data center's entire network, even their own website was down and phone lines over-loaded, which in turn affected our servers. The official explanation for what happened:

Now that the dust has settled, we have had a chance to look more closely at what happened this morning. We had one of our core routers experience a still unknown hardware error. This error caused the router to reboot in a semi-functional state. It had enough configuration remaining that it caused our other core routers to decide that they did not need to assume authority, and remove the problem router from the configuration.

Once this was determined, it was fairly simple for our engineers to remove the problem router from the network topology, allowing the other routers to take over. At that point, most of the network came back online.

We had a group of edge switches that remained confused by the port problems from the primary router and refused to fail over their RSTP configurations to the other routers. These switches required manual intervention in order to bring them back online.

Neither of these problems was expected and we are still determining why they occurred. We will be making whatever changes are necessary to our network to ensure that our redundancy and failover configuration works optimally in the future. Additionally, we will be doing a full hardware diagnosis on the original problem router to determine exactly what hardware needs to be replaced.

I apologize for the inconvenience that this has caused, and wish to reassure everyone that we will address all identified problems and build an even stronger network.

Throughout the issue the data center's spokespeople posted updates to Twitter, Facebook, and even WebHostingTalk's forums. This kind of transparency by a NOC is admirable and rare (which is why we try ourselves to emulate it, and provide as much information as possible to our own customers in as many venues as possible). I've leased servers from other companies who felt several hours' of downtime didn't require an explanation to anyone.

They called in every engineer they have to fix this issue in a timely manner. This is the first network affecting outage they've had since 2004, which doesn't make the situation today acceptable but it does put it in a bit of context.

We chose this particular data center as our server provider because of their commitment to reliability, redundancy and expedient customer service. If anything this morning's incident has solidified that view, to have an issue of this magnitude resolved in a little over an hour is a testament to their dedication. In this industry especially, a company is judged best by how they react in a worst case scenario.

Word Press Update

The following update is available through Fantastico:

* WordPress: 2.8.5 -> 2.8.6

Get'um while they're hot

The following application updates are now available from Fantastico:

* Crafty Syntax Live Help: 2.15.0 -> 2.16.3
* Joomla 1.5: 1.5.14 -> 1.5.15
* Moodle: 1.9.4 -> 1.9.6
* Open-Realty: 2.5.7 -> 2.5.8
* TikiWiki: 3.2 -> 3.3
* TYPO3: 4.2.8 -> 4.2.10
* WordPress: 2.8.4 -> 2.8.5

Plain or glazed?

Email is like donuts, and comes in a variety of flavors. I've noticed more and more companies adding more "colored sprinkles" to their email. To be honest, I was a slow convert, having been weened on the concept of plain text email. In recent years email clients have gotten better at displaying HTML email, and companies have gotten better at sending better quality HTML.

When a client recently mentioned how bad and confusing the billing system's emails look when viewed in plain text format, I started investigating how to send the system notifications and invoices in both formats, as we always like to provide an option.

At this time our billing and support system offers no way to specify sending email in either HTML format or plain text format on a per client basis, only on a per type of email basis (i.e. all due invoice notices, all support replies, etc). As soon as this functionality is added to the system we use, we'll be more than happy to make this choice available to clients.

The overwhelming number of email users today prefer to send and receive HTML email. Having my roots in "old sk00l security", I do understand the security implications of accepting HTML email from unknown recipients. Though I'd never want to promote a lax attitude towards email security, the anti-virus and spam filtering combination we offer on the server is so powerful and effective (as long as you haven't changed the default settings), I can't remember the last time anything bad made it's way through to my local computer. The personal anti-virus you should be running on your own computer is a good second line of defense, though the best defense of all is basic common sense.

If you use Mozilla Thunderbird as an email client, there is a happy medium option called Simple HTML.

View -> Message Body As -> Simple HTML.

Simple HTML will only interpret the most basic HTML commands, disables JavaScript, and doesn't display remote images. This option strips out the fancy font colors and such, yet retains basic formatting without displaying the underlying HTML code. You can use the "Simple HTML" option on the fly...leaving your settings to display email in plain text and only switching to Simple HTML as needed.

The other option would be to login to your Client Portal account, and view the email there. A copy of every email the system has sent to you in the past six (6) months is kept there for your review in the Client Portal.

Routine scheduled server maintenance

On Wednesday, November 4, 2009 between 12:00am and 6:00am US Eastern Time, we will perform routine maintenance on the server. A reboot of the server will be required. We do not expect the downtime to exceed 30 minutes.

Unplanned downtime

The NOC/Data Center had to reboot the server unexpectedly, due to an issue with the back-end. They reported they do not anticipate the downtime to last past 30 minutes, and that was 1.25 hours ago.

I'll keep on them until this issue is resolved. Thank you for your patience.

EDIT: And we're back up. Apparently there was a problem with quotas and the filesystem quotas had to all be recalculated before the machine could be brought back up and into service.

Again, we apologize for the inconvenience, we had no advance knowledge this would happen or we would have notified all users.

Perl problems

During the latest CentOS upgrade (from 4.7 -> 4.8) there was a bug introduced that affected some cPanel servers, specifically the Perl packages and version. The installed version of Perl should be 5.8.8, which it has been (and has been required to run cPanel 11.24 since late 2007). For whatever reason, the CentOS upgrade downgraded Perl to 5.8.5.

Since cPanel and a majority of the software on the server rely on the correct Perl modules being installed, this is a problem. Apache, the web server (which displays your websites) is unaffected, but mail delivery is.

We expect to have this issue resolved shortly and will keep you informed. This issue will be a problem for a wide range of servers running cPanel, so it's not just us. I just happened to be working late and noticed it. Lucky me. :)

EDIT: 1:20pm: All issues have been resolved, and we thank you for your patience.

DNS Changes For Main Website

Some DNS changes were made for the main website, http://helloworldweb.com, which affects the Client Support and Billing Area. While the changes propagate there may be some short downtime. This change won't affect any hosting clients' sites, only our own.

If you need anything we'll be available on LiveChat as much as possible.

For those of you who use OpenDNS, you can login to your dashboard, check on this (or any other domain name), and request a CacheCheck (under the "Support" tab). They also allow you to request a cache refresh, though it doesn't seem to speed up they're releasing of the cached view any quicker.

But it made me feel better, to have a button to press. :)

EDIT 9:41pm: Annnnnnd we're back, worldwide. Guess I panicked for nothing!

Weak ciphers and SSLv2 disabled

As required for both PCI Compliance and general server security, our server has been configured to disallow Secure Sockets Layer (SSL) version 2 as well as "weak" cryptography. From this point on only SSLv3 will be allowed and supported. This isn't a change anyone should notice, as all modern browsers and email clients support SSLv3.

The only wrinkle in this process has been in the POP3 mailserver's SSL version implementation. Currently users are unable to connect and check email via an email client using SSL. Using an email client without SSL works fine (for both POP3 and IMAP), as does using webmail.

Since the setting changes were made through the WHM/cPanel's interface (necessary so the settings wouldn't be overwritten by a subsequent cPanel update), it appears to be a bug in the cPanel interface.

We have been working with a series of cPanel support techs over the past 26 hours to pinpoint and fix the issue, and will report when the issue has been cleared.

EDIT: Sat., Aug 22, 2009 12:47am: This issue is now resolved.

WordPress and Joomla updates available

Fantastico has made the following updates available:

* Joomla 1.5: 1.5.13 -> 1.5.14
* WordPress: 2.8.2 -> 2.8.4

[updates] Joomla and WordPress security updates

As promised, Fantastico has provided the following updates:

* Joomla 1.5: 1.5.12 -> 1.5.13
* Open-Realty: 2.5.6 -> 2.5.7
* SMF: 1.1.9 -> 1.1.10
* TYPO3: 4.2.1 -> 4.2.8
* WordPress: 2.8 -> 2.8.2

Joomla and WordPress are critical security updates, and should be applied ASAP.

Software updates available through Fantastico

The following updates are available through Fantastico's auto-installer:

* Coppermine Photo Gallery: 1.4.20 -> 1.4.25
* Drupal: 6.10 -> 6.13
* Joomla 1.5: 1.5.11 -> 1.5.12 ***
* LimeSurvey: 1.82+ -> 1.85+
* OpenX: 2.0.11-pr1 -> 2.8.1  (was "phpAdsNew")
* osTicket: 1.3.1 -> 1.6 RC5
* PHPlist: 2.10.9 -> 2.10.10
* phpWCMS: 1.1-RC4 Rev. A -> 1.4.2 r327
* TikiWiki: 3.0 -> 3.1
* Zen Cart: 1.3.8a -> 1.3.8a
* Zikula: 0.764 -> 1.1.1

*** Important Note: According to Fantastico support, critical releases for both Joomla (v1.5.13) and WordPress (v2.8.2) have just been released within the last couple of days, and will be included in the next Fantastico update which is expected within the next 24 hours.

It should be noted (once again) that Fantastico is an installer, not an "updater." Fantastico is designed to make it easy to install software, and though their developers are very diligent about providing the most update versions, it's the responsibility of the website owner and/or the person who installs the software/script to be aware of available security updates and patches.

New SSL certificate for Apollo server

The SSL certificate for the Apollo server was been renewed. Depending on the email client/software you use, you may need to re-accept the certificate for email, and/or also re-accept the certificate in your browser (or create a new exception, in the case of a domain mismatch).

This may need to be done any time a new certificate is installed or renewed. The hostname will always be the same, only the validity dates change.

If you have any questions about this please feel free to ask.

Fantastico is once again working their update magic

The following application updates are now available through Fantastico:

* Joomla 1.5: 1.5.10 -> 1.5.11
* LimeSurvey: 1.72 -> 1.82+
* SMF: 1.1.7 -> 1.1.9
* TikiWiki: 2.4 -> 3.0
* WordPress: 2.7.1 -> 2.8

Enjoy!

The Client Management/Support Area to be moving

The online system that we use for support, billing/ordering, client management and communication will be upgraded and moved in the next 3-7 days.

Since the system upgrade is a jump in version involving many new features, and we'd been planning on moving the system to an easier-to-remember directory anyway, a fresh install in the new directory will be done and the system will be tested and confirmed working, leaving the previous version intact in the unlikely event the upgrade doesn't go as planned. In all things computers, it's always good to have a backup plan.

A redirect will be in place from the current/old to the new location, so any direct links (to open support/work order tickets, etc) should continue to work.

This is just one step in our evil plan to take over the world! Or, at the very least, provide design and hosting clients with a more pleasant visual experience and make our communications easier.

Routine scheduled maintenance

On Thursday, March 19th, 2009 between 12:00AM and 6:00AM EDT, we will perform maintenance on the server. A reboot of the server will be required. We do not expect the downtime to exceed 15 minutes.

More Fantastico Updates Available

* Coppermine Photo Gallery: 1.4.19 -> 1.4.20
* Moodle: 1.9.3 -> 1.9.4
* Noahs Classifieds: V8 (2.4.1) -> V8 (3.2.0)
* PHPlist: 2.10.8 -> 2.10.9
* WordPress: 2.7 -> 2.7.1

Fantastico application updates available

The following application updates are now available through Fantastico, for your updating pleasure:

* Drupal: 6.6 -> 6.9
* Joomla 1.5: 1.5.8 -> 1.5.9
* PHPlist: 2.10.7 -> 2.10.8
* phpBB: 3.0.3 -> 3.0.4