Moving day is done, now it's time to arrange the furniture

All sites have been transferred to the new server and DNS (the system that points your domain name to it's place on the new server) has propagated. Websites and email, ftp, etc should be functional.

And speaking of email....how 'bout all that SPAM?!?! Isn't that something, eh? Consider this an experiment to should just how POWERFUL our SPAM filtering system is. The SPAM filtering system will be reinstalled today, as will the necessary server system modules so the support ticket/client area will be up and running.

Had the old server not been ready to die again, I would have (and planned to) have everything ready on the new server before transferring client accounts, but given the circumstances it was better for all to make the switch as fast as possible.

Moving day

This notice is being emailed as well as posted to the Announcements blog, to make sure everyone receives this information. If you don't already subscribe to the announcements blog, please do so through FeedBurner.

We're in the process of backing up all accounts to an offsite/remote backup server, and have begun transferring accounts to the new server. When the move is complete a notice will be posted here, but if you'd like to be ahead of the curve, the new IP will be: 207.58.178.12 All account usernames and passwords will remain the same.

This move is being done ASAP rather than in stages, as I've been receiving warnings that the temperature on the current server is running high. Clearly, something is wrong, and I'd like to complete the move before the server goes down again.

DNS is being changed and the new nameservers are being *added*, with the old/current nameservers still listed so as to allow for some "cross-over" time. If this is all Greek to you, then we handle your DNS and you don't need to do anything, we will make the changes for you. If you handle your own DNS, please add the following nameservers:

ns5.helloworldwebhosting.com (207.58.178.14)
ns6.helloworldwebhosting.com ( 207.58.178.13)

For domains ending in .org :
ns2.helloworldwebhosting.org (207.58.178.13)
ns3.helloworldwebhosting.org (207.58.178.14)

Again, if you don't understand what that means about nameservers, don't worry about it, the nameserver information only applies if you handle your own DNS (and if you do, you'd know).

If you could please hold off uploading large files until after the transfer is complete, that would be helpful. The "all clear" will be posted when this process is done. Thank you for your patience.

FINALLY! All services restored

Everything should be up and running, and with the exception of the absence MailScanner (which will be remedied in the next hour or so) to filter out all the spam, everything should be as it was.

If you notice any issues please login to the Client Portal and submit a ticket.

Once again, we thank you for your patience during this trying time. Even though this was completely beyond my control, and i did do everything to rectify the situation, I would like to offer some type of compensation to my valued clients. Rather than me deciding what <i.you might like...I'm going to throw it out there.

What, within reason, would make YOU happy? A free month of hosting? More disk space? A free domain registration? A baby's arm holding an apple? Extra kudos to those who get that last reference.

The restoration is complete

Web services are the only thing down, as they are apparently updating Apache (the web server).

Email is working, but you will likely see SPAM that you're not used to seeing. The data center wiped out the SPAM filtering applications we had installed. Nice of them, eh? Yeah...I thought so too. They will be reinstalled as soon as the data center finishes whatever they are doing on the server.

2:43pm (PDT): Still waiting for the techs to finish restoring the accounts.

Continuing to monitor progress

I'm trying to find out what's going on, directly with the data center engineers. As soon as I can find out what's happening and when we can expect service restored (at this point just long enough to pull the backups off the 2nd drive and get the heck out of dodge) I will post that information here.

I was not informed they were replacing any hardware ahead of time, and only found out because the tech set off the intrusion detection. The server admin company seems to be having some sort of problem with their support ticket system, and I'm having trouble communicating with them, which is why I've gone over their heads directly to the data center engineers (as surly as they are, because I'm not supposed to do that).

I'm doing everything within my power to restore services to the server, short of driving to Atlanta and fixing it myself. If it weren't for the price of gas, I'd be doing that right now.

Again, I deeply apologize for this outage. I am disgusted with the way it's been handled by the server company and the data center, and at the first opportunity we will be as far away from them as possible.

Server drive being replaced

The data center has determined the drive is defective and it is being replaced tonight. Services will be down while this is being done and accounts restored to the new drive. This process typically takes an hour or two.

Server is down yet again

And yet they claim the hardware is "fine" per Tuesday night's test. Clearly it is not, and the data center is not performing according to my contract with them. We'll be moving the accounts to another server starting immediately. When we have the new IP clients will be notified individually.

I am very sorry for this disruption.

2:28pm [PDT]: Services are back up, the data center technicians are going over the server yet again. As soon as the new server (at a different data center!) set-up is complete, we will be transferring all accounts, as fast as possible.

Testing the server hardware at 9pm [PDT]

There will be brief but this time planned downtime at 9pm PDT, midnight EDT, to ascertain/troubleshoot the server problem. The test should last between 15-30 minutes, and was scheduled for a time when the lowest amount of server traffic would be expected.

Server down again

It just happened, as everything was functional not 10 minutes earlier. I've put in tickets with both the server management company AND directly with the data center. I'm also switching companies as fast as is possible, without any further disruption in services. It takes a few days for the new services to be set up on the new server.

11:30am [PDT]: Server, and all services, are backup. Sorry for the inconvenience.

Server down

The server is down with an "Internal Server 500 error", which means something's been misconfigured, although we haven't configured anything recently. I have an emergency ticket in with the data center, and will update the progress as soon as I have any news.

12:58pm (PT): Data center is still working on the issue, and have escalated the ticket. The server has been rebooted.

2:34pm (PT): I requested a progress update at 1:58pm, they have yet to reply. I will continue to monitor/prod until this is resolved.

4:12pm (PT): Still no word back, other than "We are still working on this." I've escalated the ticket, gone over their heads and submitted a ticket directly with the Engineering dept of the Data Center.

5:11pm (PT): FIXED! All services are back up and running! The only lagging I'm seeing is when the site's DNS is 100% external (meaning you have another company running your DNS).

This is never recommended, but what *is* recommended is having backup, or secondary DNS, with us as primary. That way, you're covered, no matter what. This will also keep your email flowing, etc.

There's not much that can be done when there's a hardware failure (as this appears to have been, but the specifics are unconfirmed), and we are pushing the data center for an answer as to what happened. The plan to switch data centers is being moved up, and customers will be contacted as their sites are moved. We plan to have "cross-over time" meaning the old server won't be shut off until all DNS is fully switched over to the new server.

We expect no downtime (further hardware failures on the current server not withstanding). The new server company advertises "ZERO downtime during hardware failures." It's something I hope to never have to put to the test, but no technology is ever 100%.

We apologize for this downtime, and are constantly striving to provide better service to our valued customers. We appreciate your patience.

Fantastico-installed updates available

The following script updates are available through Fantastico as of today. If you have any of the following applications installed, please update at your earliest convenience.

Updates:

* b2evolution: 1.10.2 -> 2.4.2
* Coppermine Photo Gallery: 1.4.16 -> 1.4.18
* Drupal: 6.2 -> 6.3
* Gallery: 2.2.4 -> 2.2.5
* Joomla 1.5: 1.5.2 -> 1.5.4
* Mambo Open Source: 4.6.3 -> 4.6.5
* Moodle: 1.9 -> 1.9.2
* SMF: 1.1.4 -> 1.1.5
* TikiWiki: 1.9.10.1 -> 1.9.11
* WordPress: 2.5.1 -> 2.6

Server rebooted

I just received notice from the data center that the server has been rebooted. Typically it takes 5-10 minutes to fully reboot, during which time services will be unavailable. The data center didn't say why it needed to be rebooted (it's very rare that it does), but if I can find out I'll edit this post.

Plans were already in the works to switch to another more reliable data center/network in the next 2 weeks...this just cements that idea as a sound one.