Monday, November 3, 2014

Apollo Server Issue

The Apollo server is down, and the data center engineers are actively investigating and will have the issue resolved as quickly as possible.

EDIT: 4:24AM PST The server and all services are up and running. Thank you for your patience.

Thursday, October 23, 2014

Apollo Server Migration Complete

And ahead of schedule! We now return you to your regularly scheduled website...
...already in progress.

Wednesday, October 22, 2014

Apollo Server Migration Scheduled

Our approach to upgrades has always been a proactive one, to anticipate our users' changing needs rather than waiting until hardware breaks to fix it. Within the past 2 months we've done major upgrades to both PHP and MySQL, providing improved performance and reduced memory consumption, resulting in measurable improvements to your user experience.

Shortly after that we added an additional layer DDoS mitigation, to block the attacks that have unfortunately become a daily part of life on the Internet. It's a bit like playing "Whack-a-mole", but so far it seems to be working.

And now it's time for some new shiny hardware! Solid State, even!

In the early morning / overnight hours of Thursday Oct. 23rd starting at 3am EDT / 12mid PDT** we will begin migration to a newly provisioned SSD server. There will be some downtime while your data is being migrated. We hope that the downtime will be minimal, but it is based on many variables.

This upgrade will provide us with what's called "downtime-free OS kernel updates" and allow server-level kernel updates to be accomplished without needing to reboot the server. Currently, when the OS kernel (the "heart" of the server's operating system) is updated to patch security issues, the server has to be rebooted, causing downtime as the containers come up one after the other. With the new system, such patching would happen without downtime.

But of course I saved the best for last. The new server will not only rock more RAM but a SOLID STATE DRIVE!  

What's so great about a solid state drive you ask? Only everything!
  • No moving parts, nothing to break or wear out
  • 20-100 times faster than a traditional hard drive
  • Can process 3 times as much data in the same amount of time as a traditional hard drive
  • Improved performance, faster read/write speeds allow websites to load faster
  • Improved uptime, more reliability
  • Zero downtime will be required for future upgrades
  • Green technology! SSDs require 80% less power than a traditional hard drive
 We're very excited about upgrade, and hope you will be also!

During the migration our support desk will be unavailable, but Live Chat will be.

** We apologize for the short notice. I was notified only yesterday of a planned migration that was scheduled by the data center for late Thursday night; the migration is required due to a aging hardware on our current server and they like to keep everything humming along. 

Rather than have two incidents of downtime and disruption (one for the needed migration, then another in a month to upgrade to a better SSD server), I decided now would be the time for the SSD upgrade. 

This was scheduled at the slowest time of day, but had to be scheduled to occur before the other already planned migration on Thursday.

Wednesday, September 24, 2014

Bad bash, no cookie

 There's a very nasty vulnerability that's been discovered in bash, a command processor (it can be compared to DOS in Windows) that's included in most Linux and Unix systems including Apple's OS X.  Ubuntu and other Debian-derived systems typically use Dash, and unless bash has been installed they are exempt from this vulnerability.

Bash is a command processor, allowing the user to type commands which cause actions on the server. As such, in the wrong hands it can be disastrous, which is why this particular vulnerability is very serious.

The NIST vulnerability database rates the flaw 10 out of 10 in terms of severity. Jim Reavis, chief exec of the Cloud Security Alliance, claims the hole is comparable in seriousness to the infamous password-leaking Heartbleed bug in the OpenSSL library that was uncovered earlier this year. This vulnerability also affects Apple's OS X – and is useful for privilege escalation.

The good news is, at least for Linux servers, is that a patch/update to bash was released earlier today, and it's a very simple fix. By "very simple" I mean it took less than 5 minutes to accomplish, so there is no reason for any hosting server admin to not apply this fix. OS X users  may have a harder time patching your system.

But if you're one of our hosting customers... we've got you covered.

Monday, September 15, 2014

Support system upgrade

The support, billing and ordering system is currently being upgraded. We'll be back momentarily... all shiny and stuff!

If you need help while the support system is being upgraded, we're available through Live Chat.

EDIT 11:13pm: Aaaaaand we're DONE! If you see images that look out of place, try a hard refresh (Control + F5) or clear your browser's cache. This was a major upgrade and we tested thoroughly after the upgrade, but if you find anything isn't working, feel free to let us know.

This is the first step of many planned upgrades to all areas of our website.... stay tuned for more!

WordPress Slider Revolution plugin critical vulnerability

Attention WordPress users! If you use a premium theme that has bundled with it the "Slider Revolution" plugin (one notable and extremely popular theme that includes this is Avada), then you need to upgrade your theme immediately.

There has been a critical security vulnerability found, as reported by the Sucuri security company, who specialize in WordPress security issues, and also WPTavern. The Sucuri article carries much more detailed information about what this exploit can do, but it might be a difficult read for some users.

Since there are WordPress vulnerabilities that are disclosed just about every day, you might wonder why this particular one rates a post in our status blog. Not only is this a particularly nasty exploit, but the fact that it involves a plugin that's included in premium themes, and therefore makes it more difficult for the average WordPress site owner to upgrade, or even receive a notification of the upgrade being available.

As Sarah Gooding states in the WPTavern article:
The Risk of Using Free or Commercial Extensions Without Update Notifications

If you are using a commercial plugin or theme that has no auto-update system or relies on email to notify you of updates, you need to be very proactive about keeping yourself informed. A critical security vulnerability, such as the one reported for Slider Revolution, can easily take down your site(s) if you neglect updates. Theme authors don’t always update their bundled plugins and their users cannot take advantage of the auto update system provided by the plugin author.
In addition, the Developers of the Slider Revolution plugin chose not to announce there was a security issue, nor notify users of the importance of this upgrade. It's possible a good number of WordPress owners have no idea there's a security issue, and when that involves a very popular WordPress theme, that can be a recipe for disaster.

 Envato (the marketplace that sells most WordPress premium themes) has releases a comprehensive list of the themes possibly affected by this vulnerability. I would urge you to check to see if your WordPress theme is on that list, and update accordingly.
To upgrade a premium theme such as Avada, you'll need to login to your account with the Envato marketplace. If you have trouble logging in I would recommend contacting Envato support.

We are already seeing scans for this vulnerability, so upgrading immediately is vital.

"GET /wp-admin/admin-ajax.php?action=revslider_show_image&img=../wp-config.php

We've taken steps to block any requests for that URL in Mod_Security, this should be a temporary fix until all users can upgrade their affected themes.

We do all we can to assist our hosting users and keep your websites and data safe, but ultimately it's your responsibility for what software is installed and used on your website, and your responsibility to keep it updated (per our Terms of Service).

WordPress can be deceptively simple to use, but the back-end is very powerful. And with great power, comes great responsibility.If this is something that's beyond your ability to handle, hire a knowledgeable WordPress admin (like us!). The cost of having someone who knows what they're doing at the helm is far less than what it would cost, in time, money, and lost business reputation to deal with your WordPress site being hacked.

To stay updated and notified of available WordPress updates, you should have email notifications of updates enabled in your WordPress dashboard, or (if you're our hosting customer) through Installatron.

To be notified of general Internet and hosting security issues, please either subscribe to this blog, and/or follow us on Facebook, Tumblr, or Twitter.

Sunday, August 24, 2014

Server Maintenance

In a few hours we will be performing a series of server upgrades to improve the security and stability of the server's systems, specifically the Linux kernel. These upgrades will address intermittent stability issues that have been seen on the most recent vendor-supplied Linux kernels.

 At the same time we will install a stable, live kernel image patching system - part of a series of planned service upgrades - that will limit the need for future reboots when a new kernel is installed.

The maintenance window is Monday August 25th, from 2am to 4am, EST, 11pm (Sunday night) to 1am PST. Sorry for the short notice folks, sometimes it's unavoidable.

We expect the maintenance window to last approximately two hours, and during that time, your server is likely to be inaccessible for approximately 10-15 minutes from the time of reboot.  In other words there will only be 10-15 minutes of downtime, but it will occur sometime during the two hour window.

We appreciate your patience as we work to ensure the security and stability of your content and data.

Wednesday, August 20, 2014

Server Sluggishness

We've noticed the server, while still "up" and displaying websites as it should, is being rather sluggish and slow to respond. The cause is being investigated and will hopefully be rectified momentarily.

Sunday, July 13, 2014

Scheduled Maintenance

In the overnight/early morning hours of Monday July 14th we will be performing critical but scheduled maintenance on the servers. This maintenance will require a server reboot and will result in less than 15 minutes of downtime during the reboot.

The maintenance window is 2 hours, 5am to 7am EDT (2am to 4am PDT) early Monday morning (tomorrow). The server reboot will happen sometime between these hours.

Thank you for your patience.

Saturday, June 28, 2014

Network Outage

There's currently a network outage that's affecting not only our servers but the entire data center (as well as their website/customer service portal). They are aware of the problem and are keeping us updated through Twitter (see why having multiple channels of communication is a good thing?).

We will keep you informed as we receive information. Hopefully this will be a short-lived outage (*crosses fingers*). It's extremely rare that the network goes down, but it does happen.

Update 4:28pm: Still no further word, but we're monitoring the situation and have requested an update from the NOC.

Update 5:02pm: This from the data center (which admittedly doesn't tell us much at all):
"All hands are on deck working to fix this problem. We will share all details when the engineers emerge. Thanks for your patience."

Update 5:35pm: Annnnd we're back UP! Thank you again for your patience. Had been anything we could have done to speed up the fixing we would have.

 Yesssssss, thank you, Internetz!

Update 6:24pm: I spoke too soon. The NOC engineers are working to fix the problem. During this time services will be on and off (up and down).

From the data center:
"Reserve troops are heading into our managed services team HQ so we can answer your questions as soon as normal communication is restored."

All communication into the data center is down (their website, ticket system, phone lines, etc), and I'm sure they're doing the best they can, and will update us as to what's going on as soon as possible. Preferably after they've fixed the issue.

Update 8:31pm: The servers as well as all services are back up. This time the data center's own website is back up also, which is a very good sign (it's been down for the past 5 hours). Hopefully all services will STAY up this time.

This was an extremely unusual event, as I've said before; something very catastrophic must have occurred. As soon as we know what happened, we'll certainly pass it along, if for no other reason than to satisfy all of our curiosity. And again, we do apologize for this disruption.

Tuesday, June 24, 2014

Emergency Maintenance on Apollo Hosting Server

A major security flaw has been discovered in critical vendor-supplied software that helps power our  servers' infrastructure. Emergency maintenance was done to patch the server (the kernel was upgraded), which required a server reboot and resulted in 3-10 minutes of downtime.

We apologize for this disruption in service, even I wasn't notified ahead of time, but in the interest of security it was necessary. Services are coming back up now as I write this. Thank you for your patience.

Thursday, April 17, 2014

All WordPress releases, great and small

On April 14th WordPress released WordPress version 3.8.3 maintenance release, which fixed a "small and unfortunate bug" in version 3.8.2, a security release that was published 6 days prior. The bug in v3.8.2 had to do with how WordPress auto-saves drafts in the admin dashboard:

The "Quick Draft" tool on the dashboard screen was broken in the 3.8.2 update. If you tried to use it, your draft would disappear and it wouldn't save.

Then on April 16th version 3.9 was released. Confused as to what's what, and which version to safely upgrade to?  It's easy, once you know how WordPress versions and updates work.

WordPress Minor Releases: Maintenance and Security

Any time you see 3 numbers in the WordPress version (i.e. - 3.8.2, 3.8.3), that signifies a minor version update. These are always either a maintenance update (a critical bug fix) or security update (in response to a security issue disclosed/discovered).

Minor versions are released as needed, and the update should be applied immediately.

WordPress Major Releases: A Version With New Features Introduced

When there are just 2 numbers in the WordPress version (i.e- 3.7, 3.8, 3.9), that's a major release. A major release introduces new features which may be very cool, but the update is not time sensitive or critical.

It may also take a few days, a week, or sometimes longer for your theme and all of your plugins to be made compatible. Check with each one of your plugins to verify they are compatible before upgrading a major version.

Major versions are released every 4-5 months, and updating can wait until you've had a chance to until you've had time to read about all the new features and possible changes, and verify compatibility.

 You can read more about the WordPress version release cycle from the WordPress website.

Automatic Updates Make Life Easier

As well as your WordPress blog more secure. Starting in version 3.7 automatic updates for minor releases were enabled by default in the WordPress core. In addition if you're one of our hosting customers the Auto-Installers we offer can be enabled to automatically update plugins in themes, as well as major releases.

Andrew Nacin, a WordPress Developer, wrote this guide in October 2013 when version 3.7 came out and introduced automatic minor version updates. Despite the title of the article, it gives every reason why enabling automatic updates are a GOOD thing. A few quotes from the article:

Background updates are incredibly, incredibly safe.

Minor releases don't break things.

WordPress has prompted users to install updates for years. I don't know how many declined as much as didn't pay attention or consider it a priority. Your phone buzzes in your pocket; it's something you can choose right then to act on. If you don't use your phone for a while, it's probably not a big deal if you wait for an update.

But running a site on the Internet carries some responsibility, and they don't buzz in your pocket. (Out of sight, out of mind.) For the betterment of the web, we made a conscious decision to avoid a UI option. You'd be out of your mind to consciously avoid updating to fix a critical bug or security issue. We think the vast majority of users (many who don't even know what PHP is) will celebrate this as a win in usability and security.

Which options should you choose? We recommend the following settings:

WordPress Minor Release: automatic updates enabled

WordPress Major Release: manual updates only. Take a backup of your database first in case there are any problems.

Themes: manual update only, unless you're using a WordPress default theme (twentyfourteen). Themes should definitely be updated, but I recommend giving it a week, and check the support forum of your theme to see if anyone else is reporting bugs that could affect your blog's display.

Don't blindly update your theme on the first day an update is released... unless it's a security release! Which does happen, however rarely, with WordPress themes. Take a backup of your theme directory before updating as a precaution, and if you customize a theme, always make a child theme so your customizations won't get overwritten.

Plugins: It depends on which plugins you use, if you trust the companies who develop them, and how often you access your blog.

If you login daily, then I'd recommend checking for updates yourself, and updating the plugins one at a time, so that if an update breaks anything, you'll know who the culprit is. On the other hand, if you login infrequently (once a week or less), then it's better to turn on automatic plugin updates and risk an update breaking something than having your blog/website hacked from a plugin security issue. Most WordPress compromises are done through a plugin, not the WordPress core itself.

Security risks are frequently found in WordPress plugins, you really need to stay on top of updating them when an update is released.

  And While We're Talking WordPress...

JetPack: On April 10th Jetpack release version 2.9.3, a Critical Security Update

During an internal security audit, we found a bug that allows an attacker to bypass a site's access controls and publish posts. This vulnerability could be combined with other attacks to escalate access. This bug has existed since Jetpack 1.9, released in October 2012.

This is a bad bug, and JetPack is one of the most widely used plugins in the WordPress world. Don't hesitate, update! (Thank you, thank you, I'll be here all sure to tip your servers and bartenders!)

Akismet: Version 3.0.0 of the Akismet plugin for WordPress is now available.This is a major rewrite of the plugin code. It includes many small improvements and some new features. In particular:

    An easier signup and activation process
    An even easier activation process for Jetpack users
    A redesigned configuration tab
    New stats charts (example shown below)
    A new discard feature for outright blocking of the worst spam

Hackers trick 162,000 unsuspecting WordPress sites into launching DDoS attack. As reported by IT security researcher Graham Cluley,
The attack relied upon Pingbacks – a feature of WordPress that allows a site running WordPress to inform other sites when you write a blog post linking to them. But the WordPress sites were not hacked or compromised. Instead, through use of a simple UNIX command line, a remote hacker could tell one website to send an HTTP request to the target site, via the Pingback feature.

It is strongly recommended that you disable pingbacks. Any WordPress site with Pingback enabled (which is on by default) can be used in DDOS attacks against other sites.

As detailed in the Securi blog (they specialize in Internet security), in the course of just a few hours, over 162,000 different and legitimate WordPress sites tried to attack his site. We would likely have detected a lot more sites, but we decided we had seen enough and blocked the requests at the edge firewall, mostly to avoid filling the logs with junk. From Securi's blog post:

 Can you see how powerful it can be? One attacker can use thousands of popular and clean WordPress sites to perform their DDOS attack, while being hidden in the shadows, and that all happens with a simple ping back request to the XML-RPC file.
You can use Securi's WordPress DDOS Scanner to check if your site is DDOS'ing other websites.