Overnight DNS Issues Resolved

Overnight (or in the wee hours of the morning, depending on the time zone you're in) our nameservers failed, apparently due to an upgrade of BIND (the DNS server) pushed by cPanel. The upgrade configuration file (yum.conf) was written to exclude BIND for automatic updates (so they can be done when we're watching, to fix any issues immediately), but that configuration file was over-written at the same time, presumably also by cPanel.

The update erroneously added duplicate entries to the DNS server configuration file, which caused the failure.

For those unfamiliar with how DNS works, the (DNS) nameservers are what translates your domain name to the physical server's IP address. The physical server was technically up and functioning (email continued to be processed, etc), which is why the typical error alerts that are sent by SMS after business hours didn't go out. In addition the overnight support staff weren't set up to monitor the nameservers. Those have both have been rectified (as well as some new alerts added), should this ever happen again it can be fixed immediately, and a bug report will be filed with cPanel support.

Including backup DNS for all accounts (at no additional cost) has been in the works for some time, and using an external DNS provider has also been considered, it's just been a question of finding the right, reliable DNS provider. Testing and research of these will be bumped up to a higher place on the "to do" list.

We apologize for the downtime and have taken steps to make sure this doesn't happen again.

Fantastico Updates and Installatron is Coming

The following application updates are available through Fantastico:

* 4Images Gallery: 1.7.7 -> 1.7.8
* Coppermine Photo Gallery: 1.5.6 -> 1.5.8
* Drupal: 6.16 -> 6.19
* Gallery: 2.3.1 -> 3.0
* LimeSurvey: 1.87+ -> 1.90+
* OpenX: 2.8.5 -> 2.8.7
* PHProjekt: 6.0.1 -> 6.0.4
* TikiWiki: 4.3 -> 5.2
* Typo3: 4.3.3 -> 4.4
* WebCalendar: 1.2.1 -> 1.2.3
* WordPress: 3.0 -> 3.0.1
* Zen Cart: 1.3.9d -> 1.3.9g

Unless you're brand new here (and/or new to using Fantastico to install web applications), you've probably noticed that Fantastico is consistently lagging on providing application updates, even when it's a critical security update. To be fair, an auto-installers job is to do the initial install, not keep an application updated. The problem is when you install some applications within Fantastico, they can only be upgraded through Fantastico, or problems arise.

Fantastico's sluggish updates have long been a concern for most of the web hosting community. In particular WordPress updates really need to be done ASAP, since WP is such a targeted popular script.

We've offered Fantastico for many years (5-6, if memory serves), many users come to expect it to be a part of any hosting package, and it hasn't been until recently that Fantastico has had any real competition.

Enter Installatron. It's been getting rave reviews, comes with a ton of cool features, includes scripts and applications Fantastico never has (and likely never will)**, will convert applications installed by other auto-installers (like Fantastico), and the best part...

How often are the application installers updated?
We add critical updates within hours of release, guaranteed within 24 hours, but normally in less than 12 hours. This includes non-business day releases.

Non-critical updates weigh speed with stability, and in some cases the developers actually let us know when they think the latest version is ready to add to Installatron. Standard releases for stable or popular web applications usually follow the "critical" update path, while larger version changes (and certain applications) might be given some breathing space to ensure stability.

12-24 hours for new version releases, compared to typically weeks with Fantastico. Score! We're performing testing on Installatron now, and will announce as soon as it's ready and been added to all hosting accounts.

**please note Hello World Web Hosting excludes some applications deemed security risks, as most good hosting companies do

Spam, Spam, and more Spam

Last night we became aware that all inbound email was not being scored, tagged or filtered by any of our spam filters. Every email had a score of zero and was therefore being delivered.

On Nov. 9th cPanel was updated to version 11.28, which had quite a few changes, one being the configuration file for Exim (the MTA, or Message Transfer Agent), the software that processes your email). Oddly enough the server didn't send a notification that there was a problem with the configuration file until late last night, Nov. 16th. Email continued to be processed, but for some reason it rendered our spam filtering system useless, while the main spam filtering component, MailScanner, still reported it was functioning correctly.

In troubleshooting what the problem was I also found the Bayes database used by SpamAssassin was unusually large, and contained 67 million tokens. Didn't I tell you our spam filters rock? If the database gets too big SpamAssassin can have trouble loading it and using it, so this could have been a part of the problem also.

The end result was that all email was not given any spam score from SpamAssassin and was delivered as "non-spam." Typically 84-90% of all email processed on our server is spam, so you've likely noticed a big difference in the last week or so.



Think of this past week as an experiment, "What your inbox would be like without spam filters." Scary, wasn't it??? I'll bet many of you will never take our spam filters for granted again, huh? I know I won't!

Everything appears to be back on track and filtering correctly. You may receive a few "message released from quarantine" emails if I see any false positives (newsletters or legitimate email being marked as spam). It's always a good idea to routinely check your spam mailbox for false positives, particularly for newsletters, as they frequently contain all the characteristics of spam.

In the "MailScanner" section of your cPanel there are options to adjust your own spam settings (for low-scoring and high-scoring spam) as well as whitelisting and blacklisting options for domains or email addresses.

We apologize for any inconvenience this caused.

Critical Security Update: Joomla! 1.5 0Day Exploit

A critical update has been released today for Joomla! 1.5, specifically instances that were installed by Fantastico on a cPanel hosting server. This update is in response to a 0day Exploit, which could give the attacker access to your website and files.

Affected versions of cPanel are 11.25 and below, and while we currently run version 11.26, it's recommended that any users running Joomla 1.5 are urged to update immediately.

Updates Available Through Fantastico

The following updates are available through Fantastico:

* Coppermine Photo Gallery: 1.4.26 -> 1.5.6
* Joomla 1.5: 1.5.17 -> 1.5.20
* Moodle: 1.9.8 -> 1.9.9
* Soholaunch Pro Edition: 4.9.3 r17 -> 4.9.3 r41
* TikiWiki: 4.1 -> 4.3
* WordPress: 2.9.2 -> 3.0
* Zen Cart: 1.3.9b -> 1.3.9d

If you installed WordPress through Fantastico, please remember to always upgrade through Fantastico, also.

Never Let Your Guard Down

I don't consider myself a computer/network/Internet security expert, but I've been at this a long time, and having 10+ years' experience on both sides of the fence, I'd say I know more than the average user about the issue of computer security.

Security is an issue I take very seriously, and I pretty much run my own network in "paranoid mode." My anti-virus/malware/spyware tools and programs are always up-to-date, I use a well-configured hardware firewall, anything I download personally gets scanned twice, I do regular malware/spyware/virus scans in addition to having constant protection enabled against such threats.

I say this because today I got a trojan, my first one in 9-10 years, that my anti-virus detected and warned me about, but wasn't able to quarantine or delete, though it said it would be deleted after a reboot.

Being curious about these kind of things, as well as hoping to learn where I could have picked up the trojan to begin with, I went to Google first to find what I could about this particular trojan, TR/Agent.uwi.6144, and saw some really curious behavior in my Google searches. Each search I did would redirect to another kind of search site. At first I thought it was something with the sites themselves, yet I found I could still see the intended site using Google's cached option. Google cache is your friend: Before I started using version control for Web Design projects, Google's cache saved my butt a time or two when I'd accidentally deleted the wrong file. In case you're wondering, this was using Firefox browser, not Internet Explorer (which I only use for testing purposes).

This particular trojan (which goes by a variety of names) hijacks your browser and redirects to spam search sites. After a reboot - as the anti-virus program promised - the trojan was gone and correct functionality was restored to my Google searches. Trojans are not typically caught by most anti-virus programs, as anti-virus focuses on viruses (spell-check claims "virii" isn't a word), and trojans aren't technically a virus (they're a back-door, which is worse), and usually require a separate program. The anti-virus I use, Avira, scans for and blocks trojans, malware and spyware.

My point here is this:
The average computer user views their anti-virus, anti-spyware, anti-malware applications as something you "set and forget." Setting is good...forgetting, or becoming complacent about security. It's a lot easier to defend against an intrusion than to clean it up afterward.

Just like spammers who are always finding new ways around spam filters, virus writers/creators are always finding new ways around existing computer protection. It may take hours or days for a brand new threat to be included in your anti-virus' database (and therefore detected), so even when you keep your anti-virus definitions up-to-date, there's always a window of time where a potential infection could occur.

Beyond that...no anti-virus/anti-spyware/anti-malware (or for that matter, spam filters) is, can be, or should be considered to be 100% fool-proof.

Never let your guard down.


Disclaimer:
This is not an invitation to a geek pissing contest, to see who can boast they've never had a security issue, or that they don't have to worry because they run Linux/Mac/etc. I find this kind of behavior boorish at best, and counter-productive at worst. The single most important part of computer security is user education, which is why I'm taking the time to post this. I don't need a chorus of geek snobs (and you know who you are) claiming "If you only ran Linux that wouldn't happen."

Like each computer itself is set up differently, each computer users' needs and requirements are different. I happen to have a business need for running a Windows machine...period, end of story. Windows dominates the market in personal (home) computers, and it's counter-productive to pretend otherwise.

I also work my machines to death, way more than the casual computer user, which means that I am both more aware when something isn't right (spending 18 hours a day with anyone or anything you develop a symbiotic relationship) and my machines are also exposed to more potential threats than the casual computer user. I download files from clients on a routine basis, and though they are scanned once if not twice, there's always the possibility of receiving an infected file that they themselves aren't aware of.

[cross-posted to Hello World!, our Web Design blog]

Updates Available

The following software/application updates are available through Fantastico:

* dotProject: 2.1.2 -> 2.1.3
* Drupal: 6.15 -> 6.16
* Joomla 1.5: 1.5.15 -> 1.5.17
* Moodle: 1.9.7 -> 1.9.8
* osTicket: 1.6 RC5 -> 1.6.0
* phpBB: 3.0.6 -> 3.0.7-PL1
* PHPlist: 2.10.10 -> 2.10.12
* PHProjekt: 5.2.2 -> 6.0.1
* Typo3: 4.2.10 -> 4.3.3
* WebCalendar: 1.2.0 -> 1.2.1
* Zen Cart: 1.3.8a -> 1.3.9b
* Zikula: 1.2.2 -> 1.2.3

More Update-y Goodness From Fantastico

The following applications have updates available through Fantastico:

* AccountLab Plus: 2.8 r14 -> 2.8 r14
* Coppermine Photo Gallery: 1.4.25 -> 1.4.26
* Gallery: 2.3 -> 2.3.1
* Geeklog: 1.5.1 -> 1.6.1
* OpenX: 2.8.1 -> 2.8.5
* phpCOIN: 1.6.4 -> 1.6.5
* Xoops: 2.3.3b -> 2.4.4
* Zikula: 1.1.1 -> 1.2.2

Apollo Server Migration

To avoid any further hardware issues as happened last night, we will be migrating all accounts to a newly provisioned (shiny!) server tonight. The migration is scheduled to begin on Wednesday, March 10th at 11pm PST (Thursday, March 11th at 2am EST).

There will be some downtime during the migration, though it's not expected to last longer than 15 minutes, but in my experience downtime is normally closer to 30 minutes.

Unscheduled Server Maintenance

The server is temporarily down for unscheduled maintenance. Updates will follow as we get them, thank you for your patience.

Update 1:39am: There was a corruption in one section that has been repaired and now all of the servers are coming back up in an orderly manner.

A reboot of the main server usually takes less than an hour.

Update 1:59am And we're back up.

Fantastico Updates

The following application updates are now available through Fantastico:

* b2evolution: 3.3.1-stable -> 3.3.3-stable
* Crafty Syntax Live Help: 2.16.3 -> 2.16.8
* Drupal: 6.14 -> 6.15
* LimeSurvey: 1.85+ -> 1.87+
* Moodle: 1.9.6 -> 1.9.7
* Nucleus: 3.50 -> 3.51
* phpBB: 3.0.5 -> 3.0.6
* SMF: 1.1.10 -> 1.1.11
* TikiWiki CMS/Groupware: 3.3 -> 4.1
* WordPress: 2.8.6 -> 2.9.2

Invoicing Changes

As of March 1, 2010 invoices will be emailed to you five (5) days prior to your due date, rather than the current ten (10) days prior. Five days will still give you enough time to make any payment funding source changes if needed, but not too far in advance so that manual PayPal payments are put off then forgotten. Payments are always processed on the same day each month, and the emailed invoices are just a reminder.

Any time you need to make changes to how you pay your invoice (credit card, PayPal, bank wire transfer), just submit a ticket to the billing department, and we're happy to make that change for you, usually in a matter of minutes.

A big shout-out and thank you to our clients who always pay on time! Yes, we know who you are. :) The less resources (time and money) spent on accounting issues means there are more resources available to add new services and goodies, and helps keep our overall costs down. Keeping our costs down keeps our rates down, and everyone's happy.

Apollo Server Issues

The Apollo server seems to be having a case of the Monday morning blahs and though all services are up and functional, responses are very sluggish. It's very possible there's a DOS attack of some sort underway, but I won't speculate further until that's been verified.

Technicians are investigating the problem and I'll update when there's more solid information to report, but in the meantime, we're aware of the issue and we're on it.

All Systems Are A Go

The issues related to upgrading to cPanel version 11.25 have been resolved. Oddly enough no client domains nor email accounts were affected - only our domain where the support ticket system resides.

The method in which the ticket system imports ticket emails has been changed to prevent future problems such as this, so that in the unlikely event that the ticket system was inoperable, we would still receive your support and design order requests.

We view client communication as a very important part of what we do, and strive to keep you as informed as possible on all issues relating to your website hosting.

cPanel 11.25 upgrade causes email forward issues

A cPanel upgrade to version 11.25, performed 2 days ago, has introduced a lot of changes to a number of different functions in how Exim, the mail server, routes email. So far the only issue we've seen has been a change to how email forwards are being piped to a program functions. At the moment...they're not.

POP3 and IMAP email functions, as well as regular email forwards (forwarding to another email address) are all working as intended. Unfortunately our support desk and all client portal departments (design services, etc) all use email piping to open support tickets by email. This only effects tickets replied to by email...the client portal's web-interface is working correctly.

Both the NOC and cPanel techs are currently working to resolve this issue, and if it's not resolved by morning a backup plan for the support tickets will be implemented until the issue is fixed. In the meantime, if you need to open a new ticket or reply to an ongoing one, please use the web-based interface.

If you've replied to a ticket by email in the last two days your response may not have been received. As always, every ticket notification contains a direct link to the ticket reply so you can view and reply online.

This bug not withstanding, cPanel's version 11.25 promises to bring a lot of new goodies for users. Once I get past this initial bad first impression - and finish reading the 39-page PDF detailing the newly released changes - appropriate announcements will be made.

Y2K revisited

This morning a Spam Assassin date bug was discovered, causing any email sent since the turn-over of 01/01/2010 to be tagged with the following rule:

FH_DATE_PAST_20XX The date is grossly in the future, and assigned a score of 3.2. Since the default spam score for SpamAssassin is 5.0 (on our server it's set to 6.0, to guard against false positives), this could be a problem.

It's presumably been fixed and will be pushed out in the next sa-update, but we've disabled the rule just the same. If you're not a hosting customer of ours you might want to check to see if your provider has fixed this issue, and/or check your spam folder for false positives.

With the turn of the decade there's been a lot of "Where were you 10 years ago?" memes going around on social networking sites. I didn't really think much about it until this morning, where dealing with this "date bug" has given me a Y2K flashback.

I was invited to the Sun holiday party, though it was scheduled for late January 2000, because everyone was on high alert for Y2K issues. In case you're wondering...the party was incredible, and the Y2K issues were far less than expected, mostly due to proper preparations.

I'd really thought this wouldn't be an issue again until 2038. Guess I was wrong. *smiles*

I hope everyone had a wonderful holiday, and wish you all a prosperous, happy and healthy new year.