Outgoing Email Issues [Resolved]

We're investigating an issue with outgoing email from the server. All incoming email is being queued (held for delivery), but one of the processes that "sends" email is stopped and is refusing to restart.

We are working diligently on this issue and expect to have it fixed very soon. I suspect this was caused by an update from last night, as other admins have reported the same issue in the cPanel support forums. More updates as they are available.

UPDATE 5:47PM All issues have been resolved. Last night cPanel and PHP were upgraded (the PHP upgrade was a small security update). On cPanel servers PHP is upgraded through a cPanel process called EasyApache.

Something in EasyApache apparently changed the permissions on several directories in user accounts, which I caught and fixed at the time. It also changed the permissions on the /tmp directory, which just about everything uses to temporarily write to while it's processing (think of it as a "working directory"), including email. The permissions need to be set as to securely allow other processes on the server to write to the tmp directory.

All email that was being held for processing should be on it's way to you now. We apologize for this issue, we know how important it is to receive email in a timely manner.

Server issues

We're investigating an issue with the server, details as they become available.

UPDATE 8:29 PM We're back up. If I had to guess (which I do have to, because the data center isn't being forthcoming about WHAT is going on there), I would guess that this was related to the ongoing network upgrades they're doing. But in my opinion, that's no excuse for the amount of downtime we've experienced in the last few months, and certainly not without proper notice.

UPDATE 10:20 PM We've migrated to a new hardware cluster to avoid any further issues. If this doesn't solve the issue, then the next step will be migrating to a different data center.