Postmortem -
Read details
Nov 12, 14:11 EST
Resolved -
Account recoveries have been completed. While there is some additional work to be done, and beginning new jetbackup backups the incident will be closed. Please contact support with any more specific questions or needed updates.
Nov 12, 14:10 EST
Monitoring -
Most accounts have been restored. Staff are going through sites to ensure php compatability on their site as the system has been restored to the latest OS which has a newer default php version.
Nov 11, 14:51 EST
Update -
The backup data has been fully copied over for all accounts allowing the remaining accounts to restore in parallel and faster. Exim will begin accepted the queued messages in 1 hour.
All accounts are expected to be restored over the next few hours.
Nov 11, 01:01 EST
Update -
Restores continue to run. While many accounts are have restored there are still a number in progress. The restore includes
* copying the data over
* extracting some data like mysql
* restoring
* setting the original ip
However we expect the restore to pick up more speed in the coming hours.
Nov 10, 19:36 EST
Identified -
The replacement server is online, with the ips moved and account restores have begun.
To prevent mail loss port 25 will be closed until most accounts are restored
Nov 10, 11:51 EST
Update -
A parallel restore from the jetbackup backups is being prepped on a new server build as a restore from backups appears to be the quickest resolution.
Nov 10, 09:31 EST
Update -
The webhosting2040 system raid lost a drive in a raid 10 system, and locked up and went read only when the drive failed. The reported drive has been replaced and a file system check is running on the system. This is a lengthy check and continues to run checking for any possible file corruption.
Nov 10, 05:17 EST
Investigating -
We are investigating an outage on the webhosting2040 server, that appears hardware raid related.
Nov 10, 00:16 EST