Update -
A final change over will begin today to the new hardware.
Sep 13, 12:05 EDT
Investigating -
A data copy on the openvz125 legacy openvz system is being done to new hardware. Currently the system is copying over zfs snapshots, and is copying sept 4th to today which is a few 100GB of data. Once completed another sync will run then a final change over scheduled. The final change over will have the vm's go offline, to do a final incremental copy.
Resolved -
This incident has been resolved.
Sep 12, 17:59 EDT
Identified -
The issue has been identified, queued emails from relay1 cluster are now delivering. Once this has caught up the cluster will be added back into service for new emails.
Sep 12, 17:44 EDT
Investigating -
We are investigating queued emails with retry or connection errors on the relay1 cluster. No new email will go to this cluster at this time during the investigation, and relay0,2,3 and la-relay0 clusters remain active.
Sep 12, 17:15 EDT
Update -
Unfortunatly no recovery of this system has been possible with attempts from two recovery vendors. Users affected have been given credits.
Sep 12, 16:16 EDT
Update -
We are awaiting further information from the recovery vendor. The drives were shipped overnight , work has already started and an initial file structure for the data has been identified.
Jul 20, 15:20 EDT
Identified -
Openvz31 drives will be sent to our recovery vendor.
Jul 15, 17:04 EDT
Investigating -
Openvz31 is currently offline in Teb5 / Coresite which was connected to UPS A2 which suffered a failure last night. Multiple drives in the zfs array are reporting errors.
Jul 15, 14:04 EDT