[SERVERS] Cassia server issues

The cassia server has begun to spontaneously reboot itself at random times. As yet, we have not been able to determine a reason for this: all hardware diagnostics return cleanly, there are no fan failures or heat-related issues, and all filesystem operations are normal. However, because the problem appears to be getting worse, sometimes with ...

[SERVERS] Monet controller/drive issue

We are currently experiencing issues with the backplane and primary drive on monet. This is resulting in short reads/timeouts on reading data, and causing the server to become unresponsive. We are working as quickly as possible to resolve this; however, it appears the server may need to be taken offline for component replacement and/or rebuild. ...

[NETWORK] Network issues

We are currently tracking a network issue that is causing all servers in Jax1 to be unreachable. We are working with the NOC to determine the cause and rectify the issue as quickly as possible. Thank you for your patience as we troubleshoot and resolve this issue. Edit: Currently seeing recovery within the network. Updates ...

NETWORK: Slow/timeout issue

We are currently tracking a network issue causing slow/timeout access. Updates to follow. UPDATE: Based on reports from the NOC, there was apparently an attack on an IP within our allocation – an IP attached to an established server that has been online for quite some time and has not taken any new sites in ...

GENERAL: Client area maintenance

The client area will be in maintenance mode for about five minutes for a quick update. UPDATE: The maintenance has been completed. Thanks for your patience.

[GENERAL] Client area maintenance

This evening beginning at 9 PM Eastern, the client area will be briefly unavailable as we perform some minor maintenance. Access to the system to review profile details, place new orders, or submit tickets directly will be affected; tickets submitted via email will not be affected by this maintenance. The expected impact time is approximately ...

[GENERAL] Client area application offline

We are currently experiencing a RAID issue on the server handling the client application. The client area is unavailable while we replace the failing drive in the array. We will return it to service as soon as possible. Update: the issue is larger than initially believed, as there are two drives failing. We will be ...

[NETWORK] Switch migration

Today at 3 PM (Eastern time), the NOC will be performing a switch cutover that will impact our connectivity. The impact is expected to be brief (a few minutes) as components related to this change have been tested this week and other configuration items have already been put in place. The primary impact will be ...

[SERVERS] Gaia emergency moves

The gaia server is experiencing a variety of issues, leading to high loads, services failing, and general annoyances. Based on the logging, it appears to be a combination of hardware issues creating this storm, which is not of the temporary polar vortex variety. This morning we will begin emergency migrations from this server to other ...

[MAINTENANCE] Kernel updates

We will be rolling out kernel and miscellaneous updates to all servers beginning the evening of Thursday, November 28, 2013, and continuing throughout the holiday weekend until the updates have been completed. Total impact time per server will be 1-3 minutes for the reboot necessary to finalize the updates. If you have any questions, please ...