[SERVERS] Drive replacements: various

This afternoon we will be replacing drives in the following servers: whistler, baber, parsley, and chronos. The total impact time per server will be about five minutes, to power down each server, make the drive swap, and bring each back online. During this time, services for the affected servers will not be available. If you ...

[GENERAL] Client area updates

On Saturday, May 6, we will be updating the client area. During the update, the client area will be unavailable for 5-10 minutes. During this time, direct logins will not be possible; however, any tickets can be submitted via email as usual. If you have any questions related to this maintenance, please contact support via ...

[SERVERS] Kernel updates, May 5-7

Beginning the evening of May 5 and continuing throughout the weekend, we will be rolling out kernel updates to the servers. Total impact time for each server will be 2-5 minutes for the required reboots to finalize the kernel updates. If you have any questions related to this maintenance, please contact support via the usual ...

[SERVERS] Hermes issues/emergency migrations

We are currently seeing rather significant and intermittent issues with the hermes server. Those issues include spiking loads and errors in services that fail as a result, then have to restart. In several instances, we have had to do uncontrolled reboots to bring the server back in line. This server has been in production for ...

[MAINTENANCE] Kernel, mail, client area updates

It’s time for another round of kernel updates for the servers within the network. We will be starting those this evening and continue throughout the weekend. As the number of servers in the network continues to grow, if this maintenance is not completed by late Sunday evening, we will be completing any servers that still ...

[SERVERS] Dali issues

Dali was not just surreal in real life: Dali’s namesake server is generating hardware-related notices in the system logs that are creating sustained load averages with occasional spikes that result in various services shutting down and restarting. The issues we are seeing are not in line with what we would expect to see with the ...

[SERVERS] Kernel updates, Dec 12-14, 2014

We will be rolling out kernel updates beginning Friday evening, December 12, 2014, and continuing throughout the weekend, ending on the evening of Sunday, December 14, 2014. Total impact time per server will be 1-2 minutes for reboots to finalize the new kernel builds. If you have any questions related to this routine maintenance, please ...

[SERVERS] Cratos issue

Currently investigating an issue with the cratos server, rendering it inaccessible. Updates to follow as available. Update: based on a review and testing, it appears there was a surge or short in the server. This has resulted in the primary data drive being damaged, and more specifically the boot loader, along with at least one ...

[NETWORK] High latency/timeout issue

We are currently tracking a latency/timeout connectivity issue. Update 1: Issue has been traced to a DDOS attack. Update 2: Target IP has been nullrouted. update 3: Issue has been resolved. Affected user(s) will be notified separately.

[NETWORK] High traffic warning

We are expecting higher than usual traffic this evening, although evenings are generally lower in traffic than is the case during US business hours, related to tonight’s announcement in Ferguson. We will be monitoring more closely than usual to address any issues that may arise. If you have any questions related to this announcement, please ...