CHPC - Notice history

CHPC staff have brought the filesystem back online.

Systems on granite returned to service shortly after the outage yesterday afternoon. CHPC staff have moved a switch to two separate power distribution units to prevent similar incidents in the future.

CHPC and DDC staff have identified the issue and restored power to systems. Systems on the granite cluster should be back online or in the process of coming back online. CHPC staff will continue to monitor the cluster.

Nodes are online. CHPC staff have drained a limited set of nodes (preventing new jobs from starting but not affecting currently running jobs) to rebalance power.

CHPC staff on-site at the data center brought most systems back online. Staff will rebalance affected systems among power distribution units to prevent similar issues in the future. One notchpeak node remains offline while staff work on power distribution.

CHPC staff have determined that the issue with the Protected Environment has been resolved. Systems have remained accessible since the update earlier this afternoon. If you continue to encounter issues with a resource in the Protected Environment, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience.

While we have identified the cause of the problems, and most services in the PE are available, individual systems may still have lingering problems due to the previous network loss to the file systems. If your particular service has issues, please, contact helpdesk@chpc.utah.edu.

Issues in the Protected Environment are attributable to packet loss. CHPC staff have stopped replication between the General Environment VAST and Protected Environment VAST, which significantly reduced the packet loss. Services appear to be responsive again. Logins and services in the PE should begin working. CHPC staff will continue to monitor the situation.

CHPC staff are continuing to investigate the issue. Based on user reports, this incident's impact is being updated to an outage rather than degraded performance.

CHPC system administrators restarted a service on the web server, which resolved the issue.

CHPC system administrators restarted a service on the web server, which resolved the issue.

CHPC system administrators restarted a service on the web server, which resolved the issue.

CHPC network and system administrators have identified and fixed the issue. The problem is hardware-related; staff have implemented a workaround and will engage the hardware vendor.

Developers have corrected the issue. The "mychpc batch" command and Open OnDemand job submission options are now working as expected.

CHPC storage administrators restored the file server. Services are back online.

Systems in the Protected Environment (PE) are available to users and issues should now be resolved.

The environment is operating with a single switch and three of four CNodes in its VAST storage system. (These should not affect the functionality of the PE, though they will need to be addressed at a later date.) All virtual machines are back online.

If you encounter any issues with systems in the PE, please let us know. We are grateful for your patience and support as we worked to identify and address issues.

System and network administrators believe they have identified an issue with a network interface card on storage infrastructure. They are in contact with technical support staff from the vendor.