CHPC - Notice history

All systems operational

99.94% uptime

Homepage and documentation - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Portal - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

License servers (commercial software) - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Group- and project-specific websites (hosted on VMs) - Operational

99.68% uptime
Feb 2026 · 99.19%Mar · 99.83%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Helpdesk platform - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026
99.61% uptime

HPC clusters - Operational

98.93% uptime
Feb 2026 · 100.0%Mar · 97.87%Apr · 99.06%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Open OnDemand - Operational

99.08% uptime
Feb 2026 · 98.27%Mar · 99.74%Apr · 99.17%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Windows servers - Operational

99.88% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 99.65%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Virtual machines (VMs) - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Computational servers, independent of clusters - Operational

99.54% uptime
Feb 2026 · 100.0%Mar · 99.59%Apr · 99.06%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Storage systems - Operational

99.81% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 99.44%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Data Transfer Nodes (DTNs) - Operational

100.0% uptime
Feb 2026 · 100.0%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026
99.89% uptime

HPC clusters - Operational

99.75% uptime
Feb 2026 · 99.74%Mar · 99.88%Apr · 99.65%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Open OnDemand - Operational

99.87% uptime
Feb 2026 · 99.74%Mar · 99.88%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Windows servers - Operational

99.92% uptime
Feb 2026 · 99.74%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Virtual machines (VMs) - Operational

99.92% uptime
Feb 2026 · 99.74%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Computational servers, independent of clusters - Operational

99.92% uptime
Feb 2026 · 99.74%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Storage systems - Operational

99.92% uptime
Feb 2026 · 99.74%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Data Transfer Nodes (DTNs) - Operational

99.92% uptime
Feb 2026 · 99.74%Mar · 100.0%Apr · 100.0%

Previous page

Next page

Feb 2026
Mar 2026
Apr 2026

Slack - Operational

Notice history

View current status

Apr 2026

APR30
Security-related updates to CHPC systems
CompletedMaintenance142 hours 52 minutes
  • Completed
    May 06, 2026 at 3:52 PMUTC
    Completed
    May 06, 2026 at 3:52 PMUTC

    Most user-facing systems have been rebooted to address the potential security issue. There may be a small number of outstanding systems, but most users will not encounter service interruptions when such systems are updated. Thank you for your patience and understanding. If you have any questions or concerns, please contact the CHPC at helpdesk@chpc.utah.edu.

  • Update
    May 01, 2026 at 9:09 PMUTC
    Update
    May 01, 2026 at 9:09 PMUTC

    Most CHPC systems have been updated and rebooted to mitigate the security issue. CHPC staff are still working on a small number of systems and addressing some maintenance-related issues with individual systems. Thank you for your patience.

  • Update
    May 01, 2026 at 1:21 AMUTC
    Update
    May 01, 2026 at 1:21 AMUTC

    Thank you for your patience with security-related service interruptions on Center for High Performance Computing systems today. All clusters (granite, notchpeak, kingspeak, and lonepeak in the General Environment and redwood in the Protected Environment) have been returned to service by the team at the CHPC. System administrators have also brought virtual machines (VMs) in the Protected Environment back online and are continuing work on VMs in the General Environment. We will continue to provide updates on the status of systems here.

  • Update
    May 01, 2026 at 12:26 AMUTC
    Update
    May 01, 2026 at 12:26 AMUTC

    Virtual machines in the Protected Environment have been returned to service. CHPC system administrators are now working through virtual machines in the General Environment.

  • Update
    May 01, 2026 at 12:23 AMUTC
    Update
    May 01, 2026 at 12:23 AMUTC

    The clusters in the General Environment (granite, notchpeak, kingspeak, and lonepeak) have been returned to service.

  • Update
    April 30, 2026 at 10:58 PMUTC
    Update
    April 30, 2026 at 10:58 PMUTC

    Storage systems and Data Transfer Nodes (DTNs) in the General Environment and Protected Environment have been rebooted and returned to service.

  • Update
    April 30, 2026 at 10:41 PMUTC
    Update
    April 30, 2026 at 10:41 PMUTC

    The redwood cluster in the Protected Environment has been returned to service. CHPC system administrators are continuing work on other systems.

  • Update
    April 30, 2026 at 9:10 PMUTC
    Update
    April 30, 2026 at 9:10 PMUTC

    CHPC system administrators are continuing work to mitigate potential security issues in the General Environment and Protected Environment. Systems in both environments, including cluster nodes and virtual machines, are being rebooted or will be rebooted. User-facing systems in the Citadel environment have been updated and rebooted and are now available.

  • In progress
    April 30, 2026 at 5:49 PMUTC
    In progress
    April 30, 2026 at 5:49 PMUTC

  • Planned
    April 30, 2026 at 5:00 PMUTC
    Planned
    April 30, 2026 at 5:00 PMUTC

    The Center for High Performance Computing is taking proactive, precautionary measures to mitigate a security issue that requires immediate attention. CHPC staff will reboot systems in a rolling fashion throughout the day. This will result in access and service disruptions. Running jobs and active sessions on CHPC systems, including jobs on compute nodes and sessions on interactive nodes and Open OnDemand, may be lost as systems are rebooted.

    The CHPC will provide additional information and updates throughout the day as services are brought back online. We apologize for the inconvenience and appreciate your patience and support.

Some jobs are failing to start, yielding errors in sbatch or srun
ResolvedMajor outage6 hours 42 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    The issues we are seeing stems with inability to write to the "sys" branches, where applications, SLURM information, and other things are being written. We are still investigating the reasons for that.

  • Investigating
    UTC
    Investigating

    CHPC system administrators are aware of issues users have reported when starting jobs and are currently investigating.

APR17
UETN network upgrade affecting CHPC DTNs overnight from April 16 to April 17
CompletedMaintenance4 hours
  • Completed
    April 17, 2026 at 9:00 AMUTC
    Completed
    April 17, 2026 at 9:00 AMUTC
    Maintenance has completed successfully
  • In progress
    April 17, 2026 at 5:00 AMUTC
    In progress
    April 17, 2026 at 5:00 AMUTC
    Maintenance is now in progress
  • Planned
    April 17, 2026 at 5:00 AMUTC
    Planned
    April 17, 2026 at 5:00 AMUTC

    UETN will be updating its core router that affects CHPC's DMZ (campus firewall bypass for faster data transfers). Services that use the DMZ such as CHPC's DTNs and several project data servers will experience degraded or no network connection.

Issues with module system, affecting logins and module loads
ResolvedDegraded performance40 minutes
  • Resolved
    UTC
    Resolved

    CHPC staff have fixed an issue with the module system, which should resolve any issues with logins or module loads. If you continue to encounter issues, please contact us at helpdesk@chpc.utah.edu.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Significant number of nodes on redwood cluster (Protected Environment) are down
ResolvedPartial outage3 hours 48 minutes
  • Resolved
    UTC
    Resolved

    CHPC system administrators have restored the nodes on the redwood cluster to service. Thank you for your patience. If you encounter further issues, please contact us at helpdesk@chpc.utah.edu.

  • Investigating
    UTC
    Investigating

    The CHPC is aware that many nodes on the redwood cluster (Protected Environment) are currently down. System administrators are investigating.

Mar 2026

Slurm not responding in Protected Environment (Redwood)
ResolvedMajor outage53 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    The CHPC has become aware that Slurm is not talking to Redwood and we are working to fix this.

Sites hosted on home.chpc.utah.edu are not reachable
ResolvedMajor outage1 hour 13 minutes
  • Resolved
    UTC
    Resolved

    The issue with home.chpc.utah.edu has been resolved and user- and group-specific pages are being served again. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu.

  • Investigating
    UTC
    Investigating

    Websites hosted on home.chpc.utah.edu, including user- and group-specific sites, are not reachable. The CHPC is aware of this issue and investigating the cause.

Filesystem issue requiring journal replay; some General Environment group spaces inaccessible
ResolvedPartial outage8 minutes
  • Resolved
    UTC
    Resolved

    CHPC staff have brought the filesystem back online.

  • Investigating
    UTC
    Investigating

    Excessive usage on a General Environment filesystem required it to be taken offline temporarily.

Many systems on the granite cluster went offline (lost power) at approximately 4:45 p.m.
ResolvedMajor outage15 hours 18 minutes
  • Resolved
    UTC
    Resolved

    Systems on granite returned to service shortly after the outage yesterday afternoon. CHPC staff have moved a switch to two separate power distribution units to prevent similar incidents in the future.

  • Monitoring
    UTC
    Monitoring

    CHPC and DDC staff have identified the issue and restored power to systems. Systems on the granite cluster should be back online or in the process of coming back online. CHPC staff will continue to monitor the cluster.

  • Investigating
    UTC
    Investigating

    The CHPC is investigating an issue with many of the systems on the granite cluster. Initial reports suggest there may be an issue with power distribution to login nodes, networking infrastructure, and core services.

Several nodes on notchpeak lost power
ResolvedPartial outage20 hours 11 minutes
  • Resolved
    UTC
    Resolved

    Nodes are online. CHPC staff have drained a limited set of nodes (preventing new jobs from starting but not affecting currently running jobs) to rebalance power.

  • Monitoring
    UTC
    Monitoring

    CHPC staff on-site at the data center brought most systems back online. Staff will rebalance affected systems among power distribution units to prevent similar issues in the future. One notchpeak node remains offline while staff work on power distribution.

  • Investigating
    UTC
    Investigating

    Several nodes on the notchpeak cluster lost power on the afternoon of March 5. This incident affected notch366, notch452, notch472, notch473, notch474, notch475, notch476, notch477, notch478, notch479, notch480, notch481, notch482, notch483, notch484, notch485, notch486, notch487, notch488, notch489, notch490, notch491, notch492, notch493, notch494, notch495, notch496, notch497, notch498, notch499, notch500, notch501, notchpeak32, notchpeak33, and notchpeak34.

Feb 2026

Issues with Protected Environment resources
ResolvedMajor outage6 hours 40 minutes
  • Resolved
    UTC
    Resolved

    CHPC staff have determined that the issue with the Protected Environment has been resolved. Systems have remained accessible since the update earlier this afternoon. If you continue to encounter issues with a resource in the Protected Environment, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience.

  • Update
    UTC
    Update

    While we have identified the cause of the problems, and most services in the PE are available, individual systems may still have lingering problems due to the previous network loss to the file systems. If your particular service has issues, please, contact helpdesk@chpc.utah.edu.

  • Monitoring
    UTC
    Monitoring

    Issues in the Protected Environment are attributable to packet loss. CHPC staff have stopped replication between the General Environment VAST and Protected Environment VAST, which significantly reduced the packet loss. Services appear to be responsive again. Logins and services in the PE should begin working. CHPC staff will continue to monitor the situation.

  • Update
    UTC
    Update

    CHPC staff are continuing to investigate the issue. Based on user reports, this incident's impact is being updated to an outage rather than degraded performance.

  • Investigating
    UTC
    Investigating

    The CHPC is aware of issues with the Protected Environment, leading to degraded performance. Staff are working to identify and correct the issue. We will provide updates as we learn more.

Issues with web server, affecting access to some CHPC-managed websites
ResolvedPartial outage1 hour 45 minutes
  • Resolved
    UTC
    Resolved

    CHPC system administrators restarted a service on the web server, which resolved the issue.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Issues with web server, affecting access to some CHPC-managed websites
ResolvedPartial outage5 hours 30 minutes
  • Resolved
    UTC
    Resolved

    CHPC system administrators restarted a service on the web server, which resolved the issue.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Issues with web server, affecting access to some CHPC-managed websites
ResolvedPartial outage2 hours 45 minutes
  • Resolved
    UTC
    Resolved

    CHPC system administrators restarted a service on the web server, which resolved the issue.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Significant delays when logging in to Protected Environment (identified approximately 2:30 p.m. on Friday, February 13)
ResolvedDegraded performance54 minutes
  • Resolved
    UTC
    Resolved

    CHPC network and system administrators have identified and fixed the issue. The problem is hardware-related; staff have implemented a workaround and will engage the hardware vendor.

  • Investigating
    UTC
    Investigating

    The CHPC is aware of significant delays encountered by users when logging in to systems in the Protected Environment, including SSH logins and Open OnDemand. We are investigating the issue.

Previous

Feb 2026 to Apr 2026

Next