CHPC Status - Incident history https://uofu-chpc.instatus.com CHPC Mon, 5 Oct 2026 14:00:00 +0000 Critical security Linux OS update on Monday, October 5 Type: Maintenance Affected Components: HPC clusters, Virtual machines (VMs), Open OnDemand, Virtual machines (VMs), Storage systems, Data Transfer Nodes (DTNs), Storage systems, Data Transfer Nodes (DTNs), HPC clusters, Open OnDemand, Computational servers, independent of clusters, Computational servers, independent of clusters Sep 22, 21:55:52 GMT+0 - Identified - Center for High Performance Computing (CHPC) will be undergoing a system-wide downtime on Monday, October 5th. This maintenance will impact the General Environment clusters, Protected Environment clusters, standalone servers and virtual machines. The sys branch and app repo in the GE and PE may be effected. If you have one of the websites on www.misc or www.cgi you may also be impacted. We are performing this downtime to apply critical security patches addressing multiple CVEs. Protecting your user data and research is our highest priority, making these updates essential to maintaining a secure environment for everyone. Type: Maintenance

Affected Components: , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 22</var>, <var data-var='time'>21:55:52</var> GMT+0</small><br><strong>Identified</strong> - Center for High Performance Computing (CHPC) will be undergoing a system-wide downtime on Monday, October 5th. This maintenance will impact the General Environment clusters, Protected Environment clusters, standalone servers and virtual machines. The sys branch and app repo in the GE and PE may be effected. If you have one of the websites on www.misc or www.cgi you may also be impacted. We are performing this downtime to apply critical security patches addressing multiple CVEs. Protecting your user data and research is our highest priority, making these updates essential to maintaining a secure environment for everyone..</p> ]]> Mon, 5 Oct 2026 14:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmud7qwen1y9w1npcam22t9g8 https://uofu-chpc.instatus.com/maintenance/cmud7qwen1y9w1npcam22t9g8 Issues with Narwhal (Windows server in Protected Environment) on September 24 Type: Incident Duration: 1 hour and 2 minutes Affected Components: Windows servers Sep 24, 16:57:31 GMT+0 - Investigating - The CHPC is aware of issues with the Narwhal server, which are a result of excessive load on the system. We recommend using pe-rds01 (see <https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php#pe-rds01> for details) while Narwhal is inaccessible. Thank you for your patience. Sep 24, 18:00:00 GMT+0 - Resolved - The narwhal server is available again; the system ran out of memory and needed to be rebooted. We recommend moving new Protected Environment projects to the pe-rds01 server, described at <https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php>. If you have any questions or concerns, please don't hesitate to reach out to the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 1 hour and 2 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>16:57:31</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is aware of issues with the Narwhal server, which are a result of excessive load on the system. We recommend using pe-rds01 (see <https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php#pe-rds01> for details) while Narwhal is inaccessible. Thank you for your patience..</p> <p><small>Sep <var data-var='date'> 24</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Resolved</strong> - The narwhal server is available again; the system ran out of memory and needed to be rebooted. We recommend moving new Protected Environment projects to the pe-rds01 server, described at <https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php>. If you have any questions or concerns, please don't hesitate to reach out to the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Thu, 24 Sep 2026 16:57:31 +0000 https://uofu-chpc.instatus.com/incident/cmufryx5u0bml1mrgdvqno9s8 https://uofu-chpc.instatus.com/incident/cmufryx5u0bml1mrgdvqno9s8 Issues with connections to CHPC-hosted websites and virtual machines from outside of the University of Utah network Type: Incident Duration: 7 days, 3 hours and 50 minutes Affected Components: Group- and project-specific websites (hosted on VMs) Sep 16, 21:23:50 GMT+0 - Identified - Some virtual machines and associated websites or services are now accessible following an emergency change request. Others remain inaccessible from outside of the University of Utah network and will likely be available following our campus partners' scheduled maintenance and planned implementation of firewall rules tonight. Sep 23, 16:50:10 GMT+0 - Resolved - Please contact [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu) if you notice any continued connectivity issues to CHPC services. Sep 16, 13:00:00 GMT+0 - Investigating - The CHPC team is aware of issues with connections to some CHPC-hosted infrastructure, including virtual machines and associated websites and web services, from external networks. This is related to changes to University of Utah firewall rules, not CHPC infrastructure, and network and system administrators at the CHPC are in communication with campus partners to resolve issues as quickly as possible. Where feasible, we recommend connecting to the University of Utah VPN to access systems and services that are affected by firewall changes. We recognize that this may not be feasible for systems open to the public or collaborators at other institutions, and we are working to resolve access issues with our campus partners. Thank you for your understanding and patience. Sep 16, 21:17:16 GMT+0 - Identified - Campus partners plan to update firewall rules during scheduled maintenance tonight. In anticipation of the firewall rule changes, CHPC network administrators proactively submitted exception requests. We expect the requested exceptions to take effect with tonight's updates, which should resolve most access issues, though some systems may experience service disruptions or remain unavailable. We will address issues as quickly as we can. Thank you for your patience. Type: Incident

Duration: 7 days, 3 hours and 50 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>21:23:50</var> GMT+0</small><br><strong>Identified</strong> - Some virtual machines and associated websites or services are now accessible following an emergency change request. Others remain inaccessible from outside of the University of Utah network and will likely be available following our campus partners' scheduled maintenance and planned implementation of firewall rules tonight..</p> <p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>16:50:10</var> GMT+0</small><br><strong>Resolved</strong> - Please contact [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu) if you notice any continued connectivity issues to CHPC services..</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC team is aware of issues with connections to some CHPC-hosted infrastructure, including virtual machines and associated websites and web services, from external networks. This is related to changes to University of Utah firewall rules, not CHPC infrastructure, and network and system administrators at the CHPC are in communication with campus partners to resolve issues as quickly as possible. Where feasible, we recommend connecting to the University of Utah VPN to access systems and services that are affected by firewall changes. We recognize that this may not be feasible for systems open to the public or collaborators at other institutions, and we are working to resolve access issues with our campus partners. Thank you for your understanding and patience..</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>21:17:16</var> GMT+0</small><br><strong>Identified</strong> - Campus partners plan to update firewall rules during scheduled maintenance tonight. In anticipation of the firewall rule changes, CHPC network administrators proactively submitted exception requests. We expect the requested exceptions to take effect with tonight's updates, which should resolve most access issues, though some systems may experience service disruptions or remain unavailable. We will address issues as quickly as we can. Thank you for your patience..</p> ]]> Wed, 16 Sep 2026 13:00:00 +0000 https://uofu-chpc.instatus.com/incident/cmu4efy1t006g0wph8k8j4ypc https://uofu-chpc.instatus.com/incident/cmu4efy1t006g0wph8k8j4ypc Issues with access to CHPC Portal, including forms and user management features Type: Incident Duration: 1 day, 3 hours and 52 minutes Affected Components: Portal Sep 15, 17:42:46 GMT+0 - Investigating - We are currently investigating reports of issues with connections to the CHPC Portal and how it may relate to University of Utah campus network policy changes that began yesterday. **If you encounter issues with the CHPC Portal or other CHPC-managed services, we recommend connecting to the University of Utah VPN**, which may resolve network-related issues. Thank you for your patience. Sep 16, 21:34:26 GMT+0 - Resolved - The CHPC Portal is now accessible from outside the University of Utah network following an emergency change to campus firewall rules implemented by our partners. This is _not_ an all-clear message for all virtual machines, websites, or services typically served outside of the university network; some systems remain inaccessible. Please see details and updates on other systems at <https://uofu-chpc.instatus.com/cmu4efy1t006g0wph8k8j4ypc>. Type: Incident

Duration: 1 day, 3 hours and 52 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 15</var>, <var data-var='time'>17:42:46</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating reports of issues with connections to the CHPC Portal and how it may relate to University of Utah campus network policy changes that began yesterday. **If you encounter issues with the CHPC Portal or other CHPC-managed services, we recommend connecting to the University of Utah VPN**, which may resolve network-related issues. Thank you for your patience..</p> <p><small>Sep <var data-var='date'> 16</var>, <var data-var='time'>21:34:26</var> GMT+0</small><br><strong>Resolved</strong> - The CHPC Portal is now accessible from outside the University of Utah network following an emergency change to campus firewall rules implemented by our partners. This is _not_ an all-clear message for all virtual machines, websites, or services typically served outside of the university network; some systems remain inaccessible. Please see details and updates on other systems at <https://uofu-chpc.instatus.com/cmu4efy1t006g0wph8k8j4ypc>..</p> ]]> Tue, 15 Sep 2026 17:42:46 +0000 https://uofu-chpc.instatus.com/incident/cmu2ymg6y013l1mtioi7zyl7c https://uofu-chpc.instatus.com/incident/cmu2ymg6y013l1mtioi7zyl7c Brief maintenance on Open OnDemand at 8:30 p.m. MDT on Monday, September 14 Type: Maintenance Duration: 30 minutes Affected Components: Open OnDemand, Open OnDemand Sep 11, 15:18:38 GMT+0 - Identified - The CHPC team will perform scheduled maintenance on Open OnDemand servers in the General and Protected Environments at 8:30 p.m. MDT on Monday, September 14\. There will be a brief outage of approximately 15 minutes. Jobs will continue to run during maintenance, though users may be disconnected from active sessions; Open OnDemand will be unavailable briefly as updates are applied. Sep 15, 02:30:01 GMT+0 - Identified - Maintenance is now in progress Sep 15, 03:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 30 minutes

Affected Components: ,

<p><small>Sep <var data-var='date'> 11</var>, <var data-var='time'>15:18:38</var> GMT+0</small><br><strong>Identified</strong> - The CHPC team will perform scheduled maintenance on Open OnDemand servers in the General and Protected Environments at 8:30 p.m. MDT on Monday, September 14\. There will be a brief outage of approximately 15 minutes. Jobs will continue to run during maintenance, though users may be disconnected from active sessions; Open OnDemand will be unavailable briefly as updates are applied..</p> <p><small>Sep <var data-var='date'> 15</var>, <var data-var='time'>02:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Sep <var data-var='date'> 15</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Tue, 15 Sep 2026 02:30:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmtx3pof6010i0wqomutyoy91 https://uofu-chpc.instatus.com/maintenance/cmtx3pof6010i0wqomutyoy91 Issue with Slurm in Protected Environment, affecting new job submissions and accounting features on redwood cluster Type: Incident Duration: 2 hours and 55 minutes Affected Components: HPC clusters Sep 14, 12:30:00 GMT+0 - Investigating - CHPC system administrators are aware of an issue with Slurm in the Protected Environment. The issue affects new job submissions; running jobs will continue without issue. Sep 14, 15:25:00 GMT+0 - Resolved - CHPC system administrators have brought the affected Slurm service back online. The redwood cluster should be available again. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience. Type: Incident

Duration: 2 hours and 55 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>12:30:00</var> GMT+0</small><br><strong>Investigating</strong> - CHPC system administrators are aware of an issue with Slurm in the Protected Environment. The issue affects new job submissions; running jobs will continue without issue..</p> <p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>15:25:00</var> GMT+0</small><br><strong>Resolved</strong> - CHPC system administrators have brought the affected Slurm service back online. The redwood cluster should be available again. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience..</p> ]]> Mon, 14 Sep 2026 12:30:00 +0000 https://uofu-chpc.instatus.com/incident/cmu1ecb49003013rs6gnwmm05 https://uofu-chpc.instatus.com/incident/cmu1ecb49003013rs6gnwmm05 Issue with commercial software licenses, including FastX, Abaqus, and IDL, on September 10 Type: Incident Duration: 3 hours and 29 minutes Affected Components: HPC clusters, HPC clusters, License servers (commercial software), HPC clusters Sep 10, 13:30:05 GMT+0 - Investigating - The CHPC is aware of an issue with FastX licenses, affecting access to the General Environment, Protected Environment, and Citadel. Connections with SSH and Open OnDemand remain available. Thank you for your patience with this issue; we will provide updates as soon as we can. Sep 10, 16:30:43 GMT+0 - Identified - The CHPC team has identified a license server as the issue. Some commercial software, such as Abaqus and IDL, is currently unavailable in addition to the FastX issues identified earlier. We are working to resolve this as quickly as we can. Sep 10, 16:59:08 GMT+0 - Resolved - The CHPC team has resolved the issue with the license server. Commercial software, including FastX and several scientific applications, is now available again. If you continue to encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 3 hours and 29 minutes

Affected Components: , , ,

<p><small>Sep <var data-var='date'> 10</var>, <var data-var='time'>13:30:05</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is aware of an issue with FastX licenses, affecting access to the General Environment, Protected Environment, and Citadel. Connections with SSH and Open OnDemand remain available. Thank you for your patience with this issue; we will provide updates as soon as we can..</p> <p><small>Sep <var data-var='date'> 10</var>, <var data-var='time'>16:30:43</var> GMT+0</small><br><strong>Identified</strong> - The CHPC team has identified a license server as the issue. Some commercial software, such as Abaqus and IDL, is currently unavailable in addition to the FastX issues identified earlier. We are working to resolve this as quickly as we can..</p> <p><small>Sep <var data-var='date'> 10</var>, <var data-var='time'>16:59:08</var> GMT+0</small><br><strong>Resolved</strong> - The CHPC team has resolved the issue with the license server. Commercial software, including FastX and several scientific applications, is now available again. If you continue to encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Thu, 10 Sep 2026 13:30:05 +0000 https://uofu-chpc.instatus.com/incident/cmtvmxxd606fp1mlflkph1ihk https://uofu-chpc.instatus.com/incident/cmtvmxxd606fp1mlflkph1ihk Ongoing issues with access to the redwood cluster (Protected Environment) Type: Incident Duration: 1 hour and 6 minutes Affected Components: HPC clusters, Open OnDemand, Computational servers, independent of clusters Aug 14, 15:30:34 GMT+0 - Investigating - CHPC system administrators are aware of ongoing issues with access to the redwood cluster in the Protected Environment. The issues are related to an outage of a core system that supports the cluster. System administrators were able to recover the system after an outage earlier this morning, but it subsequently failed again. System administrators are performing on-site maintenance to reseat hardware (CPUs, memory, and disks) and further diagnose the issue. At this time, the administrators do not have an estimate for when the system will be available again, but we will provide updates as we learn more. Thanks again for your patience. Aug 14, 16:36:31 GMT+0 - Resolved - CHPC system administrators have recovered the affected system by swapping hardware, and the redwood cluster is now available again. We will monitor the situation closely. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 1 hour and 6 minutes

Affected Components: , ,

<p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>15:30:34</var> GMT+0</small><br><strong>Investigating</strong> - CHPC system administrators are aware of ongoing issues with access to the redwood cluster in the Protected Environment. The issues are related to an outage of a core system that supports the cluster. System administrators were able to recover the system after an outage earlier this morning, but it subsequently failed again. System administrators are performing on-site maintenance to reseat hardware (CPUs, memory, and disks) and further diagnose the issue. At this time, the administrators do not have an estimate for when the system will be available again, but we will provide updates as we learn more. Thanks again for your patience..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>16:36:31</var> GMT+0</small><br><strong>Resolved</strong> - CHPC system administrators have recovered the affected system by swapping hardware, and the redwood cluster is now available again. We will monitor the situation closely. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Fri, 14 Aug 2026 15:30:34 +0000 https://uofu-chpc.instatus.com/incident/cmst3t61r0abc0ko546519tnq https://uofu-chpc.instatus.com/incident/cmst3t61r0abc0ko546519tnq Issues with access to the redwood cluster (Protected Environment) Type: Incident Duration: 6 minutes Affected Components: HPC clusters, Open OnDemand, Computational servers, independent of clusters Aug 14, 14:05:57 GMT+0 - Investigating - CHPC system administrators are currently investigating an issue with access to the redwood cluster in the Protected Environment following user reports. We will provide updates as we learn more. Thank you for your patience. Aug 14, 14:11:35 GMT+0 - Resolved - System administrators have identified and corrected the issue. The redwood cluster is responsive and logins are working again. If you encounter further issues, please don't hesitate to contact us at helpdesk@chpc.utah.edu. Type: Incident

Duration: 6 minutes

Affected Components: , ,

<p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>14:05:57</var> GMT+0</small><br><strong>Investigating</strong> - CHPC system administrators are currently investigating an issue with access to the redwood cluster in the Protected Environment following user reports. We will provide updates as we learn more. Thank you for your patience..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>14:11:35</var> GMT+0</small><br><strong>Resolved</strong> - System administrators have identified and corrected the issue. The redwood cluster is responsive and logins are working again. If you encounter further issues, please don't hesitate to contact us at helpdesk@chpc.utah.edu..</p> ]]> Fri, 14 Aug 2026 14:05:57 +0000 https://uofu-chpc.instatus.com/incident/cmst0sd8c08ur0kqxksf0ph1b https://uofu-chpc.instatus.com/incident/cmst0sd8c08ur0kqxksf0ph1b Slurm not responding on Notchpeak Type: Incident Affected Components: HPC clusters, Open OnDemand Aug 12, 13:25:06 GMT+0 - Resolved - This incident has been resolved. Aug 12, 14:36:04 GMT+0 - Postmortem - Slurm was restarted on Notchpeak and is back in a healthy state. Jobs should start on Notchpeak as expected now and Slurm queries on Notchpeak should return the expected output. Type: Incident

Affected Components: ,

<p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>13:25:06</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 12</var>, <var data-var='time'>14:36:04</var> GMT+0</small><br><strong>Postmortem</strong> - Slurm was restarted on Notchpeak and is back in a healthy state. Jobs should start on Notchpeak as expected now and Slurm queries on Notchpeak should return the expected output..</p> ]]> Wed, 12 Aug 2026 13:25:06 +0000 https://uofu-chpc.instatus.com/incident/cmsq4g46f00461aryn6tgbfmh https://uofu-chpc.instatus.com/incident/cmsq4g46f00461aryn6tgbfmh Open OnDemand (ondemand.chpc.utah.edu) under heavy load; consider using ondemand-class.chpc.utah.edu Type: Incident Duration: 6 days, 16 hours and 30 minutes Affected Components: Open OnDemand Aug 4, 22:30:46 GMT+0 - Identified - The CHPC has received reports of issues connecting to ondemand.chpc.utah.edu. The Open OnDemand server in the General Environment is currently under heavy load from the aggregate of user sessions. A second server, ondemand-class.chpc.utah.edu, is available to use if you are encountering issues with the primary Open OnDemand server. We apologize for the inconvenience. Aug 11, 15:01:09 GMT+0 - Resolved - Open OnDemand is responsive again. If you encounter any issues, please don't hesitate to contact the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 6 days, 16 hours and 30 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>22:30:46</var> GMT+0</small><br><strong>Identified</strong> - The CHPC has received reports of issues connecting to ondemand.chpc.utah.edu. The Open OnDemand server in the General Environment is currently under heavy load from the aggregate of user sessions. A second server, ondemand-class.chpc.utah.edu, is available to use if you are encountering issues with the primary Open OnDemand server. We apologize for the inconvenience..</p> <p><small>Aug <var data-var='date'> 11</var>, <var data-var='time'>15:01:09</var> GMT+0</small><br><strong>Resolved</strong> - Open OnDemand is responsive again. If you encounter any issues, please don't hesitate to contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Tue, 4 Aug 2026 22:30:46 +0000 https://uofu-chpc.instatus.com/incident/cmsf8f1bd00110koga70z9lyq https://uofu-chpc.instatus.com/incident/cmsf8f1bd00110koga70z9lyq Security updates (emergency maintenance) on high-performance computing clusters Type: Maintenance Duration: 22 days, 20 hours and 43 minutes Affected Components: HPC clusters, HPC clusters Jul 22, 15:23:06 GMT+0 - Identified - The Center for High Performance Computing will begin emergency maintenance to apply **security updates to high-performance computing clusters (granite, notchpeak, kingspeak, lonepeak, and redwood) in a rolling fashion today (July 22)**. The updates are necessary to address a recently announced vulnerability. System administrators will update and reboot compute nodes when no jobs are running on them. This update will not affect running jobs, but researchers may see nodes in a “DRAIN” state before updates are applied; new jobs will not be able to start on a node until updates on that node finish. The CHPC will also update interactive nodes, including login nodes (such as granite1 and granite2, redwood1 and redwood2, and similar nodes on other clusters) and research group-owned interactive nodes, on all high-performance computing clusters. **Administrators will reboot the interactive nodes today at approximately 12:00 p.m. MDT to apply important security updates.** We recommend signing out from interactive nodes prior to this time. Any active sessions or unsaved progress on interactive nodes will be lost when the nodes reboot. In accordance with the CHPC’s emergency maintenance policy, important security patches must be applied promptly to protect system integrity and safety across our shared infrastructure. Thank you for your understanding and patience. If you have any questions or concerns about upcoming security updates, please contact the CHPC by email at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu). Jul 22, 17:30:01 GMT+0 - Identified - Maintenance is now in progress Aug 4, 22:43:25 GMT+0 - Identified - Most systems have been been updated and are available for use. A small number of systems may still need to reboot once running jobs have completed. If you encounter any issues with CHPC systems, please contact us at helpdesk@chpc.utah.edu. Aug 14, 14:12:52 GMT+0 - Completed - Most systems have been updated and rebooted. If you encounter any issues, please do not hesitate to contact the CHPC at helpdesk@chpc.utah.edu. Type: Maintenance

Duration: 22 days, 20 hours and 43 minutes

Affected Components: ,

<p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>15:23:06</var> GMT+0</small><br><strong>Identified</strong> - The Center for High Performance Computing will begin emergency maintenance to apply **security updates to high-performance computing clusters (granite, notchpeak, kingspeak, lonepeak, and redwood) in a rolling fashion today (July 22)**. The updates are necessary to address a recently announced vulnerability. System administrators will update and reboot compute nodes when no jobs are running on them. This update will not affect running jobs, but researchers may see nodes in a “DRAIN” state before updates are applied; new jobs will not be able to start on a node until updates on that node finish. The CHPC will also update interactive nodes, including login nodes (such as granite1 and granite2, redwood1 and redwood2, and similar nodes on other clusters) and research group-owned interactive nodes, on all high-performance computing clusters. **Administrators will reboot the interactive nodes today at approximately 12:00 p.m. MDT to apply important security updates.** We recommend signing out from interactive nodes prior to this time. Any active sessions or unsaved progress on interactive nodes will be lost when the nodes reboot. In accordance with the CHPC’s emergency maintenance policy, important security patches must be applied promptly to protect system integrity and safety across our shared infrastructure. Thank you for your understanding and patience. If you have any questions or concerns about upcoming security updates, please contact the CHPC by email at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu)..</p> <p><small>Jul <var data-var='date'> 22</var>, <var data-var='time'>17:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>22:43:25</var> GMT+0</small><br><strong>Identified</strong> - Most systems have been been updated and are available for use. A small number of systems may still need to reboot once running jobs have completed. If you encounter any issues with CHPC systems, please contact us at helpdesk@chpc.utah.edu..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>14:12:52</var> GMT+0</small><br><strong>Completed</strong> - Most systems have been updated and rebooted. If you encounter any issues, please do not hesitate to contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Wed, 22 Jul 2026 17:30:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmrw8ezbz00x40ro0v0idlb10 https://uofu-chpc.instatus.com/maintenance/cmrw8ezbz00x40ro0v0idlb10 Scheduled maintenance on July 14: Software updates on HPC clusters and standalone servers Type: Maintenance Duration: 3 days, 2 hours and 3 minutes Affected Components: Homepage and documentation, HPC clusters, Open OnDemand, Virtual machines (VMs), Windows servers, Storage systems, HPC clusters, Open OnDemand, Windows servers, Group- and project-specific websites (hosted on VMs), Computational servers, independent of clusters, Portal, Computational servers, independent of clusters Jun 22, 22:24:52 GMT+0 - Identified - The Center for High Performance Computing will have a **downtime on Tuesday, July 14, 2026**. The downtime will begin at 8:00 a.m. MDT and continue through the afternoon. It will affect high-performance computing clusters (granite, notchpeak, kingspeak, lonepeak, redwood; Open OnDemand) and standalone servers (including narwhal, beehive, and CryoSPARC and ColabFold servers) in the General and Protected Environments. In addition, storage systems in the General Environment will be rebooted during the maintenance window, which will affect access to storage and virtual machines. This downtime is necessary to apply patches and updates to systems following recent operating system vulnerabilities. CHPC system administrators will apply updates and reboot systems in a rolling fashion throughout the day on July 14\. There will be interruptions to the availability of HPC clusters and standalone servers during this time. CHPC staff will also migrate some internal services and systems, which will briefly affect the availability of metrics reported on the CHPC Portal. Network administrators will also update software on network devices in the Protected Environment, though network updates are not expected to cause outages or interruptions. Additionally, storage administrators will update firmware; this, too, should have no effect on availability. If you have any questions or concerns about the downtime or its effects, please contact the CHPC at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu). Thanks for your support while we work to keep research computing resources secure. Jul 14, 16:31:24 GMT+0 - Identified - Virtual machines (VMs) in the General Environment have been returned to service. Jul 15, 01:13:23 GMT+0 - Identified - Narwhal, the Windows server in the Protected Environment, has been returned to service. System administrators are continuing work on other systems; thank you for your patience. Jul 15, 01:37:38 GMT+0 - Identified - Clusters in the General Environment (granite, notchpeak, kingspeak, and lonepeak) have been returned to service. The team at the CHPC is continuing work on other systems. Jul 6, 22:30:09 GMT+0 - Identified - Update to scope of planned maintenance on July 14: CHPC storage administrators will need to reboot storage systems in the General Environment as part of planned maintenance. As a result, home directories, the sys branch (/uufs/chpc.utah.edu/sys/), and virtual machines (VMs) in the General Environment will be briefly unavailable during the planned maintenance window. This outage is necessary to replace fans that have failed in the storage system. If you have questions about the impacts of this outage, please contact the CHPC at helpdesk@chpc.utah.edu. Jul 14, 14:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 15, 04:27:33 GMT+0 - Identified - Redwood is back in production. Please, let us know at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu) if there is any outstanding issue. We will wrap things up tomorrow morning. Jul 15, 22:39:34 GMT+0 - Identified - All systems except for Beehive, the Windows general environment server, are up. Beehive got corrupted during the update and is being rebuilt. We will announce when it is up. Jul 17, 16:02:47 GMT+0 - Completed - Beehive, the Windows server in the General Environment, has been returned to service. CHPC systems and services are now available following maintenance earlier this week. Thank you for your patience. If you encounter any issues, please don't hesitate to contact us by emailing helpdesk@chpc.utah.edu. Type: Maintenance

Duration: 3 days, 2 hours and 3 minutes

Affected Components: , , , , , , , , , , , ,

<p><small>Jun <var data-var='date'> 22</var>, <var data-var='time'>22:24:52</var> GMT+0</small><br><strong>Identified</strong> - The Center for High Performance Computing will have a **downtime on Tuesday, July 14, 2026**. The downtime will begin at 8:00 a.m. MDT and continue through the afternoon. It will affect high-performance computing clusters (granite, notchpeak, kingspeak, lonepeak, redwood; Open OnDemand) and standalone servers (including narwhal, beehive, and CryoSPARC and ColabFold servers) in the General and Protected Environments. In addition, storage systems in the General Environment will be rebooted during the maintenance window, which will affect access to storage and virtual machines. This downtime is necessary to apply patches and updates to systems following recent operating system vulnerabilities. CHPC system administrators will apply updates and reboot systems in a rolling fashion throughout the day on July 14\. There will be interruptions to the availability of HPC clusters and standalone servers during this time. CHPC staff will also migrate some internal services and systems, which will briefly affect the availability of metrics reported on the CHPC Portal. Network administrators will also update software on network devices in the Protected Environment, though network updates are not expected to cause outages or interruptions. Additionally, storage administrators will update firmware; this, too, should have no effect on availability. If you have any questions or concerns about the downtime or its effects, please contact the CHPC at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu). Thanks for your support while we work to keep research computing resources secure..</p> <p><small>Jul <var data-var='date'> 14</var>, <var data-var='time'>16:31:24</var> GMT+0</small><br><strong>Identified</strong> - Virtual machines (VMs) in the General Environment have been returned to service..</p> <p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>01:13:23</var> GMT+0</small><br><strong>Identified</strong> - Narwhal, the Windows server in the Protected Environment, has been returned to service. System administrators are continuing work on other systems; thank you for your patience..</p> <p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>01:37:38</var> GMT+0</small><br><strong>Identified</strong> - Clusters in the General Environment (granite, notchpeak, kingspeak, and lonepeak) have been returned to service. The team at the CHPC is continuing work on other systems..</p> <p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>22:30:09</var> GMT+0</small><br><strong>Identified</strong> - Update to scope of planned maintenance on July 14: CHPC storage administrators will need to reboot storage systems in the General Environment as part of planned maintenance. As a result, home directories, the sys branch (/uufs/chpc.utah.edu/sys/), and virtual machines (VMs) in the General Environment will be briefly unavailable during the planned maintenance window. This outage is necessary to replace fans that have failed in the storage system. If you have questions about the impacts of this outage, please contact the CHPC at helpdesk@chpc.utah.edu..</p> <p><small>Jul <var data-var='date'> 14</var>, <var data-var='time'>14:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>04:27:33</var> GMT+0</small><br><strong>Identified</strong> - Redwood is back in production. Please, let us know at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu) if there is any outstanding issue. We will wrap things up tomorrow morning..</p> <p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>22:39:34</var> GMT+0</small><br><strong>Identified</strong> - All systems except for Beehive, the Windows general environment server, are up. Beehive got corrupted during the update and is being rebuilt. We will announce when it is up..</p> <p><small>Jul <var data-var='date'> 17</var>, <var data-var='time'>16:02:47</var> GMT+0</small><br><strong>Completed</strong> - Beehive, the Windows server in the General Environment, has been returned to service. CHPC systems and services are now available following maintenance earlier this week. Thank you for your patience. If you encounter any issues, please don't hesitate to contact us by emailing helpdesk@chpc.utah.edu..</p> ]]> Tue, 14 Jul 2026 14:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmqps9tl30auq2opftb57nvqc https://uofu-chpc.instatus.com/maintenance/cmqps9tl30auq2opftb57nvqc Issues with FastX licenses on July 1 Type: Incident Duration: 10 hours Affected Components: Computational servers, independent of clusters, HPC clusters, Computational servers, independent of clusters, HPC clusters, HPC clusters Jul 1, 06:00:00 GMT+0 - Investigating - The CHPC is aware that users connecting to Linux systems through FastX are receiving error messages related to the license. System administrators are working to correct the issue as quickly as possible and have contacted the software vendor. In the interim, we recommend using Open OnDemand (<https://ondemand.chpc.utah.edu/> in the General Environment and <https://pe-ondemand.chpc.utah.edu/> in the Protected Environment) or SSH to connect to CHPC systems. We apologize for the inconvenience and appreciate your patience. If you have any questions or concerns about connecting to CHPC systems while FastX is unavailable, please contact the CHPC at helpdesk@chpc.utah.edu; we'd be happy to help. Jul 1, 15:59:46 GMT+0 - Resolved - The issue with FastX licenses has been resolved by the vendor. FastX sessions should now be functional again. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu. Thanks again for your patience. Type: Incident

Duration: 10 hours

Affected Components: , , , ,

<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is aware that users connecting to Linux systems through FastX are receiving error messages related to the license. System administrators are working to correct the issue as quickly as possible and have contacted the software vendor. In the interim, we recommend using Open OnDemand (<https://ondemand.chpc.utah.edu/> in the General Environment and <https://pe-ondemand.chpc.utah.edu/> in the Protected Environment) or SSH to connect to CHPC systems. We apologize for the inconvenience and appreciate your patience. If you have any questions or concerns about connecting to CHPC systems while FastX is unavailable, please contact the CHPC at helpdesk@chpc.utah.edu; we'd be happy to help..</p> <p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:59:46</var> GMT+0</small><br><strong>Resolved</strong> - The issue with FastX licenses has been resolved by the vendor. FastX sessions should now be functional again. If you encounter further issues, please contact the CHPC at helpdesk@chpc.utah.edu. Thanks again for your patience..</p> ]]> Wed, 1 Jul 2026 06:00:00 +0000 https://uofu-chpc.instatus.com/incident/cmr28xfbb00j90zpy6wxendix https://uofu-chpc.instatus.com/incident/cmr28xfbb00j90zpy6wxendix VM outage due to new hardware migration Type: Maintenance Duration: 1 day, 16 hours and 48 minutes Affected Components: Virtual machines (VMs), Virtual machines (VMs) Jun 3, 22:48:22 GMT+0 - Completed - CHPC system administrators have migrated most virtual machines and are in contact with groups with any remaining virtual machines. If you encounter any issues with a virtual machine, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience and continued support. Jun 2, 06:00:00 GMT+0 - Identified - On June 2nd and 3rd, we will be performing mandatory maintenance on all CHPC hosted VMs in the both the protected and general environments. During this time there will be up to a 2-hour downtime for individual VMs where we will be migrating them to new hardware and the VM will need to be shutdown. The individual VM outages should not exceed 2 hours, but we will be migrating all remaining VMs that have not yet been migrated over the 2-day outage window. Jun 2, 06:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 day, 16 hours and 48 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>22:48:22</var> GMT+0</small><br><strong>Completed</strong> - CHPC system administrators have migrated most virtual machines and are in contact with groups with any remaining virtual machines. If you encounter any issues with a virtual machine, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience and continued support..</p> <p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Identified</strong> - On June 2nd and 3rd, we will be performing mandatory maintenance on all CHPC hosted VMs in the both the protected and general environments. During this time there will be up to a 2-hour downtime for individual VMs where we will be migrating them to new hardware and the VM will need to be shutdown. The individual VM outages should not exceed 2 hours, but we will be migrating all remaining VMs that have not yet been migrated over the 2-day outage window..</p> <p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>06:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Tue, 2 Jun 2026 06:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmpcwgeny01rlqj7672gyw0wu https://uofu-chpc.instatus.com/maintenance/cmpcwgeny01rlqj7672gyw0wu Nvidia GPU driver critical update Type: Incident Duration: 1 day, 19 hours and 6 minutes Affected Components: HPC clusters, Virtual machines (VMs), Windows servers, HPC clusters, Windows servers May 21, 15:25:50 GMT+0 - Identified - We are continuing to work on a fix for this incident. May 20, 23:00:00 GMT+0 - Identified - Nvidia released new GPU drivers that address critical vulnerability, <https://nvidia.custhelp.com/app/answers/detail/a%5Fid/5821> All systems with Nvidia GPUs is affected. We will be updating the drivers, or have updated them already, on HPC clusters, owner interactive nodes such as the Cryosparc machines, and Virtual Machines. The processes running on GPUs have to be killed for the driver update to apply. On the Linux systems, the update should not require a reboot, but, it's possible that some machines will have to be rebooted. If you have any questions, please, contact our helpdesk, [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu) May 21, 23:12:46 GMT+0 - Identified - All the HPC cluster GPU drivers were updated and queues were released. Most of standalone servers have been updated as well. May 22, 18:05:57 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 day, 19 hours and 6 minutes

Affected Components: , , , ,

<p><small>May <var data-var='date'> 21</var>, <var data-var='time'>15:25:50</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>May <var data-var='date'> 20</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Identified</strong> - Nvidia released new GPU drivers that address critical vulnerability, <https://nvidia.custhelp.com/app/answers/detail/a%5Fid/5821> All systems with Nvidia GPUs is affected. We will be updating the drivers, or have updated them already, on HPC clusters, owner interactive nodes such as the Cryosparc machines, and Virtual Machines. The processes running on GPUs have to be killed for the driver update to apply. On the Linux systems, the update should not require a reboot, but, it's possible that some machines will have to be rebooted. If you have any questions, please, contact our helpdesk, [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu).</p> <p><small>May <var data-var='date'> 21</var>, <var data-var='time'>23:12:46</var> GMT+0</small><br><strong>Identified</strong> - All the HPC cluster GPU drivers were updated and queues were released. Most of standalone servers have been updated as well. .</p> <p><small>May <var data-var='date'> 22</var>, <var data-var='time'>18:05:57</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 20 May 2026 23:00:00 +0000 https://uofu-chpc.instatus.com/incident/cmpfn5tjd00ojqiv3ll99gdg3 https://uofu-chpc.instatus.com/incident/cmpfn5tjd00ojqiv3ll99gdg3 Debugging disabled on Linux system due to vulnerability Type: Incident Duration: 22 days, 4 hours and 35 minutes Affected Components: HPC clusters, Virtual machines (VMs), HPC clusters, HPC clusters, Computational servers, independent of clusters, Computational servers, independent of clusters May 15, 18:00:00 GMT+0 - Identified - A new Linux kernel vulnerability has been announced last night. A workaround mitigation before patched kernels are released involves disabling the ptrace scope, which among others prevents debuggers from attaching to executables. Due to this, debugging on our Linux systems will be disabled till the kernels are patched. More details on this vulnerability are at <https://almalinux.org/blog/2026-05-15-ssh-keysign-pwn-cve-2026-46333/>. If you have immediate debugging needs, please, consult with our helpdesk@chpc.utah.edu. Aug 4, 22:35:28 GMT+0 - Resolved - Debugging features have been reenabled on CHPC systems following a recent operating system kernel update. The features that were disabled as a precaution should now be available. Thank you for your patience. If you encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 22 days, 4 hours and 35 minutes

Affected Components: , , , , ,

<p><small>May <var data-var='date'> 15</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Identified</strong> - A new Linux kernel vulnerability has been announced last night. A workaround mitigation before patched kernels are released involves disabling the ptrace scope, which among others prevents debuggers from attaching to executables. Due to this, debugging on our Linux systems will be disabled till the kernels are patched. More details on this vulnerability are at <https://almalinux.org/blog/2026-05-15-ssh-keysign-pwn-cve-2026-46333/>. If you have immediate debugging needs, please, consult with our helpdesk@chpc.utah.edu..</p> <p><small>Aug <var data-var='date'> 4</var>, <var data-var='time'>22:35:28</var> GMT+0</small><br><strong>Resolved</strong> - Debugging features have been reenabled on CHPC systems following a recent operating system kernel update. The features that were disabled as a precaution should now be available. Thank you for your patience. If you encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Fri, 15 May 2026 18:00:00 +0000 https://uofu-chpc.instatus.com/incident/cmp7dran5006oqvozt59x5kkb https://uofu-chpc.instatus.com/incident/cmp7dran5006oqvozt59x5kkb Security-related updates to CHPC systems Type: Maintenance Duration: 4 days and 6 hours Affected Components: Portal, Homepage and documentation, HPC clusters, Open OnDemand, License servers (commercial software), Virtual machines (VMs), Windows servers, Computational servers, independent of clusters, Storage systems, Data Transfer Nodes (DTNs), HPC clusters, Open OnDemand, Windows servers, Virtual machines (VMs), Computational servers, independent of clusters, Storage systems, Data Transfer Nodes (DTNs), HPC clusters, Windows servers, Data Transfer Nodes (DTNs), Virtual machines (VMs), Storage systems, Group- and project-specific websites (hosted on VMs), Helpdesk platform Apr 30, 21:10:16 GMT+0 - Identified - CHPC system administrators are continuing work to mitigate potential security issues in the General Environment and Protected Environment. Systems in both environments, including cluster nodes and virtual machines, are being rebooted or will be rebooted. User-facing systems in the Citadel environment have been updated and rebooted and are now available. Apr 30, 22:41:24 GMT+0 - Identified - The redwood cluster in the Protected Environment has been returned to service. CHPC system administrators are continuing work on other systems. Apr 30, 22:58:11 GMT+0 - Identified - Storage systems and Data Transfer Nodes (DTNs) in the General Environment and Protected Environment have been rebooted and returned to service. May 1, 00:23:31 GMT+0 - Identified - The clusters in the General Environment (granite, notchpeak, kingspeak, and lonepeak) have been returned to service. May 1, 00:26:21 GMT+0 - Identified - Virtual machines in the Protected Environment have been returned to service. CHPC system administrators are now working through virtual machines in the General Environment. Apr 30, 17:49:34 GMT+0 - Identified - Apr 30, 17:00:00 GMT+0 - Identified - The Center for High Performance Computing is taking proactive, precautionary measures to mitigate a security issue that requires immediate attention. **CHPC staff will reboot systems in a rolling fashion throughout the day. This will result in access and service disruptions. Running jobs and active sessions on CHPC systems, including jobs on compute nodes and sessions on interactive nodes and Open OnDemand, may be lost as systems are rebooted.** The CHPC will provide additional information and updates throughout the day as services are brought back online. We apologize for the inconvenience and appreciate your patience and support. May 1, 01:21:14 GMT+0 - Identified - Thank you for your patience with security-related service interruptions on Center for High Performance Computing systems today. **All clusters (granite, notchpeak, kingspeak, and lonepeak in the General Environment and redwood in the Protected Environment) have been returned to service by the team at the CHPC.** System administrators have also brought virtual machines (VMs) in the Protected Environment back online and are continuing work on VMs in the General Environment. We will continue to provide updates on the status of systems here. May 1, 21:09:36 GMT+0 - Identified - Most CHPC systems have been updated and rebooted to mitigate the security issue. CHPC staff are still working on a small number of systems and addressing some maintenance-related issues with individual systems. Thank you for your patience. May 6, 15:52:45 GMT+0 - Completed - Most user-facing systems have been rebooted to address the potential security issue. There may be a small number of outstanding systems, but most users will not encounter service interruptions when such systems are updated. Thank you for your patience and understanding. If you have any questions or concerns, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Maintenance

Duration: 4 days and 6 hours

Affected Components: , , , , , , , , , , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>21:10:16</var> GMT+0</small><br><strong>Identified</strong> - CHPC system administrators are continuing work to mitigate potential security issues in the General Environment and Protected Environment. Systems in both environments, including cluster nodes and virtual machines, are being rebooted or will be rebooted. User-facing systems in the Citadel environment have been updated and rebooted and are now available..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>22:41:24</var> GMT+0</small><br><strong>Identified</strong> - The redwood cluster in the Protected Environment has been returned to service. CHPC system administrators are continuing work on other systems..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>22:58:11</var> GMT+0</small><br><strong>Identified</strong> - Storage systems and Data Transfer Nodes (DTNs) in the General Environment and Protected Environment have been rebooted and returned to service..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>00:23:31</var> GMT+0</small><br><strong>Identified</strong> - The clusters in the General Environment (granite, notchpeak, kingspeak, and lonepeak) have been returned to service..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>00:26:21</var> GMT+0</small><br><strong>Identified</strong> - Virtual machines in the Protected Environment have been returned to service. CHPC system administrators are now working through virtual machines in the General Environment..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>17:49:34</var> GMT+0</small><br><strong>Identified</strong> - .</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Identified</strong> - The Center for High Performance Computing is taking proactive, precautionary measures to mitigate a security issue that requires immediate attention. **CHPC staff will reboot systems in a rolling fashion throughout the day. This will result in access and service disruptions. Running jobs and active sessions on CHPC systems, including jobs on compute nodes and sessions on interactive nodes and Open OnDemand, may be lost as systems are rebooted.** The CHPC will provide additional information and updates throughout the day as services are brought back online. We apologize for the inconvenience and appreciate your patience and support..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>01:21:14</var> GMT+0</small><br><strong>Identified</strong> - Thank you for your patience with security-related service interruptions on Center for High Performance Computing systems today. **All clusters (granite, notchpeak, kingspeak, and lonepeak in the General Environment and redwood in the Protected Environment) have been returned to service by the team at the CHPC.** System administrators have also brought virtual machines (VMs) in the Protected Environment back online and are continuing work on VMs in the General Environment. We will continue to provide updates on the status of systems here..</p> <p><small>May <var data-var='date'> 1</var>, <var data-var='time'>21:09:36</var> GMT+0</small><br><strong>Identified</strong> - Most CHPC systems have been updated and rebooted to mitigate the security issue. CHPC staff are still working on a small number of systems and addressing some maintenance-related issues with individual systems. Thank you for your patience..</p> <p><small>May <var data-var='date'> 6</var>, <var data-var='time'>15:52:45</var> GMT+0</small><br><strong>Completed</strong> - Most user-facing systems have been rebooted to address the potential security issue. There may be a small number of outstanding systems, but most users will not encounter service interruptions when such systems are updated. Thank you for your patience and understanding. If you have any questions or concerns, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Thu, 30 Apr 2026 17:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmolrsegm02x3s9segns89alz https://uofu-chpc.instatus.com/maintenance/cmolrsegm02x3s9segns89alz Some jobs are failing to start, yielding errors in sbatch or srun Type: Incident Duration: 6 hours and 43 minutes Affected Components: HPC clusters, Open OnDemand, Computational servers, independent of clusters, Storage systems Apr 17, 22:25:42 GMT+0 - Investigating - The issues we are seeing stems with inability to write to the "sys" branches, where applications, SLURM information, and other things are being written. We are still investigating the reasons for that. Apr 18, 02:18:44 GMT+0 - Resolved - This incident has been resolved. Apr 17, 19:35:56 GMT+0 - Investigating - CHPC system administrators are aware of issues users have reported when starting jobs and are currently investigating. Type: Incident

Duration: 6 hours and 43 minutes

Affected Components: , , ,

<p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>22:25:42</var> GMT+0</small><br><strong>Investigating</strong> - The issues we are seeing stems with inability to write to the "sys" branches, where applications, SLURM information, and other things are being written. We are still investigating the reasons for that..</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>02:18:44</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>19:35:56</var> GMT+0</small><br><strong>Investigating</strong> - CHPC system administrators are aware of issues users have reported when starting jobs and are currently investigating..</p> ]]> Fri, 17 Apr 2026 19:35:56 +0000 https://uofu-chpc.instatus.com/incident/cmo3b6c9300az78d3hrotww84 https://uofu-chpc.instatus.com/incident/cmo3b6c9300az78d3hrotww84 UETN network upgrade affecting CHPC DTNs overnight from April 16 to April 17 Type: Maintenance Duration: 4 hours Affected Components: Data Transfer Nodes (DTNs), Data Transfer Nodes (DTNs) Apr 17, 09:00:00 GMT+0 - Completed - Maintenance has completed successfully Apr 17, 05:00:00 GMT+0 - Identified - UETN will be updating its core router that affects CHPC's DMZ (campus firewall bypass for faster data transfers). Services that use the DMZ such as CHPC's DTNs and several project data servers will experience degraded or no network connection. Apr 17, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 4 hours

Affected Components: ,

<p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>09:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - UETN will be updating its core router that affects CHPC's DMZ (campus firewall bypass for faster data transfers). Services that use the DMZ such as CHPC's DTNs and several project data servers will experience degraded or no network connection..</p> <p><small>Apr <var data-var='date'> 17</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Fri, 17 Apr 2026 05:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmnsdlvph086x1u1zqnsem7t0 https://uofu-chpc.instatus.com/maintenance/cmnsdlvph086x1u1zqnsem7t0 Issues with module system, affecting logins and module loads Type: Incident Duration: 2 hours and 40 minutes Affected Components: HPC clusters, Open OnDemand, Computational servers, independent of clusters, HPC clusters, Open OnDemand, Computational servers, independent of clusters Apr 13, 18:10:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 13, 18:50:00 GMT+0 - Resolved - CHPC staff have fixed an issue with the module system, which should resolve any issues with logins or module loads. If you continue to encounter issues, please contact us at helpdesk@chpc.utah.edu. Type: Incident

Duration: 2 hours and 40 minutes

Affected Components: , , , , ,

<p><small>Apr <var data-var='date'> 13</var>, <var data-var='time'>18:10:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 13</var>, <var data-var='time'>18:50:00</var> GMT+0</small><br><strong>Resolved</strong> - CHPC staff have fixed an issue with the module system, which should resolve any issues with logins or module loads. If you continue to encounter issues, please contact us at helpdesk@chpc.utah.edu..</p> ]]> Mon, 13 Apr 2026 18:10:00 +0000 https://uofu-chpc.instatus.com/incident/cmnxk2b0804gmez38f3mkne2y https://uofu-chpc.instatus.com/incident/cmnxk2b0804gmez38f3mkne2y Significant number of nodes on redwood cluster (Protected Environment) are down Type: Incident Duration: 3 hours and 49 minutes Affected Components: HPC clusters Apr 9, 19:00:45 GMT+0 - Investigating - The CHPC is aware that many nodes on the redwood cluster (Protected Environment) are currently down. System administrators are investigating. Apr 9, 22:49:17 GMT+0 - Resolved - CHPC system administrators have restored the nodes on the redwood cluster to service. Thank you for your patience. If you encounter further issues, please contact us at helpdesk@chpc.utah.edu. Type: Incident

Duration: 3 hours and 49 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>19:00:45</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is aware that many nodes on the redwood cluster (Protected Environment) are currently down. System administrators are investigating..</p> <p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>22:49:17</var> GMT+0</small><br><strong>Resolved</strong> - CHPC system administrators have restored the nodes on the redwood cluster to service. Thank you for your patience. If you encounter further issues, please contact us at helpdesk@chpc.utah.edu..</p> ]]> Thu, 9 Apr 2026 19:00:45 +0000 https://uofu-chpc.instatus.com/incident/cmnrx62md019innrfutcvupjk https://uofu-chpc.instatus.com/incident/cmnrx62md019innrfutcvupjk Degraded file system performance with various impacts (Open OnDemand sessions not starting, reduced input and output performance) Type: Incident Duration: 20 hours and 36 minutes Affected Components: Open OnDemand Apr 7, 01:50:08 GMT+0 - Monitoring - CHPC staff have identified a number of jobs that were impacting file system performance. The team at the CHPC canceled jobs with a significant impact, which has improved performance. Systems, including Open OnDemand, should now be more responsive. CHPC staff will continue to monitor the situation. Apr 7, 16:06:25 GMT+0 - Resolved - This incident has been resolved. CHPC systems have returned to responsive states. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu. Apr 6, 19:30:39 GMT+0 - Investigating - We are receiving user reports of Open OnDemand sessions not starting. So far, issues have been reported on notchpeak and kingspeak but appear to be intermittent. A retry may resolve the issue in some cases. The CHPC is currently investigating the cause. Type: Incident

Duration: 20 hours and 36 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>01:50:08</var> GMT+0</small><br><strong>Monitoring</strong> - CHPC staff have identified a number of jobs that were impacting file system performance. The team at the CHPC canceled jobs with a significant impact, which has improved performance. Systems, including Open OnDemand, should now be more responsive. CHPC staff will continue to monitor the situation..</p> <p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>16:06:25</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. CHPC systems have returned to responsive states. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>19:30:39</var> GMT+0</small><br><strong>Investigating</strong> - We are receiving user reports of Open OnDemand sessions not starting. So far, issues have been reported on notchpeak and kingspeak but appear to be intermittent. A retry may resolve the issue in some cases. The CHPC is currently investigating the cause..</p> ]]> Mon, 6 Apr 2026 19:30:39 +0000 https://uofu-chpc.instatus.com/incident/cmnnl56fi0u4yyn6v8kjklzrz https://uofu-chpc.instatus.com/incident/cmnnl56fi0u4yyn6v8kjklzrz Issues with logins to beehive (Windows server in General Environment) Type: Incident Duration: 2 hours and 30 minutes Affected Components: Windows servers Apr 3, 14:20:00 GMT+0 - Investigating - Users have reported issues logging in to the beehive server. Apr 3, 16:50:00 GMT+0 - Resolved - Users report that logins to beehive are working again. The cause of the issue was very high memory consumption that exhausted the system's memory and the space available for the pagefile (swap). Type: Incident

Duration: 2 hours and 30 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>14:20:00</var> GMT+0</small><br><strong>Investigating</strong> - Users have reported issues logging in to the beehive server..</p> <p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>16:50:00</var> GMT+0</small><br><strong>Resolved</strong> - Users report that logins to beehive are working again. The cause of the issue was very high memory consumption that exhausted the system's memory and the space available for the pagefile (swap)..</p> ]]> Fri, 3 Apr 2026 14:20:00 +0000 https://uofu-chpc.instatus.com/incident/cmnj0wb6s0eihbhe5sa5b6z9h https://uofu-chpc.instatus.com/incident/cmnj0wb6s0eihbhe5sa5b6z9h Slurm not responding in Protected Environment (Redwood) Type: Incident Duration: 54 minutes Affected Components: HPC clusters, Open OnDemand Mar 18, 13:49:48 GMT+0 - Identified - The CHPC has become aware that Slurm is not talking to Redwood and we are working to fix this. Mar 18, 14:43:20 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 54 minutes

Affected Components: ,

<p><small>Mar <var data-var='date'> 18</var>, <var data-var='time'>13:49:48</var> GMT+0</small><br><strong>Identified</strong> - The CHPC has become aware that Slurm is not talking to Redwood and we are working to fix this..</p> <p><small>Mar <var data-var='date'> 18</var>, <var data-var='time'>14:43:20</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 18 Mar 2026 13:49:48 +0000 https://uofu-chpc.instatus.com/incident/cmmw3lo3a09uk558gs6vyx63m https://uofu-chpc.instatus.com/incident/cmmw3lo3a09uk558gs6vyx63m Sites hosted on home.chpc.utah.edu are not reachable Type: Incident Duration: 1 hour and 14 minutes Affected Components: Group- and project-specific websites (hosted on VMs) Mar 16, 17:30:00 GMT+0 - Investigating - Websites hosted on home.chpc.utah.edu, including user- and group-specific sites, are not reachable. The CHPC is aware of this issue and investigating the cause. Mar 16, 18:43:59 GMT+0 - Resolved - The issue with [home.chpc.utah.edu](http://home.chpc.utah.edu) has been resolved and user- and group-specific pages are being served again. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Incident

Duration: 1 hour and 14 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>17:30:00</var> GMT+0</small><br><strong>Investigating</strong> - Websites hosted on home.chpc.utah.edu, including user- and group-specific sites, are not reachable. The CHPC is aware of this issue and investigating the cause..</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>18:43:59</var> GMT+0</small><br><strong>Resolved</strong> - The issue with [home.chpc.utah.edu](http://home.chpc.utah.edu) has been resolved and user- and group-specific pages are being served again. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Mon, 16 Mar 2026 17:30:00 +0000 https://uofu-chpc.instatus.com/incident/cmmtile5g002h37xbiznci6yf https://uofu-chpc.instatus.com/incident/cmmtile5g002h37xbiznci6yf Filesystem issue requiring journal replay; some General Environment group spaces inaccessible Type: Incident Duration: 8 minutes Affected Components: Storage systems Mar 13, 16:44:00 GMT+0 - Investigating - Excessive usage on a General Environment filesystem required it to be taken offline temporarily. Mar 13, 16:52:00 GMT+0 - Resolved - CHPC staff have brought the filesystem back online. Type: Incident

Duration: 8 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>16:44:00</var> GMT+0</small><br><strong>Investigating</strong> - Excessive usage on a General Environment filesystem required it to be taken offline temporarily..</p> <p><small>Mar <var data-var='date'> 13</var>, <var data-var='time'>16:52:00</var> GMT+0</small><br><strong>Resolved</strong> - CHPC staff have brought the filesystem back online..</p> ]]> Fri, 13 Mar 2026 16:44:00 +0000 https://uofu-chpc.instatus.com/incident/cmmp507ng0017wfhgfqm305ke https://uofu-chpc.instatus.com/incident/cmmp507ng0017wfhgfqm305ke Many systems on the granite cluster went offline (lost power) at approximately 4:45 p.m. Type: Incident Duration: 15 hours and 18 minutes Affected Components: HPC clusters, , Computational servers, independent of clusters, General Environment (GE) → Mar 12, 14:11:16 GMT+0 - Resolved - Systems on granite returned to service shortly after the outage yesterday afternoon. CHPC staff have moved a switch to two separate power distribution units to prevent similar incidents in the future. Mar 11, 22:52:48 GMT+0 - Investigating - The CHPC is investigating an issue with many of the systems on the granite cluster. Initial reports suggest there may be an issue with power distribution to login nodes, networking infrastructure, and core services. Mar 11, 23:00:17 GMT+0 - Monitoring - CHPC and DDC staff have identified the issue and restored power to systems. Systems on the granite cluster should be back online or in the process of coming back online. CHPC staff will continue to monitor the cluster. Type: Incident

Duration: 15 hours and 18 minutes

Affected Components: , ,

<p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>14:11:16</var> GMT+0</small><br><strong>Resolved</strong> - Systems on granite returned to service shortly after the outage yesterday afternoon. CHPC staff have moved a switch to two separate power distribution units to prevent similar incidents in the future..</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>22:52:48</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is investigating an issue with many of the systems on the granite cluster. Initial reports suggest there may be an issue with power distribution to login nodes, networking infrastructure, and core services..</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>23:00:17</var> GMT+0</small><br><strong>Monitoring</strong> - CHPC and DDC staff have identified the issue and restored power to systems. Systems on the granite cluster should be back online or in the process of coming back online. CHPC staff will continue to monitor the cluster..</p> ]]> Wed, 11 Mar 2026 22:52:48 +0000 https://uofu-chpc.instatus.com/incident/cmmmmx0gh0049mr5c56koxxlp https://uofu-chpc.instatus.com/incident/cmmmmx0gh0049mr5c56koxxlp Several nodes on notchpeak lost power Type: Incident Duration: 20 hours and 12 minutes Affected Components: HPC clusters Mar 6, 22:03:39 GMT+0 - Resolved - Nodes are online. CHPC staff have drained a limited set of nodes (preventing new jobs from starting but not affecting currently running jobs) to rebalance power. Mar 6, 03:20:00 GMT+0 - Monitoring - CHPC staff on-site at the data center brought most systems back online. Staff will rebalance affected systems among power distribution units to prevent similar issues in the future. One notchpeak node remains offline while staff work on power distribution. Mar 6, 01:52:00 GMT+0 - Investigating - Several nodes on the notchpeak cluster lost power on the afternoon of March 5\. This incident affected notch366, notch452, notch472, notch473, notch474, notch475, notch476, notch477, notch478, notch479, notch480, notch481, notch482, notch483, notch484, notch485, notch486, notch487, notch488, notch489, notch490, notch491, notch492, notch493, notch494, notch495, notch496, notch497, notch498, notch499, notch500, notch501, notchpeak32, notchpeak33, and notchpeak34. Type: Incident

Duration: 20 hours and 12 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>22:03:39</var> GMT+0</small><br><strong>Resolved</strong> - Nodes are online. CHPC staff have drained a limited set of nodes (preventing new jobs from starting but not affecting currently running jobs) to rebalance power..</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>03:20:00</var> GMT+0</small><br><strong>Monitoring</strong> - CHPC staff on-site at the data center brought most systems back online. Staff will rebalance affected systems among power distribution units to prevent similar issues in the future. One notchpeak node remains offline while staff work on power distribution..</p> <p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>01:52:00</var> GMT+0</small><br><strong>Investigating</strong> - Several nodes on the notchpeak cluster lost power on the afternoon of March 5\. This incident affected notch366, notch452, notch472, notch473, notch474, notch475, notch476, notch477, notch478, notch479, notch480, notch481, notch482, notch483, notch484, notch485, notch486, notch487, notch488, notch489, notch490, notch491, notch492, notch493, notch494, notch495, notch496, notch497, notch498, notch499, notch500, notch501, notchpeak32, notchpeak33, and notchpeak34..</p> ]]> Fri, 6 Mar 2026 01:52:00 +0000 https://uofu-chpc.instatus.com/incident/cmmf2f0cr00qq11ala35n3rmx https://uofu-chpc.instatus.com/incident/cmmf2f0cr00qq11ala35n3rmx Several notchpeak nodes lost power Type: Incident Duration: 12 minutes Affected Components: HPC clusters Mar 5, 20:20:00 GMT+0 - Investigating - CHPC staff received alerts that several notchpeak nodes lost power. The outage is related to power infrastructure serving the rack. Staff are on-site and investigating. Mar 5, 20:32:00 GMT+0 - Resolved - On-site staff restored affected power infrastructure to service. Affected nodes should return to service shortly. Type: Incident

Duration: 12 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>20:20:00</var> GMT+0</small><br><strong>Investigating</strong> - CHPC staff received alerts that several notchpeak nodes lost power. The outage is related to power infrastructure serving the rack. Staff are on-site and investigating..</p> <p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>20:32:00</var> GMT+0</small><br><strong>Resolved</strong> - On-site staff restored affected power infrastructure to service. Affected nodes should return to service shortly..</p> ]]> Thu, 5 Mar 2026 20:20:00 +0000 https://uofu-chpc.instatus.com/incident/cmmdxzyq417vwu1ratu2vwz8z https://uofu-chpc.instatus.com/incident/cmmdxzyq417vwu1ratu2vwz8z Issues loading Open OnDemand in the General Environment Type: Incident Duration: 1 hour and 54 minutes Affected Components: Open OnDemand Mar 4, 14:10:00 GMT+0 - Investigating - CHPC staff are aware of an issue with access to Open OnDemand, [ondemand.chpc.utah.edu](http://ondemand.chpc.utah.edu), in the General Environment. We are investigating the cause of the issue. At this time, we believe the issue is related to a system that lost ethernet connectivity at 7:10 a.m. We will provide updates on this issue as we learn more. Mar 4, 16:04:27 GMT+0 - Resolved - CHPC staff have restored a storage system that lost ethernet connectivity to service. Open OnDemand is responsive again. Thank you for your patience. If you continue to encounter issues, please contact us at helpdesk@chpc.utah.edu. Type: Incident

Duration: 1 hour and 54 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:10:00</var> GMT+0</small><br><strong>Investigating</strong> - CHPC staff are aware of an issue with access to Open OnDemand, [ondemand.chpc.utah.edu](http://ondemand.chpc.utah.edu), in the General Environment. We are investigating the cause of the issue. At this time, we believe the issue is related to a system that lost ethernet connectivity at 7:10 a.m. We will provide updates on this issue as we learn more..</p> <p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>16:04:27</var> GMT+0</small><br><strong>Resolved</strong> - CHPC staff have restored a storage system that lost ethernet connectivity to service. Open OnDemand is responsive again. Thank you for your patience. If you continue to encounter issues, please contact us at helpdesk@chpc.utah.edu..</p> ]]> Wed, 4 Mar 2026 14:10:00 +0000 https://uofu-chpc.instatus.com/incident/cmmc7y0ht0r63u1rawn2cc7hc https://uofu-chpc.instatus.com/incident/cmmc7y0ht0r63u1rawn2cc7hc Issues with Protected Environment resources Type: Incident Duration: 6 hours and 41 minutes Affected Components: HPC clusters, , Open OnDemand, Windows servers, Virtual machines (VMs), Computational servers, independent of clusters, Storage systems, Data Transfer Nodes (DTNs), Protected Environment (PE) → Feb 27, 14:30:00 GMT+0 - Investigating - The CHPC is aware of issues with the Protected Environment, leading to degraded performance. Staff are working to identify and correct the issue. We will provide updates as we learn more. Feb 27, 17:22:51 GMT+0 - Investigating - CHPC staff are continuing to investigate the issue. Based on user reports, this incident's impact is being updated to an outage rather than degraded performance. Feb 27, 19:07:24 GMT+0 - Monitoring - Issues in the Protected Environment are attributable to packet loss. CHPC staff have stopped replication between the General Environment VAST and Protected Environment VAST, which significantly reduced the packet loss. Services appear to be responsive again. Logins and services in the PE should begin working. CHPC staff will continue to monitor the situation. Feb 27, 19:59:41 GMT+0 - Monitoring - While we have identified the cause of the problems, and most services in the PE are available, individual systems may still have lingering problems due to the previous network loss to the file systems. If your particular service has issues, please, contact helpdesk@chpc.utah.edu. Feb 27, 21:10:46 GMT+0 - Resolved - CHPC staff have determined that the issue with the Protected Environment has been resolved. Systems have remained accessible since the update earlier this afternoon. If you continue to encounter issues with a resource in the Protected Environment, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience. Type: Incident

Duration: 6 hours and 41 minutes

Affected Components: , , , , , , ,

<p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>14:30:00</var> GMT+0</small><br><strong>Investigating</strong> - The CHPC is aware of issues with the Protected Environment, leading to degraded performance. Staff are working to identify and correct the issue. We will provide updates as we learn more..</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>17:22:51</var> GMT+0</small><br><strong>Investigating</strong> - CHPC staff are continuing to investigate the issue. Based on user reports, this incident's impact is being updated to an outage rather than degraded performance..</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>19:07:24</var> GMT+0</small><br><strong>Monitoring</strong> - Issues in the Protected Environment are attributable to packet loss. CHPC staff have stopped replication between the General Environment VAST and Protected Environment VAST, which significantly reduced the packet loss. Services appear to be responsive again. Logins and services in the PE should begin working. CHPC staff will continue to monitor the situation..</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>19:59:41</var> GMT+0</small><br><strong>Monitoring</strong> - While we have identified the cause of the problems, and most services in the PE are available, individual systems may still have lingering problems due to the previous network loss to the file systems. If your particular service has issues, please, contact helpdesk@chpc.utah.edu. .</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>21:10:46</var> GMT+0</small><br><strong>Resolved</strong> - CHPC staff have determined that the issue with the Protected Environment has been resolved. Systems have remained accessible since the update earlier this afternoon. If you continue to encounter issues with a resource in the Protected Environment, please contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience..</p> ]]> Fri, 27 Feb 2026 14:30:00 +0000 https://uofu-chpc.instatus.com/incident/cmm50sszv00gq8aqit9d9g7wu https://uofu-chpc.instatus.com/incident/cmm50sszv00gq8aqit9d9g7wu Scheduled maintenance on power infrastructure: General Environment clusters (granite, notchpeak, kingspeak, lonepeak) offline late January 21 to early January 22 (overnight) Type: Maintenance Duration: 15 hours and 7 minutes Affected Components: HPC clusters Jan 22, 03:00:01 GMT+0 - Identified - Maintenance is now in progress Jan 22, 03:00:00 GMT+0 - Identified - Scheduled maintenance on power infrastructure at the Downtown Data Center will require all General Environment clusters (granite, notchpeak, kingspeak, and lonepeak) and standalone biochemistry and cryo-EM servers to be taken offline from the evening of January 21 to the morning of January 22. Critical services, storage systems, virtual machines (VMs), and the redwood cluster in the Protected Environment are on generated power and should not be affected by this outage. Jan 22, 18:07:01 GMT+0 - Completed - The power outage is complete and systems will be made available to users shortly. Notably, the change to shared and exclusive partitions has been implemented in the General Environment (the change will follow in the Protected Environment later today). Please see <https://chpc.utah.edu/newsletter%5Fupdates/read.php?year=2026&month=01&article=9d2b0> for more information about this change. Type: Maintenance

Duration: 15 hours and 7 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>03:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - Scheduled maintenance on power infrastructure at the Downtown Data Center will require all General Environment clusters (granite, notchpeak, kingspeak, and lonepeak) and standalone biochemistry and cryo-EM servers to be taken offline from the evening of January 21 to the morning of January 22. Critical services, storage systems, virtual machines (VMs), and the redwood cluster in the Protected Environment are on generated power and should not be affected by this outage..</p> <p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>18:07:01</var> GMT+0</small><br><strong>Completed</strong> - The power outage is complete and systems will be made available to users shortly. Notably, the change to shared and exclusive partitions has been implemented in the General Environment (the change will follow in the Protected Environment later today). Please see <https://chpc.utah.edu/newsletter%5Fupdates/read.php?year=2026&month=01&article=9d2b0> for more information about this change..</p> ]]> Thu, 22 Jan 2026 03:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmk7by35e09um9vouwtqv7e3f https://uofu-chpc.instatus.com/maintenance/cmk7by35e09um9vouwtqv7e3f Protected Environment resources offline for troubleshooting of network issue (updated Friday, January 16) Type: Maintenance Duration: 5 days, 19 hours and 34 minutes Affected Components: HPC clusters Jan 15, 03:00:01 GMT+0 - Identified - Maintenance is now in progress Jan 15, 03:00:00 GMT+0 - Identified - Scheduled maintenance on power infrastructure at the Downtown Data Center will require all General Environment clusters (granite, notchpeak, kingspeak, and lonepeak) and standalone biochemistry and cryo-EM servers to be taken offline from the evening of January 14 to the morning of January 15. Critical services, storage systems, virtual machines (VMs), and the redwood cluster in the Protected Environment are on generated power and should not be affected by this outage. Jan 16, 01:34:55 GMT+0 - Identified - Protected Environment network issues persist. System and network administrators at the CHPC have engaged vendors of networking hardware and are on-site at the data center to address the problem as quickly as possible. Jan 16, 16:45:32 GMT+0 - Identified - CHPC system and network administrators are taking systems in the Protected Environment offline to expedite triage and troubleshooting of networking issues. Jan 16, 23:58:48 GMT+0 - Identified - Network administrators are continuing work to diagnose packet loss issues by bringing devices and interfaces online individually. Network issues persist. Jan 17, 06:43:05 GMT+0 - Identified - Network and system administrators at the CHPC have isolated and triaged network issues and brought most systems back online. The Protected Environment is now functional and accessible to users. Some components are in a degraded or diminished state as a result of reconfigurations to bring the PE back online quickly and will require further troubleshooting next week to return to a normal, fully operational state. We appreciate your patience with this issue. Jan 20, 22:33:59 GMT+0 - Completed - Moving to an incident. Issues are ongoing. Type: Maintenance

Duration: 5 days, 19 hours and 34 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>03:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jan <var data-var='date'> 15</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - Scheduled maintenance on power infrastructure at the Downtown Data Center will require all General Environment clusters (granite, notchpeak, kingspeak, and lonepeak) and standalone biochemistry and cryo-EM servers to be taken offline from the evening of January 14 to the morning of January 15. Critical services, storage systems, virtual machines (VMs), and the redwood cluster in the Protected Environment are on generated power and should not be affected by this outage..</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>01:34:55</var> GMT+0</small><br><strong>Identified</strong> - Protected Environment network issues persist. System and network administrators at the CHPC have engaged vendors of networking hardware and are on-site at the data center to address the problem as quickly as possible..</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>16:45:32</var> GMT+0</small><br><strong>Identified</strong> - CHPC system and network administrators are taking systems in the Protected Environment offline to expedite triage and troubleshooting of networking issues..</p> <p><small>Jan <var data-var='date'> 16</var>, <var data-var='time'>23:58:48</var> GMT+0</small><br><strong>Identified</strong> - Network administrators are continuing work to diagnose packet loss issues by bringing devices and interfaces online individually. Network issues persist..</p> <p><small>Jan <var data-var='date'> 17</var>, <var data-var='time'>06:43:05</var> GMT+0</small><br><strong>Identified</strong> - Network and system administrators at the CHPC have isolated and triaged network issues and brought most systems back online. The Protected Environment is now functional and accessible to users. Some components are in a degraded or diminished state as a result of reconfigurations to bring the PE back online quickly and will require further troubleshooting next week to return to a normal, fully operational state. We appreciate your patience with this issue..</p> <p><small>Jan <var data-var='date'> 20</var>, <var data-var='time'>22:33:59</var> GMT+0</small><br><strong>Completed</strong> - Moving to an incident. Issues are ongoing..</p> ]]> Thu, 15 Jan 2026 03:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmk7bv7v709cfg8e8i3mhiwbt https://uofu-chpc.instatus.com/maintenance/cmk7bv7v709cfg8e8i3mhiwbt Scheduled maintenance (downtime) affecting all CHPC resources Type: Maintenance Duration: 4 days and 6 hours Affected Components: Open OnDemand, Portal, , Homepage and documentation, HPC clusters, Applications and infrastructure → Dec 16, 14:30:01 GMT+0 - Identified - Maintenance is now in progress Dec 17, 16:14:51 GMT+0 - Identified - As of approximately 10:45 p.m. on December 16, storage systems, the CHPC website, and standalone servers (such as biochemistry cryo-EM servers) are functional. Network issues on the morning on December 17, however, have affected access to CHPC resources. CHPC staff are working to resolve issues. Remaining services, including virtual machines (VMs) and HPC clusters, are still under maintenance. We anticipate having systems functional today (December 17). Dec 17, 16:43:23 GMT+0 - Identified - Networking (DNS) issues have been resolved. The CHPC website and portal are now accessible. HPC systems and virtual machines (VMs) remain under maintenance. Dec 18, 00:42:19 GMT+0 - Identified - We have released HPC clusters and jobs should now start running. All services should now be functional, apart from the MySQL database server and associated websites. Work on this will continue with expected release sometime tomorrow (December 18). Thank you for your patience. If you notice issues with anything other than MySQL and websites, please let us know at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu). Dec 19, 02:30:00 GMT+0 - Identified - The community MySQL server is still being migrated to a new server. We also continue to troubleshoot issues with the older VM farm, which still runs a few VMs, which should be available but may be less responsive. We will update on the progress again tomorrow. Thank you for your understanding. Dec 16, 14:30:00 GMT+0 - Identified - A downtime on December 16 and 17 will affect all CHPC resources, including clusters (redwood, granite, notchpeak, kingspeak, and lonepeak), standalone servers, virtual machines (VMs), and storage. This downtime is necessary to replace core network infrastructure. Reservations on the computing clusters have been put in place to ensure jobs will complete prior this downtime. Systems will be brought up on December 17 as the network replacement and any related system administration work is completed. An announcement will be made as resources are made available. Dec 20, 20:30:00 GMT+0 - Completed - The scheduled maintenance is now complete. If you encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu. Type: Maintenance

Duration: 4 days and 6 hours

Affected Components: , , , ,

<p><small>Dec <var data-var='date'> 16</var>, <var data-var='time'>14:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 17</var>, <var data-var='time'>16:14:51</var> GMT+0</small><br><strong>Identified</strong> - As of approximately 10:45 p.m. on December 16, storage systems, the CHPC website, and standalone servers (such as biochemistry cryo-EM servers) are functional. Network issues on the morning on December 17, however, have affected access to CHPC resources. CHPC staff are working to resolve issues. Remaining services, including virtual machines (VMs) and HPC clusters, are still under maintenance. We anticipate having systems functional today (December 17)..</p> <p><small>Dec <var data-var='date'> 17</var>, <var data-var='time'>16:43:23</var> GMT+0</small><br><strong>Identified</strong> - Networking (DNS) issues have been resolved. The CHPC website and portal are now accessible. HPC systems and virtual machines (VMs) remain under maintenance..</p> <p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>00:42:19</var> GMT+0</small><br><strong>Identified</strong> - We have released HPC clusters and jobs should now start running. All services should now be functional, apart from the MySQL database server and associated websites. Work on this will continue with expected release sometime tomorrow (December 18). Thank you for your patience. If you notice issues with anything other than MySQL and websites, please let us know at [helpdesk@chpc.utah.edu](mailto:helpdesk@chpc.utah.edu)..</p> <p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>02:30:00</var> GMT+0</small><br><strong>Identified</strong> - The community MySQL server is still being migrated to a new server. We also continue to troubleshoot issues with the older VM farm, which still runs a few VMs, which should be available but may be less responsive. We will update on the progress again tomorrow. Thank you for your understanding..</p> <p><small>Dec <var data-var='date'> 16</var>, <var data-var='time'>14:30:00</var> GMT+0</small><br><strong>Identified</strong> - A downtime on December 16 and 17 will affect all CHPC resources, including clusters (redwood, granite, notchpeak, kingspeak, and lonepeak), standalone servers, virtual machines (VMs), and storage. This downtime is necessary to replace core network infrastructure. Reservations on the computing clusters have been put in place to ensure jobs will complete prior this downtime. Systems will be brought up on December 17 as the network replacement and any related system administration work is completed. An announcement will be made as resources are made available..</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>20:30:00</var> GMT+0</small><br><strong>Completed</strong> - The scheduled maintenance is now complete. If you encounter any issues, please contact the CHPC at helpdesk@chpc.utah.edu..</p> ]]> Tue, 16 Dec 2025 14:30:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmi3oi0iz001213v9bmfl1y5s https://uofu-chpc.instatus.com/maintenance/cmi3oi0iz001213v9bmfl1y5s Scheduled maintenance affecting University of Utah systems including websites, email, VPN logins, and some authentication services Type: Maintenance Duration: 4 hours Affected Components: Homepage and documentation Dec 14, 07:30:00 GMT+0 - Identified - Scheduled maintenance on University Information Technology systems may affect access to CHPC systems, including the CHPC website. Logins and VPN connections may also be affected. See <https://uofu.status.io/> for further details. Dec 14, 07:30:01 GMT+0 - Identified - Maintenance is now in progress Dec 14, 11:30:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 4 hours

Affected Components:

<p><small>Dec <var data-var='date'> 14</var>, <var data-var='time'>07:30:00</var> GMT+0</small><br><strong>Identified</strong> - Scheduled maintenance on University Information Technology systems may affect access to CHPC systems, including the CHPC website. Logins and VPN connections may also be affected. See <https://uofu.status.io/> for further details..</p> <p><small>Dec <var data-var='date'> 14</var>, <var data-var='time'>07:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 14</var>, <var data-var='time'>11:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sun, 14 Dec 2025 07:30:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmj3ix71w0a2k12dn56ahok51 https://uofu-chpc.instatus.com/maintenance/cmj3ix71w0a2k12dn56ahok51 Scheduled maintenance on portal.chpc.utah.edu Type: Maintenance Duration: 15 minutes Affected Components: Portal Dec 10, 04:30:00 GMT+0 - Identified - [portal.chpc.utah.edu](http://portal.chpc.utah.edu) will be inaccessible briefly for scheduled maintenance at 9:30 p.m. MST on Tuesday, December 9. Dec 10, 04:30:01 GMT+0 - Identified - Maintenance is now in progress Dec 10, 04:45:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 15 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>04:30:00</var> GMT+0</small><br><strong>Identified</strong> - [portal.chpc.utah.edu](http://portal.chpc.utah.edu) will be inaccessible briefly for scheduled maintenance at 9:30 p.m. MST on Tuesday, December 9..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>04:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>04:45:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Wed, 10 Dec 2025 04:30:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmiz2eczl045i9s0s01yhuwxf https://uofu-chpc.instatus.com/maintenance/cmiz2eczl045i9s0s01yhuwxf Slurm upgrade on CHPC clusters (granite, notchpeak, kingspeak, lonepeak, and redwood) Type: Maintenance Duration: 9 hours Affected Components: HPC clusters, Open OnDemand May 13, 14:00:01 GMT+0 - Identified - Maintenance is now in progress May 13, 23:00:00 GMT+0 - Completed - Maintenance has completed successfully May 13, 14:00:00 GMT+0 - Identified - This upgrade will cause disruptions to job submission. Existing jobs should run without issue or interruption, but there will be a period of time during which job submission will not work. Please see <https://chpc.utah.edu/news/read.php?source=recent&article=20250501%5Fslurm%5Fupgrade> for more information. Type: Maintenance

Duration: 9 hours

Affected Components: ,

<p><small>May <var data-var='date'> 13</var>, <var data-var='time'>14:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>May <var data-var='date'> 13</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - This upgrade will cause disruptions to job submission. Existing jobs should run without issue or interruption, but there will be a period of time during which job submission will not work. Please see <https://chpc.utah.edu/news/read.php?source=recent&article=20250501%5Fslurm%5Fupgrade> for more information..</p> ]]> Tue, 13 May 2025 14:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cmafj9ixo001lmz46rv45k3pc https://uofu-chpc.instatus.com/maintenance/cmafj9ixo001lmz46rv45k3pc Open OnDemand upgrade Type: Maintenance Duration: 10 minutes Affected Components: Open OnDemand Apr 20, 03:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 20, 03:00:00 GMT+0 - Identified - Please see <https://chpc.utah.edu/news/read.php?source=recent&article=20250419%5Fondemand%5Fupgrade> for more information. Apr 20, 03:10:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 10 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Identified</strong> - Please see <https://chpc.utah.edu/news/read.php?source=recent&article=20250419%5Fondemand%5Fupgrade> for more information..</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:10:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Sun, 20 Apr 2025 03:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cm9nbo7pr005xsk1yd0p5djes https://uofu-chpc.instatus.com/maintenance/cm9nbo7pr005xsk1yd0p5djes Datacenter cooling upgrade Type: Maintenance Duration: 1 day and 12 hours Affected Components: HPC clusters Mar 25, 12:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 27, 00:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 25, 12:00:00 GMT+0 - Identified - This is a planned downtime at the Downtown Data Center to perform electrical work for the new cooling system, and **will impact the** **general environment clusters lonepeak, kingspeak, notchpeak, and granite.** **Parts of the protected environment including the redwood cluster may be shut down for system administration work as well.** The electrical work by the DDC staff is expected to take the entire day on the 25th, and systems will be brought up the following day after the power outage and any system admin work is complete. Type: Maintenance

Duration: 1 day and 12 hours

Affected Components:

<p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>12:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 27</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 25</var>, <var data-var='time'>12:00:00</var> GMT+0</small><br><strong>Identified</strong> - This is a planned downtime at the Downtown Data Center to perform electrical work for the new cooling system, and **will impact the** **general environment clusters lonepeak, kingspeak, notchpeak, and granite.** **Parts of the protected environment including the redwood cluster may be shut down for system administration work as well.** The electrical work by the DDC staff is expected to take the entire day on the 25th, and systems will be brought up the following day after the power outage and any system admin work is complete..</p> ]]> Tue, 25 Mar 2025 12:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cm84slz9l00fd1pkvs0yxfper https://uofu-chpc.instatus.com/maintenance/cm84slz9l00fd1pkvs0yxfper Datacenter power maintenance Type: Maintenance Duration: 8 hours Affected Components: HPC clusters Mar 16, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully Mar 16, 14:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 16, 14:00:00 GMT+0 - Identified - Rocky Mountain Power has just notified us of a planned power outage to the Downtown Data Center and vicinity this Sunday, March 16, at 9am to address some urgent electrical changes for the city. The CHPC systems with power backed up by generators (the Protected Environment, network, data storage, and virtual machines) will be fine, but the systems with battery backup only will need to be shut down. **Therefore we will shut down the general environment clusters lonepeak, kingspeak, notchpeak, and granite at 8AM on Sunday March 16**. We estimate these servers will be down for several hours. Type: Maintenance

Duration: 8 hours

Affected Components:

<p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>14:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - Rocky Mountain Power has just notified us of a planned power outage to the Downtown Data Center and vicinity this Sunday, March 16, at 9am to address some urgent electrical changes for the city. The CHPC systems with power backed up by generators (the Protected Environment, network, data storage, and virtual machines) will be fine, but the systems with battery backup only will need to be shut down. **Therefore we will shut down the general environment clusters lonepeak, kingspeak, notchpeak, and granite at 8AM on Sunday March 16**. We estimate these servers will be down for several hours..</p> ]]> Sun, 16 Mar 2025 14:00:00 +0000 https://uofu-chpc.instatus.com/maintenance/cm84sjb4200ff8o0uaje1lf0f https://uofu-chpc.instatus.com/maintenance/cm84sjb4200ff8o0uaje1lf0f