CHPC - Notice history

Experiencing partially degraded performance

98.67% uptime

Homepage and documentation - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Portal - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

License servers (commercial software) - Operational

99.98% uptime
Sep 2026 · 99.93%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Group- and project-specific websites (hosted on VMs) - Operational

92.04% uptime
Sep 2026 · 76.13%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Helpdesk platform - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

GitLab - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026
100.0% uptime

HPC clusters - Degraded performance

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Open OnDemand - Operational

99.97% uptime
Sep 2026 · 100.0%Oct · 99.91%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Windows servers - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Virtual machines (VMs) - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Computational servers, independent of clusters - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Storage systems - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Data Transfer Nodes (DTNs) - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026
99.93% uptime

HPC clusters - Degraded performance

99.86% uptime
Sep 2026 · 99.59%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Open OnDemand - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Windows servers - Operational

99.63% uptime
Sep 2026 · 99.93%Oct · 98.99%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Virtual machines (VMs) - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Computational servers, independent of clusters - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Storage systems - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Data Transfer Nodes (DTNs) - Operational

100.0% uptime
Sep 2026 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2026
Oct 2026
Nov 2026

Slack - Operational

Notice history

View current status

Nov 2026

No notices reported this month

Oct 2026

Linux OS kernel vulnerability patching on CHPC clusters
IdentifiedDegraded performance
  • Another set of critical vulnerability patches requires updating OS kernel on CHPC Linux clusters. We will be starting rolling reboots of compute nodes momentarily that will update to the latest kernel. Unfortunately, just today another vulnerability has been found which does not have a patch, which means that another rolling reboot will be needed as soon as the patch is released. That is, there is a potential for two rolling reboots in the near future.

    This process will proceed as follows:

    • Compute nodes will be drained so they stop accepting new jobs. Running jobs will be allowed to finish normally.

    • Once a node is idle it will be rebooted onto the new kernel and go back into service automatically when it comes up.

    • Nodes still to be updated will show the drain reason "auto_kernel_update" in sinfo -R.

    • With fewer nodes accepting work, some jobs may wait in the queue longer than usual, especially large or long-walltime jobs.

    • To accelerate the rollong reboot process, guest, OSG and freecycle jobs may be cancelled.

    We thank you for your understanding in this difficult period of many newly discovered vulnerabilities. Contact helpdesk@chpc.utah.edu if you notice any irregularities.

    UTC
Issues with pe-rds01 (Windows server in Protected Environment) on October 29
ResolvedPartial outage6 hours 45 minutes
  • ResolvedUTC

    CHPC system administrators returned pe-rds01 to service yesterday evening. Thank you, once again, for your patience.

  • InvestigatingUTC

    The CHPC is investigating recurring issues with pe-rds01, a Windows server in the Protected Environment. The system is currently unavailable while CHPC system administrators investigate. Thank you for your patience.

    While pe-rds01 is being investigated, the narwhal server remains available for use. For details, please see https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php.

ondemand.chpc.utah.edu is unavailable (undergoing urgent security updates); please use ondemand-class.chpc.utah.edu
ResolvedPartial outage1 hour 18 minutes
  • ResolvedUTC

    ondemand.chpc.utah.edu has been returned to service. If you continue to encounter any issues, please don't hesitate to reach out to us at helpdesk@chpc.utah.edu.

  • IdentifiedUTC

    ondemand.chpc.utah.edu is undergoing urgent security updates and it is currently unavailable. We apologize for the inconvenience and appreciate your patience, particularly as security issues have become increasingly common in recent months. The other General Environment Open OnDemand instance, ondemand-class.chpc.utah.edu, has already been updated and is available to users. Please use ondemand-class.chpc.utah.edu while updates are being applied to the main Open OnDemand system.

OCT26
gitlab.chpc.utah.edu: CHPC GitLab will have urgent downtime for important security update
CompletedMaintenance25 hours 35 minutes
  • CompletedOctober 27, 2026 at 5:35 PMUTC

    GitLab has been updated and it is now available for use. If you encounter any issues, please don't hesitate to contact the CHPC at helpdesk@chpc.utah.edu. Thank you for your patience with the update process.

  • UpdateOctober 26, 2026 at 10:32 PMUTC

    The security-related upgrade to GitLab is ongoing. The team at the CHPC is continuing work to restore the service as quickly as possible. Thank you again for your patience.

  • In progressOctober 26, 2026 at 4:00 PMUTC

    The Center for High Performance Computing will be upgrading its GitLab instance, gitlab.chpc.utah.edu, to apply urgent, necessary security updates and address recent critical vulnerabilities. During the upgrade, GitLab will be unavailable.

    The upgrade process requires the manual installation of a new release, and this process will likely take several hours, based on the prior experience of CHPC system administrators. Administrators hope to complete the upgrade today, but it is possible that the installation could extend to tomorrow.

    Thank you for your patience with this urgent security update. The CHPC applies essential security updates to maintain system security and keep your research data and software secure. If you have any questions or concerns, please contact the CHPC at helpdesk@chpc.utah.edu.

    Thanks again,

    The team at the Center for High Performance Computing

Sep 2026

Issues with Narwhal (Windows server in Protected Environment) on September 24
ResolvedPartial outage1 hour 2 minutes
  • ResolvedUTC

    The narwhal server is available again; the system ran out of memory and needed to be rebooted. We recommend moving new Protected Environment projects to the pe-rds01 server, described at https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php. If you have any questions or concerns, please don't hesitate to reach out to the CHPC at helpdesk@chpc.utah.edu.

  • InvestigatingUTC

    The CHPC is aware of issues with the Narwhal server, which are a result of excessive load on the system. We recommend using pe-rds01 (see https://www.chpc.utah.edu/documentation/guides/pe-windows-servers.php#pe-rds01 for details) while Narwhal is inaccessible. Thank you for your patience.

Issues with connections to CHPC-hosted websites and virtual machines from outside of the University of Utah network
ResolvedMajor outage171 hours 50 minutes
  • ResolvedUTC

    Please contact helpdesk@chpc.utah.edu if you notice any continued connectivity issues to CHPC services.

  • UpdateUTC

    Some virtual machines and associated websites or services are now accessible following an emergency change request. Others remain inaccessible from outside of the University of Utah network and will likely be available following our campus partners' scheduled maintenance and planned implementation of firewall rules tonight.

  • IdentifiedUTC

    Campus partners plan to update firewall rules during scheduled maintenance tonight. In anticipation of the firewall rule changes, CHPC network administrators proactively submitted exception requests. We expect the requested exceptions to take effect with tonight's updates, which should resolve most access issues, though some systems may experience service disruptions or remain unavailable. We will address issues as quickly as we can. Thank you for your patience.

  • InvestigatingUTC

    The CHPC team is aware of issues with connections to some CHPC-hosted infrastructure, including virtual machines and associated websites and web services, from external networks. This is related to changes to University of Utah firewall rules, not CHPC infrastructure, and network and system administrators at the CHPC are in communication with campus partners to resolve issues as quickly as possible. Where feasible, we recommend connecting to the University of Utah VPN to access systems and services that are affected by firewall changes. We recognize that this may not be feasible for systems open to the public or collaborators at other institutions, and we are working to resolve access issues with our campus partners. Thank you for your understanding and patience.

Issues with access to CHPC Portal, including forms and user management features
ResolvedDegraded performance27 hours 51 minutes
  • ResolvedUTC

    The CHPC Portal is now accessible from outside the University of Utah network following an emergency change to campus firewall rules implemented by our partners. This is not an all-clear message for all virtual machines, websites, or services typically served outside of the university network; some systems remain inaccessible. Please see details and updates on other systems at https://uofu-chpc.instatus.com/cmu4efy1t006g0wph8k8j4ypc.

  • InvestigatingUTC

    We are currently investigating reports of issues with connections to the CHPC Portal and how it may relate to University of Utah campus network policy changes that began yesterday. If you encounter issues with the CHPC Portal or other CHPC-managed services, we recommend connecting to the University of Utah VPN, which may resolve network-related issues. Thank you for your patience.

Previous

Sep 2026 to Nov 2026

Next