Nvidia GPU driver critical update

Resolved

Major outage

Started 5 months agoMay 20, 2026Lasted 43 hours 5 minutesMay 20, 202611:00 PMMay 22, 20266:05 PMUTC

Affected

General Environment (GE)

HPC clusters

Windows servers

Virtual machines (VMs)

Protected Environment (PE)

HPC clusters

Updates

Resolved
May 22, 2026 at 6:05 PMUTC
Resolved
May 22, 2026 at 6:05 PMUTC
This incident has been resolved.
Update
May 21, 2026 at 11:12 PMUTC
Update
May 21, 2026 at 11:12 PMUTC
All the HPC cluster GPU drivers were updated and queues were released. Most of standalone servers have been updated as well.
Update
May 21, 2026 at 3:25 PMUTC
Update
May 21, 2026 at 3:25 PMUTC
We are continuing to work on a fix for this incident.
Identified
May 20, 2026 at 11:00 PMUTC
Identified
May 20, 2026 at 11:00 PMUTC
Nvidia released new GPU drivers that address critical vulnerability, https://nvidia.custhelp.com/app/answers/detail/a_id/5821
All systems with Nvidia GPUs is affected. We will be updating the drivers, or have updated them already, on HPC clusters, owner interactive nodes such as the Cryosparc machines, and Virtual Machines. The processes running on GPUs have to be killed for the driver update to apply. On the Linux systems, the update should not require a reboot, but, it's possible that some machines will have to be rebooted.
If you have any questions, please, contact our helpdesk, helpdesk@chpc.utah.edu

CHPC - Nvidia GPU driver critical update – Incident details