CHPC - Nvidia GPU driver critical update – Incident details

Nvidia GPU driver critical update

ResolvedMajor outage43 hours 5 minutes

Affected

  • ResolvedUTC
    This incident has been resolved.
  • UpdateUTC

    All the HPC cluster GPU drivers were updated and queues were released. Most of standalone servers have been updated as well.

  • UpdateUTC
    We are continuing to work on a fix for this incident.
  • IdentifiedUTC

    Nvidia released new GPU drivers that address critical vulnerability, https://nvidia.custhelp.com/app/answers/detail/a_id/5821

    All systems with Nvidia GPUs is affected. We will be updating the drivers, or have updated them already, on HPC clusters, owner interactive nodes such as the Cryosparc machines, and Virtual Machines. The processes running on GPUs have to be killed for the driver update to apply. On the Linux systems, the update should not require a reboot, but, it's possible that some machines will have to be rebooted.

    If you have any questions, please, contact our helpdesk, helpdesk@chpc.utah.edu