CHPC - Critical security Linux OS update on Monday, October 5 – Maintenance details

The system-wide downtime has been completed successfully. Critical security patches addressing multiple vulnerabilities (CVEs) have been applied. The General Environment clusters, Protected Environment cluster, standalone servers, and virtual machines are now available. Partitions have been released from their reservations, and the sys branch and app repo finished syncing to the Protected Environment overnight. If you notice any outstanding problems, please let us know at helpdesk@chpc.utah.edu. Thank you for your patience with the downtime.

All CHPC services are back in operation, if you see any outstanding issue, please, let us know at helpdesk@chpc.utah.edu.

The app repo in the Protected Environment has been synced. The CHPC storage team is removing the prior version at /scratch/general/pevast/app-repo-ORIGSL. The updated version is available at /scratch/general/pevast/app-repo.

The kingspeak cluster has been returned to service.

The redwood cluster in the Protected Environment has been returned to service.

Please note: The team at the CHPC is syncing files to the app repo—a directory used to store large, common files like large language model weights, bioinformatics reference databases, and protein-folding models—in the Protected Environment. This involves a transfer of tens of terabytes of data and will take some time. We are releasing the reservation on redwood now to allow researchers to begin submitting jobs again, but some software may not work as expected until the transfer is complete. If you use software like (including, but not limited to) AlphaFold and similar structural prediction tools, inference engines such as Ollama or vLLM with common models, cryo-EM processing tools, or alignment or sequencing tools that rely on large datasets, common files may be temporarily inaccessible while the sync is taking place (which may take on the order of 10 hours). There is a temporary directory, available at /scratch/general/pevast/app-repo-ORIGSL, with the prior contents of the app repo (before the sync began) if you need to access such files in the immediate future. This directory is temporary and will be removed when the sync of the app repo has finished. Accordingly, you may need to update your scripts to both (a) use the temporary location now and (b) return to the standard location once the sync has finished.

The narwhal, beehive, and pe-rds01 Windows servers in the General and Protected Environments have been returned to service.

The Citadel environment (including Linux and Windows systems) has been returned to service.

The notchpeak cluster has been released; thank you for your patience. The CHPC team is continuing work on other clusters.

The lonepeak and granite clusters have been released from maintenance and are available for use. CHPC system administrators are continuing work on other clusters. We appreciate your patience as we work to update systems and bring them back online.