CHPC - Protected Environment resources offline for troubleshooting of network issue (updated Friday, January 16) – Maintenance details

All systems operational

JAN15

Protected Environment resources offline for troubleshooting of network issue (updated Friday, January 16)

CompletedJanuary 15, 2026 at 3:00 AM – January 20, 2026 at 10:33 PMUTC

Affects

General Environment (GE)

Under maintenance from 3:00 AM to 10:33 PM

HPC clusters

Under maintenance from 3:00 AM to 10:33 PM

Updates
  • Completed
    January 20, 2026 at 10:33 PMUTC
    Completed
    January 20, 2026 at 10:33 PMUTC

    Moving to an incident. Issues are ongoing.

  • Update
    January 17, 2026 at 6:43 AMUTC
    Update
    January 17, 2026 at 6:43 AMUTC

    Network and system administrators at the CHPC have isolated and triaged network issues and brought most systems back online. The Protected Environment is now functional and accessible to users. Some components are in a degraded or diminished state as a result of reconfigurations to bring the PE back online quickly and will require further troubleshooting next week to return to a normal, fully operational state. We appreciate your patience with this issue.

  • Update
    January 16, 2026 at 11:58 PMUTC
    Update
    January 16, 2026 at 11:58 PMUTC

    Network administrators are continuing work to diagnose packet loss issues by bringing devices and interfaces online individually. Network issues persist.

  • Update
    January 16, 2026 at 4:45 PMUTC
    Update
    January 16, 2026 at 4:45 PMUTC

    CHPC system and network administrators are taking systems in the Protected Environment offline to expedite triage and troubleshooting of networking issues.

  • Update
    January 16, 2026 at 1:34 AMUTC
    Update
    January 16, 2026 at 1:34 AMUTC

    Protected Environment network issues persist. System and network administrators at the CHPC have engaged vendors of networking hardware and are on-site at the data center to address the problem as quickly as possible.

  • In progress
    January 15, 2026 at 3:00 AMUTC
    In progress
    January 15, 2026 at 3:00 AMUTC
    Maintenance is now in progress
  • Planned
    January 15, 2026 at 3:00 AMUTC
    Planned
    January 15, 2026 at 3:00 AMUTC

    Scheduled maintenance on power infrastructure at the Downtown Data Center will require all General Environment clusters (granite, notchpeak, kingspeak, and lonepeak) and standalone biochemistry and cryo-EM servers to be taken offline from the evening of January 14 to the morning of January 15.

    Critical services, storage systems, virtual machines (VMs), and the redwood cluster in the Protected Environment are on generated power and should not be affected by this outage.