CHPC - Degraded file system performance with various impacts (Open OnDemand sessions not starting, reduced input and output performance) – Incident details

All systems operational

Degraded file system performance with various impacts (Open OnDemand sessions not starting, reduced input and output performance)

Resolved
Degraded performance
Started 6 months agoLasted 20 hours 35 minutes

Affected

General Environment (GE)

Degraded performance from 7:30 PM to 1:50 AM, Operational from 1:50 AM to 4:06 PM

Open OnDemand

Degraded performance from 7:30 PM to 1:50 AM, Operational from 1:50 AM to 4:06 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved. CHPC systems have returned to responsive states. If you continue to encounter issues, please contact the CHPC at helpdesk@chpc.utah.edu.

  • Monitoring
    UTC
    Monitoring

    CHPC staff have identified a number of jobs that were impacting file system performance. The team at the CHPC canceled jobs with a significant impact, which has improved performance. Systems, including Open OnDemand, should now be more responsive. CHPC staff will continue to monitor the situation.

  • Investigating
    UTC
    Investigating

    We are receiving user reports of Open OnDemand sessions not starting. So far, issues have been reported on notchpeak and kingspeak but appear to be intermittent. A retry may resolve the issue in some cases. The CHPC is currently investigating the cause.