Incident Report: Service Disruption - Loadbalancer

Date/Time of Incident:

• 1/15/2025, 2:02 PM MST: Degraded gateway service reported. Symptoms observed include:
  • Gateway service excessive TCP queue depth.
• 1/15/2025, 2:09 PM MST: All services fully restored.

Actions Taken:

• Load Balancer: Even though services were restored at the primary load balancer, we migrated processing to the secondary load balancer so we could gather logs and reboot the primary load balancer.
• Support Ticket: We have an open ticket with the loadbalancers technical support team to review the logs and evaluate processes to prevent similar issues in the future.