Incident Overview: On November 3rd, at approximately 4:00 PM CST, our ISP experienced a network disruption that impacted a subset of customer traffic. This incident temporarily prevented certain external sources from reaching specific destinations within our network. While most services continued to operate normally, certain destinations were unreachable from some external sources.
Cause of the Incident: The issue was traced to a switch within our ISP’s network infrastructure that handles Layer2 traffic for part of our services. Layer2 ports responsible for directing traffic to specific endpoints experienced forwarding issues, which impacted connectivity. Once the affected switch was isolated and traffic rerouted to an alternative path, services began to stabilize.
Resolution Steps: Our ISP promptly took the impacted switch out of service, conducted diagnostic checks, and applied updates to address the issue and prevent similar incidents in the future. After thorough testing, the switch was restored to service, and stable operations have resumed.
Impact on Services: This incident may have affected connectivity for customers whose traffic relied on the impacted switch paths. We apologize for any inconvenience and appreciate your patience as we worked with our ISP to restore full functionality. We are committed to ensuring a robust and resilient network and continue to collaborate with our ISP to minimize future risks.