Monitoring Alert Unhealthy - Connections being lost
Incident Report for xByte Cloud
Postmortem

Incident Overview: On November 3rd, at approximately 4:00 PM CST, our ISP experienced a network disruption that impacted a subset of customer traffic. This incident temporarily prevented certain external sources from reaching specific destinations within our network. While most services continued to operate normally, certain destinations were unreachable from some external sources.

 

Cause of the Incident: The issue was traced to a switch within our ISP’s network infrastructure that handles Layer2 traffic for part of our services. Layer2 ports responsible for directing traffic to specific endpoints experienced forwarding issues, which impacted connectivity. Once the affected switch was isolated and traffic rerouted to an alternative path, services began to stabilize.

 

Resolution Steps: Our ISP promptly took the impacted switch out of service, conducted diagnostic checks, and applied updates to address the issue and prevent similar incidents in the future. After thorough testing, the switch was restored to service, and stable operations have resumed.

 

Impact on Services: This incident may have affected connectivity for customers whose traffic relied on the impacted switch paths. We apologize for any inconvenience and appreciate your patience as we worked with our ISP to restore full functionality. We are committed to ensuring a robust and resilient network and continue to collaborate with our ISP to minimize future risks.

Posted Nov 04, 2024 - 11:15 CST

Resolved
We are still reviewing the incident and awaiting the release of a full post-mortem report.
Posted Nov 03, 2024 - 19:06 CST
Monitoring
Our team will be reaching out to affected customers to confirm that functionality is as expected, as we are currently not seeing any impacted subnets.

We are still reviewing the incident and awaiting the release of a full post-mortem report.
Posted Nov 03, 2024 - 18:44 CST
Update
Our network team reports that traffic from previously affected subnets is now successfully reaching our network and establishing connections.
Posted Nov 03, 2024 - 18:18 CST
Update
The service issue persists, affecting specific subnets for inbound requests. We are still assessing the incident and are not yet able to provide confirmed details.
Posted Nov 03, 2024 - 18:13 CST
Update
Our network team is still observing connection losses on approximately 50% of our inbound requests. We are continuing to investigate the issue beyond our network.
Posted Nov 03, 2024 - 17:32 CST
Update
We’re still observing connection losses in transit to our network, and our network team is actively continuing the investigation.
Posted Nov 03, 2024 - 16:59 CST
Update
The xByte Cloud network monitors are currently showing all systems are healthy. We’re reaching out to our upstream ISP providers to investigate any specific connection routing issues.
Posted Nov 03, 2024 - 16:30 CST
Investigating
xByte Cloud is investigating a network connection issue that is impacting connection requests from a specific location. Further details will be provided as soon as they are available.
Posted Nov 03, 2024 - 16:18 CST
This incident affected: Network Infrastructure (Zone C - US Central).