Subject: Dual Fiber cut causing connectivity impact to Federal Way, WA data center
Incident Status: Resolved
Estimated Resolution: N/A
Next Update: N/A Final Update
Event Start: May 16, 02:57 AM PST
Event End: May 17, 00:20 AM PST
Event Duration: 21 hours and 23 minutes
Update #5 - May 17, 07:52 PST
Splicing work for all active fiber pairs in the 288F have been completed and services have been running stable for multiple hours at this point now. All current customers have had credits applied to their account or open invoice for one month of service as we may need to deal with these items, but you shouldn't have to! Full root cause assessment (RCA) is still pending from both transit providers and we will post a full analysis once we are able to review and create our own. Prior to this fiber cut due to theft we had 100% uptime on our network, after this event we are currently as of this writing at 99.1%
Update #4 - May 16, 19:30 PST
Splicing work to splice in the repair 288F cable is underway, currently no ETA on restoration at this time.
Update #3 - May 16, 14:48 PST
Provider #2 has a team on-site and working to repair the impacted fiber for their transit path.
Update #2 - May 16, 11:32 PST
Provider #2 has identified a fiber break in their physical network path that runs through Tacoma, WA and have engaged fiber technicians to work on splicing repairs for the break. No resolution estimate given at this time.
Update #1 - May 16, 11:17 PST
Based on our contact with our providers and their investigations, we have concluded that the issue affecting the data center is due to multiple fiber cuts. Our providers are working diligently to isolate and restore services to the data center. We use redundant providers and routers to ensure connectivity and reduce the risk of such events.
While the full root cause is still being determined, it is evident that an overlapping segment in the diverse provider paths has contributed to this risk. A fiber cut is a physical break in the fiber-optic cable, which can occur due to various reasons such as construction activities, natural disasters, or accidents. Although fiber cuts are rare, they can happen even with redundant providers and paths. Our team is committed to minimizing these risks and ensuring reliable service.
For our existing customers, we will be adding a credit to your account equivalent to a full month of service.
Initial Update:
Starting on May 16, 02:55 AM PST our network located at 32275 32ND AVENUE SOUTH in Federal Way, WA area may be experiencing loss of network connectivity. We and the data center are looking into the issue but currently have no estimated time of resolution (ETR).
