Calling Impairment 6/13/2023

Calling impairment.Subscribe

Monitoring - NDP has been restored. At this time we believe all Partner and Customer facing services have been restored. Please contact the Control Tower with any issues you are still experiencing. We are continuing to restore internal tooling as well as unrestored capacity for Partner and End User services. We will continue to post to this incident as new updates occur.
Jun 14, 2023 - 03:44 EDT

Update - Inbound Fax to Portal has been restored. Restoration of NDP is underway.
Jun 14, 2023 - 01:46 EDT

Update - We believe ReachUC has been restored. Please provide feedback to our Customer Service if you are still experiencing issues with ReachUC. We are turning our attention to restoring Inbound Instant Fax Portal (ETA: 60 minutes) and NDP (ETA: 90 minutes).
Jun 13, 2023 - 22:39 EDT

Update - Update: The necessary elements to restore ReachUC have been established. We are currently working on validating the environment and then will be making the necessary DNS changes to transition clients to the new environment. We anticipate this will happen with the next 45 minutes. We will then turn our attention to restoring NDP and Instant Fax Portal Inbound.
Jun 13, 2023 - 21:08 EDT

Update - Update: Egress in LAS (i.e. outbound calling to the PSTN in the Las Vegas, NV region) has been completely restored. This includes 911 calling to local PSAP and international. This completes all Egress restore activities.

All outbound calling to the PSTN (both "house" and "BYOT") should be completely restored.

Work to restore ReachUC is underway.

Known affected services remaining:
1. ReachUC
2. SD-WAN (Infiot, RabbitRun & SimpleWAN)
3. Inbound Instant Fax (Fax to Portal)
Jun 13, 2023 - 18:02 EDT

Update - Update: Egress in DAL (i.e. outbound calling to the PSTN in the Dallas, TX region) has been completely restored. This includes 911 calling to local PSAP and international. Work continues to restore complete egress functionality in the remaining zone: Las Vegas.

Based upon on the observed pace of the restore process we anticipate all services will be restored this evening. We are pursuing the restoration of these services in the following order:
1. Egress PSTN calling
2. ReachUC
3. SD-WAN (Infiot, RabbitRun & SimpleWAN)
4. Inbound Instant Fax (Fax to Portal).
Jun 13, 2023 - 17:45 EDT

Update - Update: Egress in the ATL data center has been completely restored. This includes 911 calling to local PSAP and international. Work to restore complete egress functionality in the remaining zones continues.
Jun 13, 2023 - 17:25 EDT

Update - Update: Egress in the NYJ data center has been completely restored. This includes 911 calling to local PSAP and international. Work to restore complete egress functionality in the remaining zones is underway.

AWS S3 stability is now facilitating the VM restore process, enabling work to progress on recovering additional services, such as Instant Fax, ReachUC, and Device Provisioning.
Jun 13, 2023 - 17:11 EDT

Update - AWS us-east-1 is currently experiencing issues which is delaying our ability to restore VMs. https://health.aws.amazon.com/health/status

We are continuing to restore services via alternative, but slower methods, until S3 functionality is restored.
Jun 13, 2023 - 16:05 EDT

Update - 911 calls are now completing to the national answering center. We are working to restore routing to local PSAP based on address.
Jun 13, 2023 - 15:51 EDT

Update - We are continuing to work on a fix for this issue.
Jun 13, 2023 - 15:50 EDT

Update - We believe BYOT Ingress, i.e. calls from the PSTN destined for the PBX, are backup and operational.

We are turning our complete attention to restoring outbound calls to 911.
Jun 13, 2023 - 15:46 EDT

Update - Update: We continue to focus on restoring Ingress BYOT (i.e. BYOT DIDs).

Impact will still be felt on such additional services as:
- Instant Fax (Inbound)
- ReachUC registration & connectivity
- NDP Access
Jun 13, 2023 - 14:48 EDT

Update - ENUM hosts have been restored in all data centers. Internal testing suggests inbound calling from the PSTN to "house" DIDs has been restored. We are transitioning our focus to restoring Ingress BYOT (i.e. BYOT DIDs).
Jun 13, 2023 - 13:15 EDT

Update - Update: We believe we are 15 minutes away from having successfully processing inbound calls from the PSTN.
Jun 13, 2023 - 12:59 EDT

Update - We have rebuilt enum01-las, the host type responsible for processing inbound calls from the PSTN. We are working on circulating this host to 3 other data centers. Predicted ETA for this work is 12:30 PM ET. Between now and then inbound calling completion rates should increase until full restore of this service.
Jun 13, 2023 - 11:16 EDT

Update - UPDATE: We are continuing to work on a fix for this issue. More updates to follow.
Jun 13, 2023 - 10:31 EDT

Update - We are continuing to work on a fix for this issue. More updates to follow.
Jun 13, 2023 - 09:32 EDT

Update - Dear Partner,
There is an issue affecting virtual hosts in all data centers. This has originally broken all PSTN connectivity. We have since restored outbound PSTN connectivity.

We are currently working on restoring other affected services in the order listed below which include:
Inbound PSTN access
ReachUC registration & connectivity
NDP Access

This is not a comprehensive list of services that are negatively impacted by this outage. Current ETA of repair for Inbound PSTN access is the second half of the day.
Jun 13, 2023 - 08:09 EDT

Identified - UPDATE Outbound PSTN calling should be working. We are still working to bring up the inbound calling from the PSTN. More updates to come.
Jun 13, 2023 - 07:12 EDT

Update - UPDATE: We are seeing issues related to the VMWare infrastructure and are attempting a fix at this time. Update to follow.
Jun 13, 2023 - 06:45 EDT

Investigating - Dear Partner,

We are aware of issues related to PSTN call completion across data centers. We are working to identify and resolve. More updates to follow.

Sincerely,
ThreePBX Engineering
Jun 13, 2023 - 06:02 EDT