This incident has officially been closed by Azure now additional bandwidth is online the services are no longer at risk.

If you require more information about this incident, please feel free to reach out to your account manager for more details.

Latest Update from Azure

With the current bandwidth demand and network utilization, all services are currently healthy with no issues detected. We are still detecting some minor latency for small transient periods. In preparation for demand increases, we are on track to take all available measures to increase capacity. Substantial additional capacity remains on track to be provisioned by March 19, 2024.

We continue to closely monitor all telemetry to ensure network stability leading into the South Africa region's business hours on Monday, March 18, 2024.

The majority of Azure Services affected by this issue have returned to normal functionality. Customers should see recovery and minimal impact at this time.

--

We continue to see a good level of service across all carriers other than Seacom. We also see significant improvements on zScaler CPT IV since around 09:30am local time in SA, which was the only monitored zScaler region in SA still affected.

Update from Azure is that international connectivity is restricted at present to 400Gbps, and they continue to experience issues with stability on additional capacity at this time. An additional 1Tbps of capacity is due online by Tuesday.

We continue to see normal levels of service all on carriers other that Seacom.

As per our last update - we continue to see good service across Africa with the exception of Seacom which is still routing domestic traffic via Europe and USA.

Latest Update from Azure:

We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa.

In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions.

We've added the additional network capacity, however, there has been a slight amount of instability in the newly added links. We're working with the partner carrier to investigate the issues. Additionally, we're planning to add additional capacity within the region over the weekend and into next week to bring more headroom into the region.

Once our additional capacity is stable, we plan to re-enable Azure Front Door (AFD) routing within the South Africa region to improve some application performance.

As we continue to work on further increasing the capacity, our telemetry shows signs of recovery, and packet drops have reduced significantly.

Many of the Azure Services that were affected by this issue earlier have now returned to normal functionality. We are validating for any remaining impacted services, however customers should notice considerable improvements.

Latest from Azure is they are experiencing difficulties still bringing the new capacity online and updated ETA is now 18:00 UTC.

Connectivity across the board in South Africa remains good with the exception of Seacom.

We have seen normal levels of latency return on all Vodacom links. We are seeing issues remain with Seacom where capacity remains constrained. All other mainstream carriers and ISPs continue to show normal performance with low latency and zero packet loss.

Updates to follow.

Azure now estimate this work to have completed by 14:00 UTC - we continue to monitor.

Updates to follow.

We continue to see the issue limited to Vodacom and Seacom at this time. Azure have also updated the ETA back to 02:00 UTC for the additional capacity.

Update from Azure

Current Status: We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa.

In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions.

The overall network health for the region has been steadily improving. Many of the Azure Services that were affected earlier by this issue have now returned to their normal levels. We are still validating for any remaining services, however the users for these would already notice considerable improvements.

We are working on adding more capacity and we anticipate that these steps will help further with ongoing capacity bottleneck. The new estimated time of completion for this activity is 2:00 UTC on 15 March 2024

We continue to see elevated latency to Seacom and Vodacom.

While we see no other issues at present, should issues persist on Friday, for South African clients we have provisioned resources within Amazon Web Services in Cape Town, and are able to move voice service there on a temporary basis.

Update from Azure

We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa.

In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions.

Our efforts through simultaneous workstreams to reduce non-critical traffic has helped to normalize the network bandwidth. We believe most of the customers should no longer be seeing impact and we are continuing to validate the health metrics across the Azure Services which were impacted initially.

Our teams continue to work on bringing additional capacity to minimize any impact across services.

We expect that this work should take about 4 hrs from now, which would be around 22:00 UTC on 14 March 2024.

International latency has now recovered to usual levels. Microsoft themselves continue to re-route local 365 resources to both Europe and South Africa West.

We continue to see higher latency to Vodacom and Seacom, but packet loss is now zero on these links too.

We have seen significant improvements on international connectivity in the past 10 minutes. While latency is still around 35% higher than usual, we are currently seeing zero packet loss (down from up to 80% packet loss on international links earlier in the incident)

We have seen service largely restored domestically. We continue to see issues remain with ISPs that use Seacom or Vodacom for upstream connectivity or SIP trunks. We are also still seeing high international connectivity latency and packet loss, but significant improvement on earlier.

Azure Update

Current Status: We have determined that multiple fiber cables on the West Coast of Africa -- WACS, MainOne, SAT3, ACE – have been impacted which reduced total capacity supporting our Regions in South Africa.

In addition to these cable impacts, the on-going cable cuts in the Red Sea -- EIG, Seacom, AAE-1 -- are also impacting capacity on the East Coast of Africa.

This combination of incidents has impacted all Africa capacity – including other Cloud providers and public Internet as well.

Current Workstreams: We are exploring the following mitigation workstreams

Working with our local network partner to fix defective links.
Exploring traffic redirection and optimization on our network

We estimate that these workstreams will take up to 12 hours to complete.

Customers who are experiencing severe disruption to their operations may want to consider their BCDR options.

The next update will be provided in 4 hours or as events warrant.

Update from Azure

Current Status: We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa.

In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting capacity on the East Coast of Africa.

This combination of incidents has impacted all Africa capacity – including other Cloud providers and public Internet as well.

Current Workstreams: We are exploring the following mitigation workstreams

Working with our local network partner to fix defective links.
Exploring traffic redirection and optimization on our network
The next update will be provided in 60 minutes, or as events warrant

This issue continues to affected certain customers accessing resources in Azure South Africa. The issue is affecting access to both Azure and Microsoft 365 across South Africa, as well as other service providers.

Update from Azure

We are currently investigating a networking issue that has resulted in higher-than-expected latency for users whose resources are hosted in or connect with South Africa North and West. The next update will be provided in 60 minutes, or as events warrant.

Azure have notified that they are rerouting connections to alternate components to mitigate impact.

Update from Azure

Starting at 10:30 UTC on 14 Mar 2024, Azure Services in South Africa North and South Africa West may experience service degradation and availability issues. Additionally, customers may have observed higher than expected latency and packet drops for resources hosted in these regions.

Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant.

Azure have assigned the reoccurrence issue VT60-RPZ

https://app.azure.com/h/VT60-RPZ/8ffce0

This issue has reoccured - monitoring

Azure have confirmed the fibre cut was resolved, and have closed the issue

Update from Azure:

In addition to addressing a fiber cut and assessing how quickly we can expedite its repair, our efforts to restore optimum network performance have included reducing the reservation on certain links, leading to a notable decrease in packet drops and an overall improvement in services.

While we have optimized and load balanced our network to utilize all available capacity, we are still assessing impact with some of our routing on segments whether or not we can take on a peak load.

Azure Front Door is planning to redirect traffic away from South Africa to another region, provided that no further packet drops occur. The next update will be provided in 2 hours, or as events warrant.

Update from Azure

We determined that this issue was caused by a fiber cut, resulting in congestion specifically affecting the traffic in the Africa region. This has impacted the connectivity between South Africa and Europe, causing disruptions to your services. Our teams are actively working to address these issues and restore normal operations. The next update will be provided in 60 minutes, or as events warrant.

Azure have notified us of network issues within South Africa North and South Africa West. We are monitoring

https://app.azure.com/h/_NF5-7PG/8ffce0