ZA North Azure Connectivity
Resolved
Jul 29 at 03:53pm UTC
As we have now received confirmation from Microsoft the issue is deemed resolved from their side, we are closing this issue. We will await further RCA from Azure, but their current provided Root Cause is an unexpected increase in traffic - most likely a large scale DDoS attack given the number of regions affected.
Update from Microsoft (MO406867)
Current status: Our monitoring telemetry indicates that the issue has recovered. We're continuing to actively monitor the environment and perform a set of proactive changes to prevent this type of issue from reoccurring.
Scope of impact: This issue is specific to a subset of traffic routed through infrastructure in South Africa, Brazil, North America, Canada, Japan and Korea. This issue is also causing impact for users located in regions surrounding the infrastructure hosted in those locations.
Root cause: An unexpected increase in user requests resulted in network congestion.
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 02:29pm UTC
Update from Microsoft (MO406867)
Current status: Our monitoring telemetry indicates that the issue has recovered. We're continuing to actively monitor the environment and perform a set of proactive changes to prevent this type of issue from reoccurring.
Scope of impact: This issue is specific to a subset of traffic routed through infrastructure in South Africa, Brazil, North America, Canada, Japan and Korea. This issue is also causing impact for users located in regions surrounding the infrastructure hosted in those locations.
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 02:07pm UTC
We continue to see no further issues, but are awaiting for an official confirmation of resolution from Microsoft
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 01:40pm UTC
We have seen zero packet loss and latency spikes for past 10 minutes.
We will keep issue open pending official confirmation of resolution
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 01:30pm UTC
Latest update from Microsoft
We've determined an unexpected spike in traffic is causing network congestion for some customers, resulting in the impact seen. We are currently applying multiple mitigation strategies to alleviate the connectivity issues. An update will be provided in the next 60 minutes or as events warrant.
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 01:00pm UTC
Further update from Microsoft (MO406867)
We’ve identified a potential networking issue resulting in impact. We’re reviewing service monitoring telemetry to verify that this is the root cause, before formulating a plan for remediation.
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 12:50pm UTC
We are seeing a change in pattern of the issue - previously we would see periodic packet loss and prolonged latency spikes.
We are currently seeing shorter periods of packet loss and no latency spikes. This should lead to an improvement in call quality.
Microsoft confirm issue began at 10am SAST - we are still waiting on an update as to a fix. However we hope this change in pattern indicates a solution is in progress.
We fully appreciate the severity of this issue, and will await a full RCA from Azure in due course.
Affected services
Voice - ZA North (JNB)
Updated
Jul 29 at 12:08pm UTC
Update from Microsoft
We've identified a potential root cause and are currently investigating mitigation options. An update will be provided in the next 60 minutes or as events warrant.
Affected services
Voice - ZA North (JNB)
Created
Jul 29 at 09:30am UTC
Azure ZA North Region is experiencing periodic periods of increased latency to the public internet. This will affect call quality.
We await an official announcement with an ETA from Azure.
Affected services
Voice - ZA North (JNB)