Incidents | QContact Ltd Incidents reported on status page for QContact Ltd https://status.qcontact.com/ https://d1lppblt9t2x15.cloudfront.net/logos/3da5f16f2ac09e7792e1805863f54301.png Incidents | QContact Ltd https://status.qcontact.com/ en Media 7 Health Event https://status.qcontact.com/incident/568634 Sat, 17 May 2025 10:59:00 -0000 https://status.qcontact.com/incident/568634#8e5a261ca0c97fe513f4b6bb61429af9cbda925b6aedd08565e8e1694295d4fc This incident has been resolved. The underlying host experienced hardware failure within Azure, and the server was re-provisioned on an alternative host. Media 7 Health Event https://status.qcontact.com/incident/568634 Sat, 17 May 2025 10:55:00 -0000 https://status.qcontact.com/incident/568634#9bb61b7d2e4008eafc10aef16b48c10a0162b6bb7fb887189f86d8a3d7b67e89 Investigating Azure ZA North Connectivity https://status.qcontact.com/incident/563681 Wed, 14 May 2025 10:47:00 -0000 https://status.qcontact.com/incident/563681#5d9131721ca88d39d72ee0ecc941f1f61dcba5c84a5b8251d3f52688607bfc15 Update from Azure: What happened? Between 23:30 UTC on 13 May 2025 and 08:20 UTC on 14 May 2025, you were identified as an Azure customer in the South Africa North and South Africa West regions who may have experienced intermittent connectivity loss and increased network latency. What went wrong, and why? A third-party telecom network disruption in the affected regions was identified as a contributing factor to this event. How did we respond? 23:30 UTC on 13 May 2025 - Customer impact began. 03:30 UTC on 14 May 2025 - As customer reports increased, we were alerted to the issue and promptly began our investigation. 05:00 UTC on 14 May 2025 - We identified that the issue was due to a third-party network disruption in the affected regions. 08:20 UTC on 14 May 2025 - We confirmed that the issue was self-healed with no need for manual intervention. We confirmed that customer impact was mitigated, and service functionality was fully restored. What happens next? This Mitigation Statement is the final communication for this incident. For details about which Azure incidents qualify for which Post Incident Reviews (PIRs), refer to https://aka.ms/AzurePIRs. The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring. To stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts. For broader guidance on preparing for cloud incidents, refer to https://aka.ms/incidentreadiness. Azure ZA North Connectivity https://status.qcontact.com/incident/563681 Wed, 14 May 2025 09:16:00 -0000 https://status.qcontact.com/incident/563681#5e4083d7ad231ce4112a3cf37de74681948f45f558e8e2664e506fdb5682738b Azure have added a service advisory https://app.azure.com/h/5_YX-JWG/8ffce0 Starting at 23:30 UTC on 13 May 2025, you were identified as an Azure customer in the South Africa North and South Africa West region who may have experienced intermittent connectivity loss and increased network latency in the region. This issue is now mitigated. Further details will be provided shortly. Azure ZA North Connectivity https://status.qcontact.com/incident/563681 Wed, 14 May 2025 08:27:00 -0000 https://status.qcontact.com/incident/563681#1c697e5f0b8c9e4894932388506da28d080d2518da0c08754b4eb23fcde544a9 We see DNS recovery and international connectivity restored. We will continue to monitor Azure ZA North Connectivity https://status.qcontact.com/incident/563681 Wed, 14 May 2025 06:38:00 -0000 https://status.qcontact.com/incident/563681#a25d5145ac66e3ca5bdf2de9729fc456467ecf46b1c8aa3a987f5337dc43dd89 An outage is occurring with Azure DNS infrastructure in South Africa North. We have implemented a workaround to use external DNS infrastructure. We also note international connectivity issues into ZA North from outside South Africa. We await details from Azure. Azure ZA North Connectivity https://status.qcontact.com/incident/563681 Wed, 14 May 2025 06:18:00 -0000 https://status.qcontact.com/incident/563681#37b90a1d02b87b417b1a0eec02872e3fe937f39ea0710ddafb2f8e7d88910e62 We are investigating connectivity from Azure ZA North Database Failover https://status.qcontact.com/incident/530637 Wed, 19 Mar 2025 07:31:00 -0000 https://status.qcontact.com/incident/530637#ce271f1d3a9629974d75df7025a9bb8e7bfd2217a55f8b18a2cc29f77a035022 This was caused by the automatic application of a critical security update. We have adjusted the automatic security update process to ensure failover fully completes before updates are applied. We apologise for the inconvenience caused. Database Failover https://status.qcontact.com/incident/530637 Wed, 19 Mar 2025 07:11:00 -0000 https://status.qcontact.com/incident/530637#8706814ad8be5b1719480904dcc57a90097c99e10145cdacbdcc26bd581358dc At 06:52:59 a database server failed. An automatic recovery was started and completed by 06:54:36 Telkom Mobile Inbound Calling https://status.qcontact.com/incident/523563 Thu, 06 Mar 2025 07:52:00 -0000 https://status.qcontact.com/incident/523563#7ca0a1f9fe091d3d4c3fdad58c3c98b64277e4cd89f59ebb6ba116155144039e This issue appears to have been resolved and calls are successfully connecting. Telkom Mobile Inbound Calling https://status.qcontact.com/incident/523563 Thu, 06 Mar 2025 05:10:00 -0000 https://status.qcontact.com/incident/523563#47f972fc89f0cca2794a6ae296b7965549a5a9fd4f107e3be7baae202096f550 We are aware of an issue where Telkom Mobile customers in South Africa are unable to dial certain inbound numbers provided by QContact. This issue is within Telkom's network, and a fault has been raised on why the interconnect is failing. Kenya Numbering https://status.qcontact.com/incident/510638 Tue, 11 Feb 2025 05:00:32 +0000 https://status.qcontact.com/incident/510638#c82c4afbecbfa1793fdc48e3eb22e2597d2866d4d3a42d626dd0763f156cbb0f Maintenance completed Kenya Numbering https://status.qcontact.com/incident/510638 Tue, 11 Feb 2025 02:30:32 -0000 https://status.qcontact.com/incident/510638#e246b5a4ca96f59cb380daeeb91635267d6cad61857ba78341bb340fde7d1bc5 Please be aware of a maintenance window for Kenyan numbering by the upstream Carrier between 5:30am and 8:00am East Africa Time. There may be intermittent disruptions to service during this period. Angani Voice https://status.qcontact.com/incident/508310 Thu, 06 Feb 2025 11:49:00 -0000 https://status.qcontact.com/incident/508310#65837a9a1935eb871c618a021514d078418d4c2fc54bc2de5c031d030ad697ba Service is resolved to normal levels - we have not received details from Angani yet as to the cause. Angani Voice https://status.qcontact.com/incident/508310 Thu, 06 Feb 2025 11:23:00 -0000 https://status.qcontact.com/incident/508310#aadf44ae43ad57fbc2f2ff54a5814d7eca9dd6bce312655be58256a219d067b2 We are aware of a disruption at Angani in Kenya. Awaiting feedback WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 17:17:00 -0000 https://status.qcontact.com/incident/507516#d602f32bf99330e84437cfcb2fca30980bc6d572588813892a0c3a836a9c055d We await an RCA from Meta, but have been informed this was due to an unannounced code change at Meta which changed the way webhooks (used for incoming messages) were sent. This prevented 360dialog from receiving the messages in the correct format, and forwarding them to the correct recipient. 360Dialog setup a workaround for accounts which only had a single number on the WABA while awaiting Meta to reverse the change. The clients who only had a single number on their WABA recovered at around 9:30am UTC. Those who had multiple numbers started to recover around 2 hours later at 11:20 UTC. We have been informed that messages sent during the disruption *can not* be retrieved, as they were not stored by Meta or 360Dialog for data protection reasons. We understand this is very disappointing news to hear, and have communicated this back - but there is unfortunately no way to recover these. If we receive a formal RCA, we will publish it for our clients. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 14:33:00 -0000 https://status.qcontact.com/incident/507516#513d6c88978391d3758100685898dd462a93f35df538ca18e24d465e5fbff7fc 360Dialog have confirmed the issue is resolved, but we await further confirmation of more details. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 14:22:00 -0000 https://status.qcontact.com/incident/507516#e7ea6f24da08f807f5fc821eda65f6da10f0b06b260d88089d31546f6bfc6e83 Meta have advised the issue is resolved on their status page, but awaiting official comms from 360Dialog "We have recovered from an earlier outage impacting Cloud API, and services have now been restored. We apologize for any inconvenience that this may have caused." WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 11:20:00 -0000 https://status.qcontact.com/incident/507516#97c8254947afec571514c7a5317859541bc15f2ed85feac8e52a4705b1ce6f42 We have started to receive messages from accounts which have multiple numbers under the same WABA. No official comms from Meta or 360Dialog at this time. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 09:41:00 -0000 https://status.qcontact.com/incident/507516#e3f7c2b513fc9667fd5c7e7a865cb5d104800cb651c7d82d9b2ec235bbbb5e4e We have started receiving a small amount of incoming messages. We understand only users who have a single number under the WABA will be receiving messages at this time. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 08:28:00 -0000 https://status.qcontact.com/incident/507516#690f9251c13ea345740cf2fd6be2fccdcd20f10969583796f9353d44c2d01861 Update from Meta https://metastatus.com/whatsapp-business-api Some disruptions: Cloud API Feb 5 2025 9:11 AM GMT+1 Our engineering teams have identified the root cause of the medium disruptions impacting webhooks on WhatsApp Cloud API. We are currently deploying the fix and the expected deployment time is in 8hours. We will provide another update within 4 hours or sooner if additional information is available. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 07:29:00 -0000 https://status.qcontact.com/incident/507516#871bcfcd84899fe675f72a2f9d5187fe09c8acb4d2058b2deda9fe9a4784cae6 Latest update from 360Dialog: We are monitoring the situation closely and waiting on Meta. WhatsApp Disruption https://status.qcontact.com/incident/507516 Wed, 05 Feb 2025 00:10:00 -0000 https://status.qcontact.com/incident/507516#537050281eeeac42bfa33b04daff735e1883b169fb37ef1526f60230dfca96eb Tracking on 360Dialog website: https://status.360dialog.com/cm6r59kif000nhwyo48km38ut WhatsApp Disruption https://status.qcontact.com/incident/507516 Tue, 04 Feb 2025 22:30:00 -0000 https://status.qcontact.com/incident/507516#9e0d9d33f563b6d8334f87ee1147efcf5dc9d78d8269a74e1d6ac0d4bb511aff Please note 360Dialog is experiencing an issue with incoming WhatsApp deliveries. Outbound messages are still being delivered. Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 13:28:00 -0000 https://status.qcontact.com/incident/491924#df1623fb3d460e2673fd89e9f792c91af7ea1fdf3151cc696555ecfa86928bde Replacement redundancy is now also now back in place. Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 12:06:00 -0000 https://status.qcontact.com/incident/491924#abeb0695bc6ba60b0b78e087b2e1e14710cef4a7cae2152d7d9486e543096a39 Azure have confirmed the issue is fully resolved with the following details.. Between 10:34 UTC and 11:05 UTC on 07 January 2025, a platform issue resulted in an impact to the Virtual Machines service in the South Africa North region. Impacted customers experienced connection failures when trying to access some VMs hosted in the region. These VMs may have also restarted unexpectedly. What went wrong, and why? During this incident, a top-of-rack (ToR) network device, connecting a single rack of servers, experienced an unexpected reboot, consequently the hosts connected to this ToR experienced a loss in connectivity. Customers may have been unable to connect to their resources and Virtual Machines (VMs) that were deployed on the affected resources may have experienced a restart. Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 11:03:00 -0000 https://status.qcontact.com/incident/491924#7e5c71e2e09138c6d1dea598608556df3d3b85cb438f0b7b530a7def7ee0fb87 Due to the network disruption in Azure North, we have moved all affected and degraded workloads on affected services away from unhealthy nodes. We will keep workloads away from problem resources until we see full service restored. Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 10:43:00 -0000 https://status.qcontact.com/incident/491924#f76efb49ad9311ffc7f1870928117c72999e8abacd52fd4e878e098385ea925c Failover is complete - more updates to come Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 10:41:00 -0000 https://status.qcontact.com/incident/491924#e9383349de8924b2c45df9e2c3a3cd31e9a0d493e4da6d0b5d2ec86036df73b2 Failover has been initiated on affected systems which will alleviate this issue. Updates to follow Azure network disruption https://status.qcontact.com/incident/491924 Tue, 07 Jan 2025 10:30:00 -0000 https://status.qcontact.com/incident/491924#f538cf263a569648eae68b955aad8f0e49f3e8c5577c4b28e9c2c3cdc2394c98 We are aware of an issue affecting Azure clients. This is due to a national Azure outage affecting connectivity to our systems. Please see below Azure incident status: A network device has experienced a fault, resulting in network connectivity loss to downstream resources. The unhealthy network device is being isolated from the network and traffic is being rerouted to healthy infrastructure. Further updates can be viewed here: https://app.azure.com/h/9KP2-3_8/8ffce0 Meta Disruption https://status.qcontact.com/incident/476174 Thu, 12 Dec 2024 02:03:00 -0000 https://status.qcontact.com/incident/476174#4b60a05f142e73d7c429ecd5fa7833d96006a2c6b67baa9c725b32d7947bc6f0 Meta have confirmed the update is resolved From - https://metastatus.com/whatsapp-business-api "We have recovered from an earlier outage impacting Cloud API, and services have now been restored. We apologize for any inconvenience that this may have caused." Meta Disruption https://status.qcontact.com/incident/476174 Wed, 11 Dec 2024 18:34:00 -0000 https://status.qcontact.com/incident/476174#9fb15dcd1cfc5e5792b98124009732fa9cbb6488e7d7f1d8a8c9b01d26ed9b1b Meta is currently experiencing disruption with their WhatsApp Business API, Instagram and Facebook Messenger APIs - for more information see https://metastatus.com/whatsapp-business-api We will automatically retry and failed deliveries. At present this is having limited to no impact on our customers, but we continue to monitor Postmark API https://status.qcontact.com/incident/476057 Wed, 11 Dec 2024 02:40:00 -0000 https://status.qcontact.com/incident/476057#b307baeb892d2cf53bcda50f4c1a26b7b2f1ac4b1cd3704f17403e05a61f3f89 The API has returned to normal. Any messages would have been queued our side and retried during the disruption. Postmark API https://status.qcontact.com/incident/476057 Wed, 11 Dec 2024 02:19:00 -0000 https://status.qcontact.com/incident/476057#560fc488f96af19ecc7f8d2befc8b4c559f23c4096256a0f9360693e5ecebd7b Postmark is experiencing API issues again. We are monitoring E-mail API https://status.qcontact.com/incident/472512 Wed, 04 Dec 2024 22:28:00 -0000 https://status.qcontact.com/incident/472512#67ee09bc4082e5baf693532fb86287db8b8383aef555542b4a02c1195b6e01fa Update from Postmark: We've deployed additional fixes and latency/errors have returned to normal performance. Engineering will continue to monitor into the evening for consistent results. E-mail API https://status.qcontact.com/incident/472512 Wed, 04 Dec 2024 21:56:00 -0000 https://status.qcontact.com/incident/472512#be00c7bb7ed81717d8938c71404299fc458f5446628bf1bee3a4a5a4f2d1a07a Update from Postmark: We've confirmed the source of the issue and are implementing fixes. Latency and errors has been reduced but engineers are continuing to remedy the issue. E-mail API https://status.qcontact.com/incident/472512 Wed, 04 Dec 2024 20:13:00 -0000 https://status.qcontact.com/incident/472512#eca48273853cedac2ffceb9a941a094f9e271044f0bbeb84b84d62ee8c7855d0 Postmark has notified us of some latency issues on their API again. Message delivery may be delayed, but no messages are being lost. Delays on outgoing e-mail https://status.qcontact.com/incident/465566 Fri, 22 Nov 2024 03:19:00 -0000 https://status.qcontact.com/incident/465566#e2f04d1d23f578d40c3044ea56ec6509865dbfdee77c796dd7c2fa1e39aece6b Postmark have confirmed issues with their API which they now believe it to be mitigated. You can find more information at https://status.postmarkapp.com/notices/iejilmig3r9ldil5-increased-api-latency-timeouts Delays on outgoing e-mail https://status.qcontact.com/incident/465566 Fri, 22 Nov 2024 02:18:00 -0000 https://status.qcontact.com/incident/465566#24801af1e8570cd1cec4daab53343d17ac13a72b4e10ef7fdfd2096320a12766 Update from Postmark: Additional fixes have been deployed to remedy unhealthy API instances and are now recovering to their expected performance. We will continue to monitor before resolving this status. Delays on outgoing e-mail https://status.qcontact.com/incident/465566 Thu, 21 Nov 2024 20:43:00 -0000 https://status.qcontact.com/incident/465566#844ef4bad9a33938e024c7dae136fd5e884d9b41f718df0a337b3dbc74b56ed8 Update from Postmark: We are still experiencing latency issues with some API instances affecting a small subset of customers. The engineering team is continuing to investigate and remedy the issue. Updates will be provided as more information is available. Delays on outgoing e-mail https://status.qcontact.com/incident/465566 Thu, 21 Nov 2024 17:19:00 -0000 https://status.qcontact.com/incident/465566#e714ac4bb6cb18f139af027d0c41b37b45c979388076f96b18b099d4057aa6c5 Update from Postmark: A fix was deployed reducing latency and timeouts for unhealthy API instances. We are continuing to monitor while it recovers. Delays on outgoing e-mail https://status.qcontact.com/incident/465566 Thu, 21 Nov 2024 17:06:00 -0000 https://status.qcontact.com/incident/465566#17c17ef5962691dea3a961fc8895225e324312ee96bd2ed06063393340ce101a For customers using the built-in e-mail functionality (those using Office 365 are unaffected), our e-mail provider is currently experiencing API instability. This may lead to delays in e-mails being sent, as we automatically retry to deliver your messages. We await further details from our e-mail provider (Postmark) SA Network Disruption https://status.qcontact.com/incident/456914 Wed, 06 Nov 2024 12:43:00 -0000 https://status.qcontact.com/incident/456914#9bae0582de51a8fc43330822af173a7c611208c75fc95cdca09c252a225ef4f2 The situation has continued to remain stable. During the disruption period, 1.04% of requests resulted in a timeout due to the network issues according to 3rd party monitoring. SA Network Disruption https://status.qcontact.com/incident/456914 Wed, 06 Nov 2024 10:54:00 -0000 https://status.qcontact.com/incident/456914#908e8f3c025e55cad704c56bca4bd9167e01bd91c17914e593b068dd3b0d7416 Between approximately 2am and 10:40am there have been network connectivity issues with certain ISPs within South Africa accessing OVH's network in Europe (across multiple datacentres). You can see the network effect for a sample ISP here - https://lon1-eri.smokeping.ovh.net/smokeping??&target=EMEA.AS10474& This appears to now have stabilised. However if you continue to experience problems, please contact support as we have the ability to re-route your traffic over a working international link. We encourage all users within South Africa still using OVH UK data locations, to request the migration of your data to Azure SA. MTN and Liquid Telecom Interconnect https://status.qcontact.com/incident/449163 Wed, 23 Oct 2024 07:11:00 -0000 https://status.qcontact.com/incident/449163#906164fbb1586a60b4a125a9f0913a6db82923b3bcbcac3add1599955ea21ff3 This has been confirmed as resolved. MTN and Liquid Telecom Interconnect https://status.qcontact.com/incident/449163 Wed, 23 Oct 2024 06:16:00 -0000 https://status.qcontact.com/incident/449163#f20de40a7d0d7534d72b7ade150e170f9b4b485f39c818e05d747b6cd8b93d10 Liquid confirm the issue has been raised with MTN as a priority issue, but no ETA on resolution at present MTN and Liquid Telecom Interconnect https://status.qcontact.com/incident/449163 Tue, 22 Oct 2024 15:14:00 -0000 https://status.qcontact.com/incident/449163#652432a0fcb360572879900ef3b80e0ad4e7822188f5df1b0e1d5ca01cf9ba06 We are aware of an interconnection issue between Liquid Telecom and MTN. This is disrupting outbound and inbound calls using 060 numbers and MTN. This is due to a failure of an SS7 link between the carriers. We have no ETA yet on resolution. Please note, this is a carrier issue rather than something within QContact's control. This affects all traffic between Liquid Telecom and MTN. Azure & OVH Interconnect https://status.qcontact.com/incident/446626 Thu, 17 Oct 2024 16:00:00 -0000 https://status.qcontact.com/incident/446626#b451fe542989517211eaa16679ba07b734b6398ae9979b6dff35367446f51739 Between 13:14 UTC and 13:19 UTC there was a brief network connectivity issue between Azure LHR, Azure JNB and OVH LON where connections where stalled or timed out. For any customer whose voice is hosted in one of those Azure regions, but data is held in OVH London - this may have resulted in disruption to recordings. The issue was automatically resolved when the network connectivity resumed. This issue did not affect any other region than the ones listed. Azure & OVH Interconnect https://status.qcontact.com/incident/446626 Thu, 17 Oct 2024 16:00:00 -0000 https://status.qcontact.com/incident/446626#b451fe542989517211eaa16679ba07b734b6398ae9979b6dff35367446f51739 Between 13:14 UTC and 13:19 UTC there was a brief network connectivity issue between Azure LHR, Azure JNB and OVH LON where connections where stalled or timed out. For any customer whose voice is hosted in one of those Azure regions, but data is held in OVH London - this may have resulted in disruption to recordings. The issue was automatically resolved when the network connectivity resumed. This issue did not affect any other region than the ones listed. Azure Voice https://status.qcontact.com/incident/442970 Tue, 15 Oct 2024 09:50:00 -0000 https://status.qcontact.com/incident/442970#313ce8336a1447826f452f844aa981b60d4d36efc2bea0f5a68c8e8369a06a5d This issue reoccured briefly between 11:41 and 11:49. Circuit breaker limits further increased as well as now being fully automated. Azure Voice https://status.qcontact.com/incident/442970 Tue, 15 Oct 2024 09:50:00 -0000 https://status.qcontact.com/incident/442970#313ce8336a1447826f452f844aa981b60d4d36efc2bea0f5a68c8e8369a06a5d This issue reoccured briefly between 11:41 and 11:49. Circuit breaker limits further increased as well as now being fully automated. Azure Voice https://status.qcontact.com/incident/442970 Tue, 15 Oct 2024 09:50:00 -0000 https://status.qcontact.com/incident/442970#313ce8336a1447826f452f844aa981b60d4d36efc2bea0f5a68c8e8369a06a5d This issue reoccured briefly between 11:41 and 11:49. Circuit breaker limits further increased as well as now being fully automated. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:44:00 -0000 https://status.qcontact.com/incident/442970#de2c0befeb154ca58d4470f18ed8c16e18e8fced447b5e7791b1e5bd033f2a91 This issue has been resolved. Connections to the platform were being throttled - this lead to slower refreshes and answering of calls. The throttle limits have been increased. The telemetry which monitors throttling wasn't correctly configured, meaning the limits were hit before an adjustment could be made. The limits have been increased to ensure no legitimate usage results in any further throttling. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:44:00 -0000 https://status.qcontact.com/incident/442970#de2c0befeb154ca58d4470f18ed8c16e18e8fced447b5e7791b1e5bd033f2a91 This issue has been resolved. Connections to the platform were being throttled - this lead to slower refreshes and answering of calls. The throttle limits have been increased. The telemetry which monitors throttling wasn't correctly configured, meaning the limits were hit before an adjustment could be made. The limits have been increased to ensure no legitimate usage results in any further throttling. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:44:00 -0000 https://status.qcontact.com/incident/442970#de2c0befeb154ca58d4470f18ed8c16e18e8fced447b5e7791b1e5bd033f2a91 This issue has been resolved. Connections to the platform were being throttled - this lead to slower refreshes and answering of calls. The throttle limits have been increased. The telemetry which monitors throttling wasn't correctly configured, meaning the limits were hit before an adjustment could be made. The limits have been increased to ensure no legitimate usage results in any further throttling. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:21:00 -0000 https://status.qcontact.com/incident/442970#816d5e4d483a0d28f6ce9bd12670e4a25f87bad75bab1028dcbd7e391f70fd12 Reports of delays in being able to answer calls or successfully registering (phone spinning) on refresh in Azure regions. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:21:00 -0000 https://status.qcontact.com/incident/442970#816d5e4d483a0d28f6ce9bd12670e4a25f87bad75bab1028dcbd7e391f70fd12 Reports of delays in being able to answer calls or successfully registering (phone spinning) on refresh in Azure regions. Azure Voice https://status.qcontact.com/incident/442970 Fri, 11 Oct 2024 12:21:00 -0000 https://status.qcontact.com/incident/442970#816d5e4d483a0d28f6ce9bd12670e4a25f87bad75bab1028dcbd7e391f70fd12 Reports of delays in being able to answer calls or successfully registering (phone spinning) on refresh in Azure regions. Google API access https://status.qcontact.com/incident/412693 Mon, 12 Aug 2024 13:26:00 -0000 https://status.qcontact.com/incident/412693#ddc567ebd6d0eb60093ad4d334279e9037bb87bc97136f9388e8362f15cadc51 We see network connectivity is restored. Google API access https://status.qcontact.com/incident/412693 Mon, 12 Aug 2024 13:21:00 -0000 https://status.qcontact.com/incident/412693#7ca30d9849c3d87f4d03d1a9e1ee7133f670576bb93be5b74542399ed951153b We are aware of a network connectivity issue into Google's network affecting European user's ability to use DNS lookups, GCS cloud storage and access to Dialogflow APIs Issues dialling South African mobile numbers https://status.qcontact.com/incident/411777 Sat, 10 Aug 2024 07:47:00 -0000 https://status.qcontact.com/incident/411777#cefccbcce5ccb7c71b679154cf36acb4d25f5bd2450de7692d3cc81bac41b868 Calls have been re-routed via an alternative route while we await details from the upstream carrier. Issues dialling South African mobile numbers https://status.qcontact.com/incident/411777 Sat, 10 Aug 2024 07:36:00 -0000 https://status.qcontact.com/incident/411777#d07248691d330791439a65418aaf52745aea651f6c92403ccacffb599366adf0 We are investigating issues dialling +278xxx numbers SSO https://status.qcontact.com/incident/376254 Wed, 29 May 2024 08:13:00 -0000 https://status.qcontact.com/incident/376254#9e46d74cfd8f50f0ff424160f8c125dcea3bf17077cd27e0b1982fc40c63fa65 The update has been fully deployed in all regions. SSO https://status.qcontact.com/incident/376254 Wed, 29 May 2024 07:57:00 -0000 https://status.qcontact.com/incident/376254#aedeb690bc9f95870e7497168d82ad9a24e4b85bb93bdc1d3783dba3d27c76bd We are deploying an update to resolve this issue. SSO https://status.qcontact.com/incident/376254 Wed, 29 May 2024 07:27:00 -0000 https://status.qcontact.com/incident/376254#feed4632b001f4de4475247df81944b3f04049faa0aaa818b406a431de6f1a85 We are investigating an issue with Microsoft Entra ID SSO Meta Service Issue https://status.qcontact.com/incident/350448 Wed, 03 Apr 2024 20:45:00 -0000 https://status.qcontact.com/incident/350448#0f642cb2b0cfacb5cc49fd9f8fd011349d209dd9504603fc93ca6aac6e2a46b0 Meta have confirmed the issue is resolved. The fix for the outage impacting service on Cloud API has been deployed and this issue is now mitigated as of 04-03-2024 at 1:45 PM PST. Thank you for your patience. Meta Service Issue https://status.qcontact.com/incident/350448 Wed, 03 Apr 2024 18:10:00 -0000 https://status.qcontact.com/incident/350448#5ea44db4c006f48df08f7c083c777102cf4b6c8a8cda6395c64e3244c8576954 There is currently disruption with Meta services globally. This is affecting message delivery on Facebook, Instagram and WhatsApp. Waiting on an update from Meta who will post at https://metastatus.com/whatsapp-business-api Azure South Africa Region https://status.qcontact.com/incident/340330 Mon, 18 Mar 2024 04:30:00 -0000 https://status.qcontact.com/incident/340330#11c6768d113a8f986e6a35e29de3a6c67e524e8d48507f423fb61e5daf981ad0 This incident has officially been closed by Azure now additional bandwidth is online the services are no longer at risk. If you require more information about this incident, please feel free to reach out to your account manager for more details. Azure South Africa Region https://status.qcontact.com/incident/340330 Sat, 16 Mar 2024 17:13:00 -0000 https://status.qcontact.com/incident/340330#283c72f48e2b47efe91866e9c4522d33cdb191423771c92514e2fe0e9df9edba Latest Update from Azure With the current bandwidth demand and network utilization, all services are currently healthy with no issues detected. We are still detecting some minor latency for small transient periods. In preparation for demand increases, we are on track to take all available measures to increase capacity. Substantial additional capacity remains on track to be provisioned by March 19, 2024. We continue to closely monitor all telemetry to ensure network stability leading into the South Africa region's business hours on Monday, March 18, 2024. The majority of Azure Services affected by this issue have returned to normal functionality. Customers should see recovery and minimal impact at this time. -- We continue to see a good level of service across all carriers other than Seacom. We also see significant improvements on zScaler CPT IV since around 09:30am local time in SA, which was the only monitored zScaler region in SA still affected. Azure South Africa Region https://status.qcontact.com/incident/340330 Sat, 16 Mar 2024 00:18:00 -0000 https://status.qcontact.com/incident/340330#80f5a01b89fc2f47bb40cc40dcce4545f91e0718cd76d26fac335a775c2848ed Update from Azure is that international connectivity is restricted at present to 400Gbps, and they continue to experience issues with stability on additional capacity at this time. An additional 1Tbps of capacity is due online by Tuesday. We continue to see normal levels of service all on carriers other that Seacom. Azure South Africa Region https://status.qcontact.com/incident/340330 Fri, 15 Mar 2024 20:23:00 -0000 https://status.qcontact.com/incident/340330#47e44c7f7266aff6c5a9b7e2767d091f652f952f43aaed7bc0cbe45b09159080 As per our last update - we continue to see good service across Africa with the exception of Seacom which is still routing domestic traffic via Europe and USA. Latest Update from Azure: We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa. In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions. We've added the additional network capacity, however, there has been a slight amount of instability in the newly added links. We're working with the partner carrier to investigate the issues. Additionally, we're planning to add additional capacity within the region over the weekend and into next week to bring more headroom into the region. Once our additional capacity is stable, we plan to re-enable Azure Front Door (AFD) routing within the South Africa region to improve some application performance. As we continue to work on further increasing the capacity, our telemetry shows signs of recovery, and packet drops have reduced significantly. Many of the Azure Services that were affected by this issue earlier have now returned to normal functionality. We are validating for any remaining impacted services, however customers should notice considerable improvements. Azure South Africa Region https://status.qcontact.com/incident/340330 Fri, 15 Mar 2024 14:37:00 -0000 https://status.qcontact.com/incident/340330#33fa5e16ca6abe78332596967299c7b9f9b21aabed651250a62580c375545efa Latest from Azure is they are experiencing difficulties still bringing the new capacity online and updated ETA is now 18:00 UTC. Connectivity across the board in South Africa remains good with the exception of Seacom. Azure South Africa Region https://status.qcontact.com/incident/340330 Fri, 15 Mar 2024 07:12:00 -0000 https://status.qcontact.com/incident/340330#5ebdc5d6ad77411bd2044ff42f20cfd7ce38687f5be4dae67ee4d0d821223d60 We have seen normal levels of latency return on all Vodacom links. We are seeing issues remain with Seacom where capacity remains constrained. All other mainstream carriers and ISPs continue to show normal performance with low latency and zero packet loss. Updates to follow. Azure South Africa Region https://status.qcontact.com/incident/340330 Fri, 15 Mar 2024 04:44:00 -0000 https://status.qcontact.com/incident/340330#f5106a8bfd5bc6a728b559fc7d4c1a93872cb8ebaf1b646b62bdfe322c7bd57e Azure now estimate this work to have completed by 14:00 UTC - we continue to monitor. Updates to follow. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 22:54:00 -0000 https://status.qcontact.com/incident/340330#ff43de3d03459796b35a0a4391db22bbc1a7aedf44fad6dd89b48955b0fde6af We continue to see the issue limited to Vodacom and Seacom at this time. Azure have also updated the ETA back to 02:00 UTC for the additional capacity. Update from Azure Current Status: We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa. In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions. The overall network health for the region has been steadily improving. Many of the Azure Services that were affected earlier by this issue have now returned to their normal levels. We are still validating for any remaining services, however the users for these would already notice considerable improvements. We are working on adding more capacity and we anticipate that these steps will help further with ongoing capacity bottleneck. The new estimated time of completion for this activity is 2:00 UTC on 15 March 2024 Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 19:58:00 -0000 https://status.qcontact.com/incident/340330#edd19991f7c45bfc11c68be88cf7e174955c39875056c39a2f2e7632f63cee12 We continue to see elevated latency to Seacom and Vodacom. While we see no other issues at present, should issues persist on Friday, for South African clients we have provisioned resources within Amazon Web Services in Cape Town, and are able to move voice service there on a temporary basis. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 18:17:00 -0000 https://status.qcontact.com/incident/340330#c78f7a4e4cd39344f21167579b9d18dce5e60ba222106d17db2653c367068280 Update from Azure We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa. In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting overall capacity on the East Coast of Africa. These incidents together had reduced the total network capacity for most of Africa's regions. Our efforts through simultaneous workstreams to reduce non-critical traffic has helped to normalize the network bandwidth. We believe most of the customers should no longer be seeing impact and we are continuing to validate the health metrics across the Azure Services which were impacted initially. Our teams continue to work on bringing additional capacity to minimize any impact across services. We expect that this work should take about 4 hrs from now, which would be around 22:00 UTC on 14 March 2024. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 17:18:00 -0000 https://status.qcontact.com/incident/340330#d160d7edd6e4b728a005f57a14f542ba39ceeb9758639544ccf943c28c6f7dae International latency has now recovered to usual levels. Microsoft themselves continue to re-route local 365 resources to both Europe and South Africa West. We continue to see higher latency to Vodacom and Seacom, but packet loss is now zero on these links too. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 15:42:00 -0000 https://status.qcontact.com/incident/340330#bc0dda5149a18547307f5b1b22e95ae707df3fdebc64b038472971326dd3fe08 We have seen significant improvements on international connectivity in the past 10 minutes. While latency is still around 35% higher than usual, we are currently seeing zero packet loss (down from up to 80% packet loss on international links earlier in the incident) Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 14:40:00 -0000 https://status.qcontact.com/incident/340330#6e3b86520e7dbe63273d5861a4d13748d79b9742f54a2f7626c74b29b5ce8841 We have seen service largely restored domestically. We continue to see issues remain with ISPs that use Seacom or Vodacom for upstream connectivity or SIP trunks. We are also still seeing high international connectivity latency and packet loss, but significant improvement on earlier. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 14:26:00 -0000 https://status.qcontact.com/incident/340330#74d08c05d56daef712296c9b83691329c895cf564f49b91e6a21f6a5124d9244 Azure Update Current Status: We have determined that multiple fiber cables on the West Coast of Africa -- WACS, MainOne, SAT3, ACE – have been impacted which reduced total capacity supporting our Regions in South Africa. In addition to these cable impacts, the on-going cable cuts in the Red Sea -- EIG, Seacom, AAE-1 -- are also impacting capacity on the East Coast of Africa. This combination of incidents has impacted all Africa capacity – including other Cloud providers and public Internet as well. Current Workstreams: We are exploring the following mitigation workstreams Working with our local network partner to fix defective links. Exploring traffic redirection and optimization on our network We estimate that these workstreams will take up to 12 hours to complete. Customers who are experiencing severe disruption to their operations may want to consider their BCDR options. The next update will be provided in 4 hours or as events warrant. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 13:22:00 -0000 https://status.qcontact.com/incident/340330#bcba436cebc7984202cf047611b8f40722e926198a53b722de51214142c53995 Update from Azure Current Status: We have determined that multiple fiber cables on the West Coast of Africa — WACS, MainOne, SAT3, ACE — have been impacted which reduced total capacity supporting our Regions in South Africa. In addition to these cable impacts, the on-going cable cuts in the Red Sea — EIG, Seacom, AAE-1 — are also impacting capacity on the East Coast of Africa. This combination of incidents has impacted all Africa capacity – including other Cloud providers and public Internet as well. Current Workstreams: We are exploring the following mitigation workstreams Working with our local network partner to fix defective links. Exploring traffic redirection and optimization on our network The next update will be provided in 60 minutes, or as events warrant Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 12:11:00 -0000 https://status.qcontact.com/incident/340330#de191628e889bdd1e93fef5969832b3558597a5149937a48ca8a9c6645ac628c This issue continues to affected certain customers accessing resources in Azure South Africa. The issue is affecting access to both Azure and Microsoft 365 across South Africa, as well as other service providers. Update from Azure We are currently investigating a networking issue that has resulted in higher-than-expected latency for users whose resources are hosted in or connect with South Africa North and West. The next update will be provided in 60 minutes, or as events warrant. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 12:00:00 -0000 https://status.qcontact.com/incident/340330#a88855da8eac302da33c7c26784e73508c4a5d6bc7725402afa202f42e45d750 Azure have notified that they are rerouting connections to alternate components to mitigate impact. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 11:33:00 -0000 https://status.qcontact.com/incident/340330#c4b9c8906b37ab78d59015ba0e3f23ccd603f8a2d78d01d24606efccade5a65a Update from Azure Starting at 10:30 UTC on 14 Mar 2024, Azure Services in South Africa North and South Africa West may experience service degradation and availability issues. Additionally, customers may have observed higher than expected latency and packet drops for resources hosted in these regions. Current Status: We are aware of this issue and are actively investigating. The next update will be provided in 60 minutes, or as events warrant. Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 11:21:00 -0000 https://status.qcontact.com/incident/340330#73aea0ea632396e71ea6a885af6c23b34bcc41a940c84d9ac12b694fef2f4456 Azure have assigned the reoccurrence issue VT60-RPZ https://app.azure.com/h/VT60-RPZ/8ffce0 Azure South Africa Region https://status.qcontact.com/incident/340330 Thu, 14 Mar 2024 10:52:00 -0000 https://status.qcontact.com/incident/340330#646f6e547361007cfb59b4726b566f402990a7016859a712a4545028d43c72a3 This issue has reoccured - monitoring Azure South Africa Region https://status.qcontact.com/incident/340330 Wed, 13 Mar 2024 18:00:00 -0000 https://status.qcontact.com/incident/340330#6f0e0dd75913b219b68a4aaa2d78206a3fa359965206f8eae6700796cafba092 Azure have confirmed the fibre cut was resolved, and have closed the issue Azure South Africa Region https://status.qcontact.com/incident/340330 Wed, 13 Mar 2024 17:23:00 -0000 https://status.qcontact.com/incident/340330#f76f459052ff43ffae8a438f4b41f50e1bb5634896caff5a76d222318fb1e916 Update from Azure: In addition to addressing a fiber cut and assessing how quickly we can expedite its repair, our efforts to restore optimum network performance have included reducing the reservation on certain links, leading to a notable decrease in packet drops and an overall improvement in services. While we have optimized and load balanced our network to utilize all available capacity, we are still assessing impact with some of our routing on segments whether or not we can take on a peak load. Azure Front Door is planning to redirect traffic away from South Africa to another region, provided that no further packet drops occur. The next update will be provided in 2 hours, or as events warrant. Azure South Africa Region https://status.qcontact.com/incident/340330 Wed, 13 Mar 2024 14:56:00 -0000 https://status.qcontact.com/incident/340330#212861a4db4712262dd7580d2e2ded2fadea76612ed889a2e82b51b0a656c0db Update from Azure We determined that this issue was caused by a fiber cut, resulting in congestion specifically affecting the traffic in the Africa region. This has impacted the connectivity between South Africa and Europe, causing disruptions to your services. Our teams are actively working to address these issues and restore normal operations. The next update will be provided in 60 minutes, or as events warrant. Azure South Africa Region https://status.qcontact.com/incident/340330 Wed, 13 Mar 2024 14:30:00 -0000 https://status.qcontact.com/incident/340330#635d5f41a1c004842ea00e8a81f43b871549089a29a8b68b7dde02f6315a10f6 Azure have notified us of network issues within South Africa North and South Africa West. We are monitoring https://app.azure.com/h/_NF5-7PG/8ffce0 Meta Disruptions https://status.qcontact.com/incident/336623 Tue, 05 Mar 2024 21:00:00 -0000 https://status.qcontact.com/incident/336623#3c1f5b6f3988413a454b34d4ebac2561c50ea1930a30737bedb08c5be1ab2f0e Meta have confirmed all services are restored to normal Meta Disruptions https://status.qcontact.com/incident/336623 Tue, 05 Mar 2024 16:48:00 -0000 https://status.qcontact.com/incident/336623#d153b4d3a1ce64daf38691d16759180f815668fc57cd9368d066e139bce88a82 Meta have updated the incident to say they are recovering. Meta Disruptions https://status.qcontact.com/incident/336623 Tue, 05 Mar 2024 16:00:00 -0000 https://status.qcontact.com/incident/336623#93563ca108556fdbff7fa86005b3cd302b9d79788c730d71e9badedafd0c306b Meta is experiencing a large service disruption which may be affecting WhatsApp, Facebook and Instagram services at this time More details are posted at https://metastatus.com/ UK Outbound Calling https://status.qcontact.com/incident/322576 Mon, 05 Feb 2024 15:43:00 -0000 https://status.qcontact.com/incident/322576#3893347051310b458bed633d3c8250edec4547c6de1a2e542a2a6c10d239b732 Carrier has confirmed the issue was in a DNS change made. Awaiting a detailed RFO. UK Outbound Calling https://status.qcontact.com/incident/322576 Mon, 05 Feb 2024 15:34:00 -0000 https://status.qcontact.com/incident/322576#6ac4ed8331df3ad8e1c4cf16187b562a7d50b1e225a53e6e9f282b959c0f50a6 We are seeing service return to normal - awaiting feedback from carrier. UK Outbound Calling https://status.qcontact.com/incident/322576 Mon, 05 Feb 2024 15:24:00 -0000 https://status.qcontact.com/incident/322576#8872ba7a9311c15e8d936625c8ef9272951a30fbebd9dd5b34652116dd89cdc6 This has been escalated internally with the upstream carrier to the relevant technicians. UK Outbound Calling https://status.qcontact.com/incident/322576 Mon, 05 Feb 2024 15:07:00 -0000 https://status.qcontact.com/incident/322576#7e248d4cef386d44f930d06785b59bc6a21364349356d0194d7a52684f7523ea We are aware of an issue dialling out to UK numbers. This has been flagged with the upstream carrier. WhatsApp Cloud Numbers https://status.qcontact.com/incident/321251 Fri, 02 Feb 2024 08:28:00 -0000 https://status.qcontact.com/incident/321251#c834618217badb55135d817747c2301230ff8be81c226a342928d7f4f7b630a5 360Dialog have confirmed resolution - we continue to monitor WhatsApp Cloud Numbers https://status.qcontact.com/incident/321251 Fri, 02 Feb 2024 08:17:00 -0000 https://status.qcontact.com/incident/321251#911a89614640fc8a93ca14fd6332af2a0f5b96ec6b9445eeb76b4a43dd190d6c We still await feedback from 360Dialog WhatsApp Cloud Numbers https://status.qcontact.com/incident/321251 Fri, 02 Feb 2024 07:56:00 -0000 https://status.qcontact.com/incident/321251#3dea9b9db9dbd8fe68b74701932f77e0a614cc3b675ac9d538e951a9dc766957 360Dialog has confirmed they're aware of the issue and investigating. WhatsApp Cloud Numbers https://status.qcontact.com/incident/321251 Fri, 02 Feb 2024 07:54:00 -0000 https://status.qcontact.com/incident/321251#965ac1387e58d29c3bd4ce95ff254c171f905a8a4fcc2edb464daf7d17cbbf9b We are aware of an issue affecting all WhatsApp Cloud Numbers. We are awaiting feedback from 360Dialog. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 +0000 https://status.qcontact.com/incident/308327#8f716663dd24eea67b97f02109b82be20e1f9613dcdfa62deb56a82e93167a1a Maintenance completed Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 +0000 https://status.qcontact.com/incident/308327#8f716663dd24eea67b97f02109b82be20e1f9613dcdfa62deb56a82e93167a1a Maintenance completed Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 +0000 https://status.qcontact.com/incident/308327#8f716663dd24eea67b97f02109b82be20e1f9613dcdfa62deb56a82e93167a1a Maintenance completed Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:29:35 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:00:00 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:00:00 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Database Maintainence https://status.qcontact.com/incident/308327 Fri, 05 Jan 2024 23:00:00 -0000 https://status.qcontact.com/incident/308327#14fd93250a14e64f8ef7f93bb20a8d2495bcc3d37a62ee5c1ca6e6ea099acff8 During this period, we must deploy an update to one of our database clusters. This will take approximately 60 seconds to complete, but consider this whole window at risk. Outbound UK Calling https://status.qcontact.com/incident/300582 Thu, 14 Dec 2023 14:40:00 -0000 https://status.qcontact.com/incident/300582#7f38398ca52b48e2599727f7c8283723d1807e86c72c96d95dde2afa46fad1dd This has been resolved by the upstream telephone carrier. Outbound UK Calling https://status.qcontact.com/incident/300582 Thu, 14 Dec 2023 14:18:00 -0000 https://status.qcontact.com/incident/300582#02adb0aad06f2f48571956236f7242d1674ed6ff851a2cdbdc210aa514c79777 We are aware of issues outbound calling at present to UK destinations. This is an upstream carrier issue, and has been raised with them. WhatsApp Distruption https://status.qcontact.com/incident/241282 Fri, 04 Aug 2023 07:59:00 -0000 https://status.qcontact.com/incident/241282#d7ec62272ab49574f76b51d1d9ee49f1e1ad96e12dad421800f330ae87478b93 Update from Meta: The fix for the major disruption impacting sending messages on WhatsApp Cloud API has been deployed and this issue is now mitigated as of 3rd Aug 2023 11:50 PM PDT. Thank you for your patience. WhatsApp Distruption https://status.qcontact.com/incident/241282 Fri, 04 Aug 2023 07:09:00 -0000 https://status.qcontact.com/incident/241282#5f898581464c116046c4b932c248e3f6198ef6e51e9a18a471b546ed738f9558 We continue to await updates from Meta - however to confirm however the majority of clients numbers are unaffected by this issue. WhatsApp Distruption https://status.qcontact.com/incident/241282 Fri, 04 Aug 2023 04:42:00 -0000 https://status.qcontact.com/incident/241282#6c1461669288ee70e14648d842a2d9de824ae9c705600115f73c8209f09b99ff Update from meta: Our engineering teams have identified the root cause of the major disruption impacting sending messages on WhatsApp Cloud API. We are currently deploying the fix and the expected deployment time is in 6 hours. We will provide another update within 4 hours or sooner if additional information is available. WhatsApp Distruption https://status.qcontact.com/incident/241282 Fri, 04 Aug 2023 03:38:00 -0000 https://status.qcontact.com/incident/241282#a8e95b443f8257f0e2e321c4b086973ec7051a7f03d1e14619a63e19877d728f Meta are currently experiencing a disruption to their WhatsApp Business Cloud API - for more details you can check their status updates at https://metastatus.com/whatsapp-business-api WhatsApp issues https://status.qcontact.com/incident/195075 Tue, 11 Apr 2023 12:59:00 -0000 https://status.qcontact.com/incident/195075#ba36a3c8754a436b72503218f79aefe372ffb22e2f90377043d64ef24efda9ed We continue to see stability and will mark this incident as resolved WhatsApp issues https://status.qcontact.com/incident/195075 Tue, 11 Apr 2023 12:38:00 -0000 https://status.qcontact.com/incident/195075#9698217afc710814842ec6b9f417fd2f8d001c2bbac36deb4bf529e86d2eab74 360Dialog have confirmed resolution - we will continue to monitor WhatsApp issues https://status.qcontact.com/incident/195075 Tue, 11 Apr 2023 12:32:00 -0000 https://status.qcontact.com/incident/195075#64357b8d8159b6ec61cbc841fa83b428934112f452c427904be789443f95f5b4 We are seeing inbound deliveries starting and successful outgoing messages, however we have no official update from 360dialog as yet to the problem being officially resolved. WhatsApp issues https://status.qcontact.com/incident/195075 Tue, 11 Apr 2023 11:05:00 -0000 https://status.qcontact.com/incident/195075#13e88bc6991016e704aaedad8f6e3de91ab1d2d86d00400f62cc9e8296cc78f3 For more information you can follow 360Dialog's status updates at https://status.360dialog.com/clgc5mtku4306787nhmeokwk4d WhatsApp issues https://status.qcontact.com/incident/195075 Tue, 11 Apr 2023 11:04:00 -0000 https://status.qcontact.com/incident/195075#e0c5524b8e69d97501d8939648dd21980a5584f866a48444f186dea394a8dc7a 360Dialog currently has a major service interruption affecting the sending and receiving of WhatsApp messages. Internal Transfers https://status.qcontact.com/incident/193455 Thu, 06 Apr 2023 07:02:00 -0000 https://status.qcontact.com/incident/193455#a622b12d748c60f75b8d64915e5bda8d27c5fa49466f8a1e1dd3716c9b634676 The fix has been deployed. Internal Transfers https://status.qcontact.com/incident/193455 Thu, 06 Apr 2023 06:50:00 -0000 https://status.qcontact.com/incident/193455#d72cb1cd269672ecd11b48bf0b0aeb72b7f4eb7907c5f1fa34c044fd7d6315a7 The cause has been identified and a fix is being deployed Internal Transfers https://status.qcontact.com/incident/193455 Thu, 06 Apr 2023 06:30:00 -0000 https://status.qcontact.com/incident/193455#815cbca0a17596d7f100ed581fb86bdb6cf0a418fe9c6774f131ee47a9456489 We are investigating an issue where South African users are unable to manual dial other internal users. Delayed WhatsApp https://status.qcontact.com/incident/190004 Tue, 28 Mar 2023 08:41:00 -0000 https://status.qcontact.com/incident/190004#0b7edee102e16f4929c46cae904d677fb908f4df58cc8511ae799fb6e6608e19 We continue to monitor, but issue appears to have been resolved and messages are being processed at usual speed again. Delayed WhatsApp https://status.qcontact.com/incident/190004 Tue, 28 Mar 2023 08:31:00 -0000 https://status.qcontact.com/incident/190004#1d25558de9c3db168f135a4e34a463d93fb6805ac389065723e37a0be1e44f72 360Dialog have confirmed issue should be resolved. We are monitoring Delayed WhatsApp https://status.qcontact.com/incident/190004 Tue, 28 Mar 2023 08:23:00 -0000 https://status.qcontact.com/incident/190004#ffbfb19bf6891f57bf713c548b0e71b4506336b24faf2b08e3e6056bb0624887 360Dialog have confirmed a major outage. You can track the status on their public status tracker - https://status.360dialog.com/clfrzlu6l302950ybof382vxl1z Delayed WhatsApp https://status.qcontact.com/incident/190004 Tue, 28 Mar 2023 08:09:00 -0000 https://status.qcontact.com/incident/190004#f732e5ae7312a1a72e47f08ae28afdf1ab6047bbe747fc489ce499283a6e5962 We are seeing high delays on WhatsApp message processing. We are communicating with 360Dialog to diagnose the issue Azure Issue in South Africa https://status.qcontact.com/incident/168022 Fri, 27 Jan 2023 12:03:00 -0000 https://status.qcontact.com/incident/168022#f8936aeea9354033f3483ebfaa571ca3a0ec58e8571f70e24f484b58c0d107b1 Initial RFO from Microsoft ### What happened? Between 07:05 UTC and 12:43 UTC on 25 January 2023, customers experienced issues with networking connectivity, manifesting as long network latency and/or timeouts when attempting to connect to resources hosted in Azure regions, as well as other Microsoft services including Microsoft 365 and Power Platform. While most regions and services had recovered by 09:00 UTC, intermittent packet loss issues were fully mitigated by 12:43 UTC. This incident also impacted Azure Government cloud services that were dependent on Azure public cloud. ### What went wrong and why? We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity across regions, as well as cross-premises connectivity via ExpressRoute. As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed. ### How did we respond? Our monitoring initially detected DNS and WAN related issues from 07:12 UTC. We began investigating by reviewing all recent changes. By 08:10 UTC, the network started to recover automatically. By 08:20 UTC, as the automatic recovery was happening, we identified the problematic command that triggered the issues. Networking telemetry shows that nearly all network devices had recovered by 09:00 UTC, by which point the vast majority of regions and services had recovered. Final networking equipment recovered by 09:35 UTC. Due to the WAN impact, our automated systems for maintaining the health of the WAN were paused, including the systems for identifying and removing unhealthy devices, and the traffic engineering system for optimizing the flow of data across the network. Due to the pause in these systems, some paths in the network experienced increased packet loss from 09:35 UTC until those systems were manually restarted, restoring the WAN to optimal operating conditions. This recovery was completed at 12:43 UTC. ### How are we making incidents like this less likely or less impactful? * We have blocked highly impactful commands from getting executed on the devices (Completed) * We will require all command execution on the devices to follow safe change guidelines (Estimated completion: February 2023) This is our Preliminary PIR that we endeavor to publish within 3 days of incident mitigation, to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details/learnings. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 10:42:00 -0000 https://status.qcontact.com/incident/168022#25ef0c6a23a4656bd495d37e00d6cf22ad37b4ad88e3d6e8553075ea770dfb35 We have seen incoming SIP UDP traffic successfully terminate in South Africa regions. We will monitor before removing the current redirection of SIP traffic via other regions. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 09:39:00 -0000 https://status.qcontact.com/incident/168022#45d5746b9e3c38e563405db50cc3ae5cfee37171638b72851d3eb56aa74f6c0d We are seeing issues with SIP requests sent over UDP Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 08:55:00 -0000 https://status.qcontact.com/incident/168022#df41c9f400df8cc727e7079c56d61f4bf43538960d951ecebe151dcfe04e4916 Update from Microsoft - they now confirm the issue started at 07:05 UTC (09:05 SAST). We first saw issues picked up by our automated monitoring at 07:12 UTC (09:12 SAST) Azure Networking - Multiple regions - Investigating Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services. We are actively investigating and will share updates as soon as more is known. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 08:34:00 -0000 https://status.qcontact.com/incident/168022#d4069994af09e6c55be19a423a1bc7136750d7fff422c18fc84332a06eddf294 Azure have officially confirmed the issue - we await further information and RFO Azure Networking - Multiple regions - Investigating Starting at 07:30 UTC, we're aware of a networking issue impacting connectivity to Azure for a subset of users. We are actively investigating and will share updates as soon as more is known. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 08:21:00 -0000 https://status.qcontact.com/incident/168022#44dc7b2d74bfca9c73fa880487b07508623b099ef3ffe3693e586b9d94a4236a We continue to see stabilised connectivity. We will keep monitoring but service appears to be back to normal. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 08:11:00 -0000 https://status.qcontact.com/incident/168022#35ebcb1d0a498fb0d83064f10cd0467982bc60f4a83fbc5871d40742e9a039fb We are seeing latency return to normal levels. Monitoring Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 08:08:00 -0000 https://status.qcontact.com/incident/168022#2ac222db0126c80199b866d35b74edec9af0f652c1f7909af1bc852902ba1ee0 We are seeing stabilisation of domestic connectivity within South Africa, albeit at elevated latency. Packet loss is currently near zero, but latency is around 200ms higher than usual. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 07:51:00 -0000 https://status.qcontact.com/incident/168022#d6d9dc8fbba541dc1fbf38b12c82d7fbe1dbb6f71a569a6cb5b8c329a526944c Where previously we saw no connectivity drops between other regions (for example UK South -> ZA North), we are now seeing intermitent connectivity issues within the Azure global network too. Azure have opened an emerging issue - we are unaware if this issue is linked the issues we are seeing in ZA North + ZA West. We have also raised a critical fault with our Azure representatives. Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 07:42:00 -0000 https://status.qcontact.com/incident/168022#e3c965936534ce751d9d622ffe224ba24f283877220f2b721f778365f9b22a24 We are seeing latency indicating Azure is re-routing traffic outside of South Africa. Local destinations normally <3ms from JNB, are taking 280ms+ with 60%+ packet loss Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 07:31:00 -0000 https://status.qcontact.com/incident/168022#62fd8f5e44cdb2e6033caf40132b5d61bb9c19693ab9f2e7d8bbc06da76bb5b6 We are currently seeing 60%+ packet loss from Azure to local South African networks from both South African regions (JNB + CPT) Azure Issue in South Africa https://status.qcontact.com/incident/168022 Wed, 25 Jan 2023 07:24:00 -0000 https://status.qcontact.com/incident/168022#e64a02859872f466f93ba4c37c94598be54e135b888ec1d1172c9cfbe0fd8080 We are seeing issues in Azure Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:30:00 +0000 https://status.qcontact.com/incident/165509#f40332222328cfa0316963175a1fdc9ed7ab95bbde36cf85fb250540b698788c Maintenance completed Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:30:00 +0000 https://status.qcontact.com/incident/165509#f40332222328cfa0316963175a1fdc9ed7ab95bbde36cf85fb250540b698788c Maintenance completed Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:30:00 +0000 https://status.qcontact.com/incident/165509#f40332222328cfa0316963175a1fdc9ed7ab95bbde36cf85fb250540b698788c Maintenance completed Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:00:00 -0000 https://status.qcontact.com/incident/165509#bfe8777110bab97e5120778e2c47b89ce36207e396692bcc7cad50aa1594bb0a As part of our continued commitment to system stability, we will be performing a test of our disaster recovery plans. This may involve your users logged in at this time to briefly see a reconnecting message, but we do not expect any live calls to be interrupted. Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:00:00 -0000 https://status.qcontact.com/incident/165509#bfe8777110bab97e5120778e2c47b89ce36207e396692bcc7cad50aa1594bb0a As part of our continued commitment to system stability, we will be performing a test of our disaster recovery plans. This may involve your users logged in at this time to briefly see a reconnecting message, but we do not expect any live calls to be interrupted. Disaster Recovery Test https://status.qcontact.com/incident/165509 Thu, 19 Jan 2023 01:00:00 -0000 https://status.qcontact.com/incident/165509#bfe8777110bab97e5120778e2c47b89ce36207e396692bcc7cad50aa1594bb0a As part of our continued commitment to system stability, we will be performing a test of our disaster recovery plans. This may involve your users logged in at this time to briefly see a reconnecting message, but we do not expect any live calls to be interrupted. WhatsApp Outage https://status.qcontact.com/incident/129488 Tue, 25 Oct 2022 08:53:00 -0000 https://status.qcontact.com/incident/129488#f63832a062a54a30a52ba2cf507ea3ba037b9024f60aa60746f144b6ca0a3c47 We have received no confirmation from Meta, but we see service returning to normal. WhatsApp Outage https://status.qcontact.com/incident/129488 Tue, 25 Oct 2022 07:00:00 -0000 https://status.qcontact.com/incident/129488#0552a8de010ba8b2f9cd3f22730bfc4d8f9d32f7ba4abaf594e2bf0cffe5b892 We are aware of a global WhatsApp outage at present. All messages to and from customers will be queued and retried. We await further updates from Meta - at this time we have no ETA. MTN Numbers https://status.qcontact.com/incident/116878 Sat, 10 Sep 2022 13:34:00 -0000 https://status.qcontact.com/incident/116878#0616c04900235454f1513fa12dc38709b146d158bb08101350cf54681a59d7a0 We are seeing calls connecting successful at present. We will continue to monitor, but we are marking the issue as resolved. MTN Numbers https://status.qcontact.com/incident/116878 Sat, 10 Sep 2022 11:45:00 -0000 https://status.qcontact.com/incident/116878#2bb400d5217fb897671dfd4ff2ce489947f5d6e8619f2acfcd3efded53c9c405 We are aware of an issue with some MTN numbers not connecting. This is an issue with MTN's voice network, and is affecting calls from both landlines and mobiles. If you have an MTN hosted number, your service may currently be disrupted. For further details please contact your MTN account manager, as this issue is outside of QContact. SBC Updates https://status.qcontact.com/incident/112374 Thu, 25 Aug 2022 21:30:00 +0000 https://status.qcontact.com/incident/112374#39ebb07a152c404f3ba27d7fe02062adcf55a7fafcd798397fd866b783da9a8d Maintenance completed SBC Updates https://status.qcontact.com/incident/112374 Thu, 25 Aug 2022 21:30:00 +0000 https://status.qcontact.com/incident/112374#39ebb07a152c404f3ba27d7fe02062adcf55a7fafcd798397fd866b783da9a8d Maintenance completed SBC Updates https://status.qcontact.com/incident/112374 Thu, 25 Aug 2022 21:00:00 -0000 https://status.qcontact.com/incident/112374#6a2e0d05ec2cef962745acf2e5435d96f3bd3efa834f2d1a69020c4d78a131d2 We will be upgrading our Azure SBC network to the latest software versions. Each SBC will take up to 30 seconds to update. If you are using DNS entries as advised in our Interop documents, you will not notice any disruption as any incoming calls will be automatically routed to an alternative SBC. However if you have hard-coded a fixed SBC, you may be unable to receive any calls during the 30 second window your particular hard coded SBC is being updated. SBC Updates https://status.qcontact.com/incident/112374 Thu, 25 Aug 2022 21:00:00 -0000 https://status.qcontact.com/incident/112374#6a2e0d05ec2cef962745acf2e5435d96f3bd3efa834f2d1a69020c4d78a131d2 We will be upgrading our Azure SBC network to the latest software versions. Each SBC will take up to 30 seconds to update. If you are using DNS entries as advised in our Interop documents, you will not notice any disruption as any incoming calls will be automatically routed to an alternative SBC. However if you have hard-coded a fixed SBC, you may be unable to receive any calls during the 30 second window your particular hard coded SBC is being updated. MTN SIP Trunks https://status.qcontact.com/incident/109532 Sun, 14 Aug 2022 15:00:00 -0000 https://status.qcontact.com/incident/109532#e967ad40d8438883728490848a9ee03a1a723c3bc2a0952836a47abbe8d6a869 We are closing after several hours of stability being observed. MTN SIP Trunks https://status.qcontact.com/incident/109532 Sun, 14 Aug 2022 12:10:00 -0000 https://status.qcontact.com/incident/109532#48dd0a5a1afbe7c99a03cc24716779df5b71fb21178c6725adfb6c84ebaf86f3 We have started to see calls over 5 minutes in duration from MTN trunks. We will continue to observe. MTN SIP Trunks https://status.qcontact.com/incident/109532 Sun, 14 Aug 2022 12:08:00 -0000 https://status.qcontact.com/incident/109532#246e7d502f15a5b72489e1b009ab614f0ab5e257cec584a65dc4881866918961 We have been notified that this issue started earlier this morning, and MTN are aware of the issue. MTN SIP Trunks https://status.qcontact.com/incident/109532 Sun, 14 Aug 2022 10:42:00 -0000 https://status.qcontact.com/incident/109532#b48afc0fff1dc38c1546bb9a3a1ade8bde50693952b8fe27d7a1b78721b3ae96 We are aware of an issue MTN is experiencing with their SIP trunk service. This is causing calls to end at exactly 5 minutes. The signal to end the call is being sent by MTN so we believe there is an issue with their platform. We have seen the same behaviour across multiple clients. We are waiting on feedback from MTN. QContact’s own telephony service is working without issue, so affected clients can switch to our telephony service if required. You can also instantly provision telephone numbers to point your sharecall services to work around the fault. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 15:53:00 -0000 https://status.qcontact.com/incident/105564#5c3bd0acd43e3dd6316cc6ba6532ff957d2b739b87405e8192560597b9fb8cf6 As we have now received confirmation from Microsoft the issue is deemed resolved from their side, we are closing this issue. We will await further RCA from Azure, but their current provided Root Cause is an unexpected increase in traffic - most likely a large scale DDoS attack given the number of regions affected. Update from Microsoft (MO406867) Current status: Our monitoring telemetry indicates that the issue has recovered. We're continuing to actively monitor the environment and perform a set of proactive changes to prevent this type of issue from reoccurring. Scope of impact: This issue is specific to a subset of traffic routed through infrastructure in South Africa, Brazil, North America, Canada, Japan and Korea. This issue is also causing impact for users located in regions surrounding the infrastructure hosted in those locations. Root cause: An unexpected increase in user requests resulted in network congestion. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 14:29:00 -0000 https://status.qcontact.com/incident/105564#dfcaa5347e1cc8795b380a411393856d6379fe974c61f6f4eee74b8cf8c5bc49 Update from Microsoft (MO406867) Current status: Our monitoring telemetry indicates that the issue has recovered. We're continuing to actively monitor the environment and perform a set of proactive changes to prevent this type of issue from reoccurring. Scope of impact: This issue is specific to a subset of traffic routed through infrastructure in South Africa, Brazil, North America, Canada, Japan and Korea. This issue is also causing impact for users located in regions surrounding the infrastructure hosted in those locations. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 14:07:00 -0000 https://status.qcontact.com/incident/105564#9b87874daf42698df58cd5e9f51802865e6c00160beb11eed040468d354e4c11 We continue to see no further issues, but are awaiting for an official confirmation of resolution from Microsoft ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 13:40:00 -0000 https://status.qcontact.com/incident/105564#a14411327e1dda567fa4220f6c96ddf3030bd9f94e2adc5532962b0c65be6f06 We have seen zero packet loss and latency spikes for past 10 minutes. We will keep issue open pending official confirmation of resolution ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 13:30:00 -0000 https://status.qcontact.com/incident/105564#4e31893935d2bc5788c6eeba9527c224d5422746b5991ee0ca4e29aa34ea1dd6 Latest update from Microsoft We've determined an unexpected spike in traffic is causing network congestion for some customers, resulting in the impact seen. We are currently applying multiple mitigation strategies to alleviate the connectivity issues. An update will be provided in the next 60 minutes or as events warrant. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 13:00:00 -0000 https://status.qcontact.com/incident/105564#60f94d372e4e3dbebe44b44ebacdb8c7f3d2630b8d65eea11d61dbfe85e06b1b Further update from Microsoft (MO406867) We’ve identified a potential networking issue resulting in impact. We’re reviewing service monitoring telemetry to verify that this is the root cause, before formulating a plan for remediation. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 12:50:00 -0000 https://status.qcontact.com/incident/105564#1ccf976c997f241c5bb931d4d482c8a35e67fa5ffca0180b3721f8139bc3e661 We are seeing a change in pattern of the issue - previously we would see periodic packet loss and prolonged latency spikes. We are currently seeing shorter periods of packet loss and no latency spikes. This should lead to an improvement in call quality. Microsoft confirm issue began at 10am SAST - we are still waiting on an update as to a fix. However we hope this change in pattern indicates a solution is in progress. We fully appreciate the severity of this issue, and will await a full RCA from Azure in due course. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 12:08:00 -0000 https://status.qcontact.com/incident/105564#7b3c418810aacb52158b3c2c749487a409c4aca17aab06e65924d1b1d6a65017 Update from Microsoft We've identified a potential root cause and are currently investigating mitigation options. An update will be provided in the next 60 minutes or as events warrant. ZA North Azure Connectivity https://status.qcontact.com/incident/105564 Fri, 29 Jul 2022 09:30:00 -0000 https://status.qcontact.com/incident/105564#17082f8a93a5009b22c4df36d4911d5fdf756e5baef10ac170f1442b3ff49b98 Azure ZA North Region is experiencing periodic periods of increased latency to the public internet. This will affect call quality. We await an official announcement with an ETA from Azure. UK Calling https://status.qcontact.com/incident/93880 Tue, 14 Jun 2022 21:00:00 +0000 https://status.qcontact.com/incident/93880#83656cbaea072da02d35a1d5239ba87d439cfe76c4367b6c4ed112904e053638 Maintenance completed UK Calling https://status.qcontact.com/incident/93880 Tue, 14 Jun 2022 17:00:00 -0000 https://status.qcontact.com/incident/93880#50ee448f8576254012c67332515718ce95b59a17a11360b9bb16a8575423fe5c Our primary upstream telephone carrier for UK calls is conducting maintenance on their SIP network. Whilst no interruption is expected, they have asked us to consider this period "at risk". Where necessary we will re-route calls via alternative paths. South Africa Incoming SMS https://status.qcontact.com/incident/83841 Mon, 25 Apr 2022 13:37:00 -0000 https://status.qcontact.com/incident/83841#765adc3137559533199e42628dcf1b842d941e6c19ca1431a8568cb937d71bc9 SMS MO has been restored. Resolution delay was caused by SMSC processes failing to start after forced shutdown due to AC failure. Engineers will remain on site to monitor the situation. South Africa Incoming SMS https://status.qcontact.com/incident/83841 Mon, 25 Apr 2022 06:32:00 -0000 https://status.qcontact.com/incident/83841#e207829a576b131fb81349c236a7b4a6a0f82eb1628ee184edcc3b6ef1be3733 Carrier has confirmed this is due to an AC failure in the datacentre. Technicians are on site carrying out repairs. South Africa Incoming SMS https://status.qcontact.com/incident/83841 Mon, 25 Apr 2022 06:15:00 -0000 https://status.qcontact.com/incident/83841#e979d276a1d1e8e5ee78c20c236b576148e1b52a42478d203a23389b77fa3ae7 We are aware of an issue receiving incoming SMS messages to 060 mobile numbers in South Africa due to a carrier datacentre issue. We will update when we have further information or ETA of resolution. WhatsApp sending errors https://status.qcontact.com/incident/74915 Wed, 09 Mar 2022 09:11:00 -0000 https://status.qcontact.com/incident/74915#2decc619fae609671b084512530a8b96ee0f4a8a7ec9244b6b65633f50412fdd We believe the issue is now resolved by WhatsApp WhatsApp sending errors https://status.qcontact.com/incident/74915 Wed, 09 Mar 2022 08:17:00 -0000 https://status.qcontact.com/incident/74915#db6d49c226ebb22eae84b39ac84d68fdb5ee3b5ff9852d83b65af2b8f148ef0b We are aware of a small percentage of outgoing messages on WhatsApp being rejected by WhatsApp. We are working with 360Dialog to find the cause of the issue. WhatsApp service disrupted https://status.qcontact.com/incident/71307 Wed, 16 Feb 2022 22:02:00 -0000 https://status.qcontact.com/incident/71307#cf3886fa0ee17b024311662329b84e8908dcf8f0c7660e02d192b1be4a754367 We have received no reports of this issue affecting any of our customers, and are experiencing no failures on the API, so we are closing the issue. WhatsApp service disrupted https://status.qcontact.com/incident/71307 Wed, 16 Feb 2022 21:27:00 -0000 https://status.qcontact.com/incident/71307#1099303578faf75afab72a2ba757be58afdf9c55a0b1ad515c653f1b29235ec2 We are aware there is an issue with the WhatsApp Business API for some users. Engineers from WhatsApp are investigating. South Africa - Voice Connectivity (Xneelo only) https://status.qcontact.com/incident/69128 Thu, 03 Feb 2022 12:17:00 -0000 https://status.qcontact.com/incident/69128#28ec04b918691f2bb598bfbbc866ab8c2d7e1f801b5fb5e681c06211b5fb2eaf We are seeing normal volumes traversing the Xneelo datacentre. South Africa - Voice Connectivity (Xneelo only) https://status.qcontact.com/incident/69128 Thu, 03 Feb 2022 12:07:00 -0000 https://status.qcontact.com/incident/69128#67431078cccdda2b53d162182fcbad565eeb5647f9c3f3379a9b922cd56f8953 The problematic voice media server has been cycled. South Africa - Voice Connectivity (Xneelo only) https://status.qcontact.com/incident/69128 Thu, 03 Feb 2022 12:01:00 -0000 https://status.qcontact.com/incident/69128#444b0c3b2136488a4018c17c9dd495f24a520523be58ee84580ce09eddfe54db We are aware of slow registrations for voice connectivity for South African customers with voice hosted in Xneelo - customers with their voice connectivity in Azure and Teraco are unaffected.