Webhook delivery delay
Incident Report for Persona
Resolved
All system operations have been in normal ranges for the last 12 hours. We will continue to keep a close eye on performance. We are working with our cloud vendor to better understand the underlying problems and ensure mitigations are in place on their side to prevent further recurrence.
Posted Oct 17, 2023 - 11:37 UTC
Monitoring
Our cloud vendor has successfully mitigated the issue and performance is back to normal across all our products. We'll continue to monitor the situation closely over the next 24 hours to ensure there are no regressions.
Posted Oct 17, 2023 - 00:32 UTC
Update
Our cloud vendor is still working on mitigating the issue. We are observing periodic spikes in 5xx responses on our APIs as a result of this issue and a small number of webhook events are also experiencing delivery delays.

We will continue to monitor progress towards resolution and provide updates here.
Posted Oct 16, 2023 - 23:22 UTC
Update
We are working closely with our cloud vendor to remediate the issue. We will provide another update in about 2 hours on the issue.
Posted Oct 16, 2023 - 21:24 UTC
Identified
The issue has been tracked back to a problem in our cloud vendor's system. They are working to resolve the issue. We are seeing elevated 503 rates and delayed webhook delivery across all our products as a result of the problem.

We will provide another update in an hour on the issue.
Posted Oct 16, 2023 - 20:06 UTC
Update
We are continuing to investigate this issue. Another update will be posted within two hours.
Posted Oct 16, 2023 - 17:57 UTC
Investigating
We are currently experiencing a delay in webhook event delivery. The team is investigating and we will provide another update with an hour.
Posted Oct 16, 2023 - 15:58 UTC
This incident affected: API, Reports, Verifications, Workflows, Inquiries & Collections, Webhooks, and Cases.