Delayed webhooks and scenario on US1-enterprise zone

Incident Report for Make

Resolved

This incident has been resolved.
Posted Oct 21, 2024 - 19:27 CEST

Update

All systems are back to normal operation.
We expect the queue of scenario activity logs to be fully processed within 4 hours and we will keep monitoring the situation until then.
Posted Oct 21, 2024 - 18:28 CEST

Monitoring

The cluster’s scenario execution functionality has been restored and all queued executions have been processed. The backlog of scenario execution log entries is being processed at a slower rate and activity logs for completed executions will be displayed in the scenario history with a delay.
Posted Oct 21, 2024 - 18:07 CEST

Update

We are continuing to work on a fix for this issue.
Posted Oct 21, 2024 - 17:41 CEST

Identified

We have identified the likely root cause of the problem and started implementing corrective action. We will provide another update in the next 30 minutes.
Posted Oct 21, 2024 - 17:15 CEST

Update

We are continuing to investigate this issue.
Posted Oct 21, 2024 - 16:50 CEST

Investigating

We are investigating delays with webhook processing and scenario processing on US1-enterprise zone. We will update this incident in 30 min with more details.
Posted Oct 21, 2024 - 16:49 CEST
This incident affected: Website (us1.make.celonis.com), Responsive Webhooks (us1.make.celonis.com), and Scenario Executions (us1.make.celonis.com).