Service Incident - June 1st, 2021 - Error Code 4

SUMMARY

On June 1st, 2021 from 09:48 UTC to 09:55 UTC, some customers using our software in the EU region experienced a UI message with “Error Code 4” when loading the system, which prevented them from seeing the software.

10:44 UTC | 03:44 PT

We’re happy to report that all known issues related to this incident have been resolved. We will publish a post-mortem on these events as soon as it is completed.

10:31 UTC | 03:31 PT

We have confirmed a service disruption affecting EU accounts leading to errors loading the system. We are working toward resolution and already see stability in the platform.

10:13 UTC | 03:13 PT

We are currently investigating reports of issues in the system for some customers. Affected customers will see an error code 4. Investigation is underway.

POST-MORTEM

Root Cause Analysis

This incident was caused by a deployment that caused the API to be down while rotating containers with the new code.

Resolution

To fix the issue, old jobs were replaced by the new ones with an up-to-date code version, and with the correct sync, the API became functional again.

Remediation Items

1. Create additional alerts with a smaller time frame between seeing errors and alerting engineers.
2. Establish a deployment process for this particular edge case.

FOR MORE INFORMATION

For current system status information check out our system status page. During an incident, you can also receive status updates by following us on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.