Post mortem
What happened
On 29 October, between 21:15 and 22:15 UTC, users on our US production environment could log in, but the Main UI did not load.
Root cause
The Main UI lost its connection to an internal service that manages user sessions, and it did not reconnect automatically. The application still reported itself as healthy, so our monitoring did not alert us.
What we have done
Service was restored by restarting the Main UI as soon as we got reports. On 30 October we deployed a fix to theproduction environment. With the fix, the application keeps this connection alive and recovers automatically if the connection is lost.
Next steps
We are improving our monitoring so that this type of issue triggers an alert immediately.
We apologise for the disruption.
