Increased error rate sending and receiving messages on email-based channels

Incident Report for Front

Resolved

This incident has been resolved.
Posted Sep 28, 2024 - 02:51 UTC

Update

Customers in us-west-2 and eu-west-1 regions have fully recovered. We are still working through the incoming message backlog on us-west-1.
Posted Sep 28, 2024 - 02:21 UTC

Monitoring

The fix has been deployed and we are seeing email delivery services have recovered. We're monitoring and catching up on backlog so there might be residual delays.
Posted Sep 27, 2024 - 21:47 UTC

Update

We have validated the fix and are in the process of doing a manual deploy to circumvent the current Docker outage.
Posted Sep 27, 2024 - 21:27 UTC

Update

We are validating the fix in our staging environment.
Posted Sep 27, 2024 - 21:02 UTC

Identified

We have identified the cause of the issue and are working on a fix. The issue affects some but not all channels.
Posted Sep 27, 2024 - 20:38 UTC

Update

We are continuing to investigate this issue.
Posted Sep 27, 2024 - 20:25 UTC

Investigating

We are currently investigating this issue.
Posted Sep 27, 2024 - 20:25 UTC
This incident affected: SMTP (non-Gmail / O365), Gmail, and O365.