Resolved -
This incident has been resolved.
Apr 30, 05:43 UTC
Monitoring -
As of 8:11 pm PDT (03:11 Wednesday UTC), the application is functioning for all customers. Front began applying all changes since the incident started, like syncing messages from email channels, which are now caught up. Customers may experience delays for a few more minutes as caches clear.
Front is now monitoring the application for further issues. All functionality should be restored; please contact support if you continue to see application issues.
[us-west-1] [by_company_00005]
Apr 30, 04:09 UTC
Update -
The database recovery has completed and application functionality is now restored. Messages from the last few hours since the incident started are now being re-synced. This will take some time to catch up, but we anticipate no data loss as a result of this issue.
[us-west-1] [by_company_00005]
Apr 30, 03:13 UTC
Update -
The restoration process is still underway, but nearing completion. We are targeting less than 60 minutes to restore limited application functionality.
[us-west-1] [by_company_00005]
Apr 30, 01:41 UTC
Identified -
At 4:16 pm PDT (23:16 UTC), during a routine maintenance operation on one of Front’s customer databases, the database was inadvertently disabled and made unrecoverable. Front immediately began the restoration procedure from our up-to-date backup, but as it has to copy the entire database this procedure takes time to complete. We expect this to take roughly two hours, during which time the application will be unavailable for customers on this database.
This issue affects a fraction of Front's customers, but for those customers the application will be unavailable until the restoration is complete.
[us-west-1] [by_company_00005]
Apr 29, 23:54 UTC
Investigating -
Front is investigating a major database outage affecting one shard of customers.
[us-west-1] [by_company_00005]
Apr 29, 23:34 UTC