EU infrastructure outage
Incident Report for Front
Resolved
Front is back up and running.
Posted about 2 months ago. Mar 27, 2019 - 23:00 UTC
Update
We are still processing our backlog of received messages (less than half remaining).
Front is fully accessible in the meantime.
Posted about 2 months ago. Mar 27, 2019 - 22:41 UTC
Monitoring
Front is back in EU. You can access the platform again.

We are actively working on a backlog of messages to import in the meanwhile.
Posted about 2 months ago. Mar 27, 2019 - 21:31 UTC
Identified
We are still actively working on this incident. We are almost done restoring data and will provide another update once this is done.

We have dedicated all resources to fix this issue as quick as possible. We won’t lose any data and the data will be safely restored.
Posted about 2 months ago. Mar 27, 2019 - 20:22 UTC
Update
We are still actively working on this incident.
We are working on multiple solutions to bring back Front.

We have dedicated all resources to fix this issue as quick as possible. We won’t lose any data and the data will be safely restored.
Posted about 2 months ago. Mar 27, 2019 - 17:43 UTC
Update
We are still actively working on this incident.
We are working closely with AWS and also keep on exploring other ways or bringing Front back.

We have dedicated all resources to fix this issue as quick as possible. We won’t lose any data and the data will be safely restored.
Posted about 2 months ago. Mar 27, 2019 - 16:12 UTC
Update
We are still in contact with AWS to recover our Database in Europe. We are exploring other ways to recover from this but it might take more time than expected initially. We don’t have a clear ETA yet, we have dedicated all resources to fix this issue as quick as possible. We won’t lose any data and the data will be safely restored.
Posted about 2 months ago. Mar 27, 2019 - 14:58 UTC
Update
The root cause of the incident is that the master encryption key of one of our databases was temporarily disabled. We have a failover database but unfortunately both share the same KMS key. We are restored the key and are bringing our database back online with the help of AWS.

At the same time, we are restoring from a point-in-time recovery, which has progressed to 10%. We are confident that we won't lose any data, but unfortunately we still don't have an ETA.

We are very sorry about this incident, our entire team is working on bringing back our EU infrastructure online.
Posted about 2 months ago. Mar 27, 2019 - 13:28 UTC
Update
One of our databases is our EU region is experiencing a major outage. We are looking at ways to bring it back online.
Posted about 2 months ago. Mar 27, 2019 - 12:27 UTC
Investigating
We are receiving alerts from our EU data-center. We are investigating.
Posted about 2 months ago. Mar 27, 2019 - 12:12 UTC
This incident affected: App and Sync engine.