[Match API] Outage due to back-end third party service issue
Incident Report for Addresscloud
Postmortem

Firstly, we apologise for any inconvenience this has caused our customers.

We have been using the excellent Elasticsearch for our back-end data store for 5+ years and this is the first time we have seen an issue like this. The issue occurred in a routine update of the AddressBase Premium data. This appears to have been an issue with an Elasticsearch index that was snapshotted with “include_global_state” then restored with “include_global_state” which wiped out some key security settings on the destination cluster. In future indices should only ever be snapshotted and restored using the pre-prepared scripts and we will ensure this is enforced in our operations.

We are happy that the issue was identified and resolved within 5 minutes and will take steps to ensure this does not happen again.

Mark Varley, CEO, Addresscloud 12/12/2021

Posted Dec 12, 2021 - 14:43 GMT

Resolved
This incident has been resolved.
Posted Dec 12, 2021 - 14:42 GMT
Update
We are continuing to monitor for any further issues.
Posted Dec 12, 2021 - 14:19 GMT
Monitoring
We experienced an issue due to a change by a third party managed service. We have identified and applied a fix. We have raised an incident with the third party and are monitoring the situation. The Match API service was unavailable for 2-3 minutes
Posted Dec 12, 2021 - 14:15 GMT
This incident affected: Addresscloud Match API.