Degraded performance on api endpoint

Incident Report for Blueshift

Resolved

Blueshift services are operating normally. Operations teams continue to monitor overall health.
Posted Apr 22, 2020 - 13:45 PDT

Monitoring

Blueshift Services are continuing recovery and holding up healthy.
Posted Apr 22, 2020 - 11:07 PDT

Update

A workaround has been deployed and services are continuing to recover. We are monitoring the progress.
Posted Apr 22, 2020 - 10:13 PDT

Update

We are continuing to work on a fix for this issue.
Posted Apr 22, 2020 - 09:41 PDT

Identified

We are temporarily pausing the event processing.
Posted Apr 22, 2020 - 07:55 PDT

Update

We are continuing to investigate this issue.
Posted Apr 22, 2020 - 07:33 PDT

Update

A fix has been identified and we are rolling it out.
Posted Apr 22, 2020 - 07:32 PDT

Update

Blueshift API service on AWS is experiencing intermittent DNS resolution issues and this has been more pronounced since 7:30AM UCT (12:30AM PDT). Operations and Engineering teams are working with AWS to identify the issue.
Posted Apr 22, 2020 - 07:31 PDT

Update

We are continuing to investigate the issue.
Posted Apr 22, 2020 - 05:54 PDT

Update

We are experiencing DNS issues on AWS US-West-2 region and Blueshift services are going through higher error rates. We are looking into mitigating them.
Posted Apr 22, 2020 - 04:17 PDT

Investigating

We are currently investigating this issue.
Posted Apr 22, 2020 - 02:53 PDT
This incident affected: Event processing & API.