tag:status.blueshift.com,2005:/historyBlueshift Status - Incident History2024-03-28T22:23:49-07:00Blueshifttag:status.blueshift.com,2005:Incident/203255882024-04-12T19:30:00-07:002024-03-21T23:10:29-07:00(US & EU) Database upgrades for app.getblueshift.com and app.eu.getblueshift.com<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>12</var>, <var data-var='time'>19:30</var> - <var data-var='time'>22:30</var> PDT</strong></p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>23:10</var> PDT</small><br><strong>Scheduled</strong> - We will be conducting upgrades on various US & EU databases. The maintenance window duration will be 3 hrs (180 mins), beginning at 730pm PST on Friday, 12th April 2024, and affects both our US & EU data centers.<br /><br />As a result, logging in and accessing the Blueshift US & EU platform will not be possible. This necessitates the temporary pause of ALL campaigns. While events will still be accepted, processing will be delayed. Event-triggered campaigns, especially those utilizing trigger filter segments, will be delayed. Both one-time and segment-triggered campaigns will also be impacted. Recurring campaigns scheduled during this time frame will be delayed. Syndications scheduled to go in the maintenance window will be delayed. Write APIs will be inaccessible, but READ APIs will remain available. Live Content will be unavailable, rendering "Under Maintenance" message to the user.<br /> <br />Upon completion of the maintenance window, campaigns will be unpaused, events will resume processing and will continue normally.</p>tag:status.blueshift.com,2005:Incident/202234222024-03-12T04:16:59-07:002024-03-12T04:16:59-07:00Seeing login issues on our EU datacenter (app.eu.getblueshift.com)<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>04:16</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>04:05</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>02:51</var> PDT</small><br><strong>Identified</strong> - We are creating a hotfix for this issue.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>02:48</var> PDT</small><br><strong>Investigating</strong> - We are currently seeing few failed logins on our EU datacenter.</p>tag:status.blueshift.com,2005:Incident/199098222024-02-16T11:36:56-08:002024-02-16T13:14:40-08:00(EU only) Blueshift Database Upgrade for app.eu.getblueshift.com<p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>11:36</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>10:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>22:33</var> PST</small><br><strong>Scheduled</strong> - This notice is ONLY for app.eu.getblueshift.com (our EU datacenter).<br /><br />On Friday, Feb 16th 2024, starting at 7:00 pm CET, we will be conducting upgrades on various EU databases. The maintenance window duration will be 3 hrs (180 mins).<br /><br />As a result, logging in and accessing the Blueshift EU platform will not be possible. This necessitates the temporary pause of ALL campaigns. While events will still be accepted, processing will be delayed. Event-triggered campaigns, especially those utilizing trigger filter segments, will be delayed. Both one-time and segment-triggered campaigns will also be impacted. Recurring campaigns scheduled during this time frame will be delayed. Syndications scheduled to go in the maintenance window will be delayed. Write APIs will be inaccessible, but read APIs will remain available.</p>tag:status.blueshift.com,2005:Incident/199804812024-02-13T20:35:54-08:002024-02-13T20:35:54-08:00Campaigns delays due to orchestration service outage<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>20:35</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>20:35</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>20:35</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>18:59</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>18:59</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/199785532024-02-13T16:46:06-08:002024-02-13T16:46:06-08:00Campaigns interrupted due to orchestration service outage<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>16:46</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>14:08</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>13:38</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>13:36</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/197934182024-01-22T07:15:08-08:002024-01-22T07:15:08-08:00Seeing database errors in our USA datacenter (app.getblueshift.com)<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>07:15</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>04:22</var> PST</small><br><strong>Monitoring</strong> - We have tweaked some settings on the database that were causing the errors. We are monitoring now.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>04:20</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>04:08</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/193983992024-01-19T21:05:59-08:002024-01-19T21:05:59-08:00(US only) Blueshift Database Upgrade for app.getblueshift.com<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>21:05</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>18:31</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>23:02</var> PST</small><br><strong>Scheduled</strong> - This notice is ONLY for app.getblueshift.com (our US datacenter).<br /><br />On Friday, Jan 19th 2024, at 630 pm PST we will be conducting upgrades on various US databases. The maintenance window duration will be 3 hrs (180 mins). <br /><br />During this time, logging in and accessing the Blueshift US platform will not be possible. This requires the temporary pause of ALL campaigns. While events will still be accepted, processing will be delayed. Event-triggered campaigns, especially those utilizing trigger filter segments, will be delayed. Both one-time and segment-triggered campaigns will also be impacted. Recurring campaigns scheduled during this time frame will be delayed. Syndications scheduled to go in the maintenance window will be delayed. Write APIs will be inaccessible, but READ APIs will remain available.</p>tag:status.blueshift.com,2005:Incident/191858742024-01-04T05:22:59-08:002024-01-04T05:22:59-08:00(EU only) Blueshift Database Upgrade for app.eu.getblueshift.com<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>05:22</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>05:21</var> PST</small><br><strong>Update</strong> - We are cancelling this (EU only) maintenance for app.eu.getblueshift.com. We will update you on the next date for this.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>00:48</var> PST</small><br><strong>Scheduled</strong> - This notice is ONLY for app.eu.getblueshift.com (our EU datacenter).<br /><br />On Friday, Jan 5th 2024, we will be conducting upgrades on various EU databases. The maintenance window duration will be 3 hrs (180 mins). <br /><br />As a result, logging in and accessing the Blueshift EU platform will not be possible. This necessitates the temporary pause of ALL campaigns. While events will still be accepted, processing will be delayed. Event-triggered campaigns, especially those utilizing trigger filter segments, will be delayed. Both one-time and segment-triggered campaigns will also be impacted. Recurring campaigns scheduled during this time frame will be delayed. Syndications scheduled to go in the maintenance window will be delayed. Write APIs will be inaccessible, but READ APIs will remain available.</p>tag:status.blueshift.com,2005:Incident/195978272024-01-02T18:48:07-08:002024-01-02T18:48:07-08:00(EU only) Emergency database reboot for app.eu.getblueshift.com<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>18:48</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>18:46</var> PST</small><br><strong>Monitoring</strong> - Reboot is complete. Dashboard is functioning normally now.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>18:44</var> PST</small><br><strong>Identified</strong> - Reboot started</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>18:41</var> PST</small><br><strong>Investigating</strong> - We need to perform an emergency reboot of our master database. This will take about 10 minutes. Errors will be seen on the dashboard during this time. This procedure is required because we need to enable some settings on the database that are a prerequisite to our longer scheduled database migration maintenance procedure that is scheduled on Friday, 5th January 2024.</p>tag:status.blueshift.com,2005:Incident/191216162023-11-14T13:08:58-08:002023-11-14T13:28:47-08:00Customers API Outage for api.eu.getblueshift.com<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:08</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:07</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:07</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>13:03</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/190481752023-11-07T09:44:04-08:002023-11-07T09:44:04-08:00Seeing errors in dashboard<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:44</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:40</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:36</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:32</var> PST</small><br><strong>Update</strong> - Issue is related to reporting & graphs. Other dashboard functionality is working normally.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>09:25</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/190003462023-11-02T04:30:00-07:002023-11-02T06:53:50-07:00Delayed campaigns in our EU datacenter<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>04:30</var> PDT</small><br><strong>Resolved</strong> - Campaigns that use Static/Dayparting/Send time optimization are delayed. This is specific to the EU datacenter.<br />The USA datacenter is operating normally.</p>tag:status.blueshift.com,2005:Incident/189336572023-10-26T03:00:00-07:002023-10-26T04:36:09-07:00Few one time/recurring campaigns and segment triggered campaigns were delayed<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>03:00</var> PDT</small><br><strong>Resolved</strong> - A small number of one time/recurring campaigns and segment triggered campaigns were delayed. Approximate duration for the delay was 1 hour. We have fixed the issue and services are operating normally now.</p>tag:status.blueshift.com,2005:Incident/187622722023-10-12T00:48:03-07:002023-10-12T00:48:04-07:00Observing delayed campaigns<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>00:48</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>00:42</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>00:42</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>00:18</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/186684832023-10-02T10:20:07-07:002023-10-02T10:20:07-07:00Intermittent api errors on the EU endpoint api.eu.getblueshift.com<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:20</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:12</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:11</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/185408232023-09-18T23:16:00-07:002023-09-18T23:16:00-07:00Ongoing Latencies for Blueshift API (related to AWS Outage)<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>23:16</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>22:20</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>20:37</var> PDT</small><br><strong>Identified</strong> - We continue to see AWS related network latencies. We are monitoring the state.<br /><br />Quoting from the AWS status text for more clarity.<br /><br />https://health.aws.amazon.com/health/status<br /><br />Network Connectivity Issues<br />Sep 18 8:01 PM PDT We continue to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. We continue to see improvement to network mapping latencies but they are not at normal levels yet. Other AWS services are also starting to see recovery as network mapping latencies improve. We will continue to keep you updated until network mapping latencies have returned to normal levels.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>18:20</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.<br />Associated AWS outage link: https://health.aws.amazon.com/health/status</p>tag:status.blueshift.com,2005:Incident/185076562023-09-18T08:49:02-07:002023-09-18T13:39:42-07:00Blueshift database upgrade<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>08:49</var> PDT</small><br><strong>Completed</strong> - We are cancelling this scheduled maintenance activity in Oct-2023. We are postponing this to January 2024 and will send out another update when we have the final date.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>22:45</var> PDT</small><br><strong>Scheduled</strong> - We will be doing a database upgrade during this window. Affected components will be mentioned in a notice sent out by support/csm.</p>tag:status.blueshift.com,2005:Incident/185311302023-09-17T22:33:49-07:002023-09-17T22:33:49-07:00Seeing intermittent errors on event api<p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>22:33</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>21:38</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>20:58</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/182111452023-08-21T08:19:44-07:002023-08-21T08:19:44-07:00Blueshift dashboard not loading<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>08:19</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>08:17</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>08:16</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>08:07</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/181856402023-08-18T08:52:16-07:002023-08-18T08:52:16-07:00Seeing increased latencies for internal stats service<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>08:52</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>08:16</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>08:15</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>08:11</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/181768882023-08-17T16:00:53-07:002023-08-17T16:00:53-07:00Internal stats services are experiencing failure<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>16:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>15:39</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>14:04</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>14:03</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>12:46</var> PDT</small><br><strong>Identified</strong> - We have identified the cause of the outage. We are working on recovering the service</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>12:40</var> PDT</small><br><strong>Investigating</strong> - Investigating an issue with the stats service. We are working to make it functional again.</p>tag:status.blueshift.com,2005:Incident/181454562023-08-14T13:44:22-07:002023-08-14T13:44:22-07:00Seeing increased latencies for Blueshift API api.getblueshift.com<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>13:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>13:34</var> PDT</small><br><strong>Monitoring</strong> - The associated services have been scaled appropriately and API latencies seem to be recovering.</p>tag:status.blueshift.com,2005:Incident/180765912023-08-07T11:02:35-07:002023-08-07T11:02:35-07:00Event ingestion pipeline is experiencing transient processing capacity issues<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:17</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>09:29</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.blueshift.com,2005:Incident/177790692023-07-20T20:51:23-07:002023-07-20T20:51:23-07:00Blueshift database upgrade<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>20:51</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>19:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>03:21</var> PDT</small><br><strong>Scheduled</strong> - We will be upgrading multiple databases during this time. This will require pausing ALL campaigns and ALL events. All other services will be remain functional.</p>tag:status.blueshift.com,2005:Incident/178339382023-07-12T06:30:00-07:002023-07-12T06:31:20-07:00Delays in segment triggered campaigns<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>06:30</var> PDT</small><br><strong>Resolved</strong> - We saw some delays in segment triggered campaigns. This was identified to a product bug that was resolved by rolling out a hotfix today.</p>